Support two outputs in split_reduce

[pfultz2]

No description provided.

[umangyadav]

rsum2 is at tuple elem index 0
and rsum1 is at tuple elem index 1.

Which can be error prone.

[umangyadav]

This PR fails to compile if i run
MIGRAPHX_DISABLE_LAYERNORM_FUSION=1 MIGRAPHX_ENABLE_SPLIT_REDUCE=1 ./bin/test_verify "test_layernorm_large"

[umangyadav]

This currently will not fuse
pointwise preceding "reduce" ops is pointwise is alive after the reduce_ops.

This is what happens with #3212

@6 = convolution[padding={1, 1, 1, 1},stride={1, 1},dilation={1, 1},group=1,padding_mode=0](sample,@0) -> float_type, {2, 320, 64, 64}, {1310720, 4096, 64, 1}
@7 = broadcast[axis=1,out_lens={2, 320, 64, 64}](@1) -> float_type, {2, 320, 64, 64}, {0, 1, 0, 0}
@8 = contiguous(@7) -> float_type, {2, 320, 64, 64}, {1310720, 4096, 64, 1}
@9 = reshape[dims={2, 32, 10, 64, 64}](@6) -> float_type, {2, 32, 10, 64, 64}, {1310720, 40960, 4096, 64, 1}
@10 = reshape[dims={2, 32, 10, 64, 64}](@8) -> float_type, {2, 32, 10, 64, 64}, {1310720, 40960, 4096, 64, 1}
@11 = multibroadcast[out_lens={2, 320, 64, 64},out_dyn_dims={}](@3) -> float_type, {2, 320, 64, 64}, {0, 1, 0, 0}
@12 = contiguous(@11) -> float_type, {2, 320, 64, 64}, {1310720, 4096, 64, 1}
@13 = multibroadcast[out_lens={2, 320, 64, 64},out_dyn_dims={}](@4) -> float_type, {2, 320, 64, 64}, {0, 1, 0, 0}
@14 = contiguous(@13) -> float_type, {2, 320, 64, 64}, {1310720, 4096, 64, 1}
@15 = reshape[dims={2, 32, 10, 64, 64}](@12) -> float_type, {2, 32, 10, 64, 64}, {1310720, 40960, 4096, 64, 1}
@16 = reshape[dims={2, 32, 10, 64, 64}](@14) -> float_type, {2, 32, 10, 64, 64}, {1310720, 40960, 4096, 64, 1}
@17 = pointwise(@9,@10), [main:pointwise0] -> float_type, {2, 32, 10, 64, 64}, {1310720, 40960, 4096, 64, 1}
@18 = split_fused_reduce[axes={2, 3, 4},assign=assign_add](@17), [main:pointwise0:main:pointwise2:main:reduce_sum1:main:pointwise4:main:pointwise6:main:pointwise1:main:reduce_sum0_reshape_reshape:main:pointwise10_split] -> [float_type, {2, 32, 1, 1, 1}, {32, 1, 1, 1, 1}, float_type, {2, 32, 1, 1, 1}, {32, 1, 1, 1, 1}]
@19 = get_tuple_elem[index=0](@18) -> float_type, {2, 32, 1, 1, 1}, {32, 1, 1, 1, 1}
@20 = get_tuple_elem[index=1](@18) -> float_type, {2, 32, 1, 1, 1}, {32, 1, 1, 1, 1}
@21 = multibroadcast[out_lens={2, 32, 10, 64, 64},out_dyn_dims={}](@19) -> float_type, {2, 32, 10, 64, 64}, {32, 1, 0, 0, 0}
@22 = multibroadcast[out_lens={2, 32, 10, 64, 64},out_dyn_dims={}](@20) -> float_type, {2, 32, 10, 64, 64}, {32, 1, 0, 0, 0}
@23 = pointwise(@21,@22,@17,@21,@15,@16), [main:pointwise4] -> float_type, {2, 32, 10, 64, 64}, {1310720, 40960, 4096, 64, 1}
@24 = reshape[dims={2, 320, 64, 64}](@23) -> float_type, {2, 320, 64, 64}, {1310720, 4096, 64, 1}
@25 = convolution[padding={1, 1, 1, 1},stride={1, 1},dilation={1, 1},group=1,padding_mode=0](@24,@2) -> float_type, {2, 320, 64, 64}, {1310720, 4096, 64, 1}

@17 is not fused with @18 because @17 isused later at @23.

[pfultz2]

@17 is not fused with @18 because @17 isused later at https://github.com/23.

Yea we need to support multi-output fusion in the future. We can probably just do the fusion initially with mlir.

[pfultz2]

This PR fails to compile if i run MIGRAPHX_DISABLE_LAYERNORM_FUSION=1 MIGRAPHX_ENABLE_SPLIT_REDUCE=1 ./bin/test_verify "test_layernorm_large"

This is fixed.