[Flow] Add pattern to reassociate dequantization + matmul `linalg.gen…

…eric` ops

Dequantization ops that are consumed by matmuls are currently only fused
into a dispatch region, but we can do even better by reassociating these
fused operations (see iree-org#14951).

It is important to note that this pattern does affect precision, and is
a trade off between precision and performance. It is set to opt-in with
`--iree-flow-enable-quantized-matmul-reassociation`

This pattern rewrites a sequence of dequantization->matmul `linalg.generic`
ops into a new sequence of `linalg.generic` ops. The new sequence of ops
is as follows:

  1. A sequence of `linalg.generic` ops that dynamically quantize the
     non-quantized input to the matmul. This is very cheap in skinny
     matmul cases, where the non-quantized input is small compared to
     the quantized input.
  2. A `linalg.generic` op that performs an integer matmul. This is the
     key performance optimization here. On CPU, we want to be doing
     integer matmuls where we can, but the matmul needs to be picked
     up by a VectorContractCustomKernel for now. Eventually it will
     be better to rewrite to `linalg.matmul` here to target ukernels.
  3. A final `linalg.generic` op that performs the dequantization
     scale and zero point math, as well as performing the remaining
     reduction of the matmul. The matmul from 2. only reduces within
     quantized groups, while this op does the reduction across groups.

compiler/src/iree/compiler/Dialect/Flow/Transforms/BUILD.bazel

@@ -83,6 +83,7 @@ iree_compiler_cc_library(
     ],
     deps = [
         ":PassesIncGen",
+        "//compiler/src/iree/compiler/Codegen/Dialect:IREECodegenDialect",
         "//compiler/src/iree/compiler/Dialect/Flow/Conversion/TensorToFlow",
         "//compiler/src/iree/compiler/Dialect/Flow/IR",
         "//compiler/src/iree/compiler/Dialect/HAL/IR",

compiler/src/iree/compiler/Dialect/Flow/Transforms/CMakeLists.txt

@@ -113,6 +113,7 @@ iree_cc_library(
     MLIRTransformDialectTransforms
     MLIRTransformUtils
     MLIRTransforms
+    iree::compiler::Codegen::Dialect::IREECodegenDialect
     iree::compiler::Dialect::Flow::Conversion::TensorToFlow
     iree::compiler::Dialect::Flow::IR
     iree::compiler::Dialect::HAL::IR