Merge branch 'main' into add_averagepool_onnx_to_tosa

.azure-pipelines/Windows-CI.yml

@@ -11,7 +11,8 @@ parameters:
 
 jobs:
 - job: Build_onnx_mlir_Windows
-  timeoutInMinutes: 240
+  # 4h timeout is sometimes a tiny bit short when llvm-project is rebuilt
+  timeoutInMinutes: 270
   pool:
     vmImage: 'windows-2019'
 

CMakeLists.txt

@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 # Match the minimum required version of LLVM and MLIR
-cmake_minimum_required(VERSION 3.13.4)
+cmake_minimum_required(VERSION 3.20.0)
 
 project(onnx-mlir)
 

docs/BuildOnLinuxOSX.md

@@ -15,7 +15,7 @@ Firstly, install MLIR (as a part of LLVM-Project):
 ``` bash
 git clone -n https://github.com/llvm/llvm-project.git
 # Check out a specific branch that is known to work with ONNX-MLIR.
-cd llvm-project && git checkout 91088978d712cd7b33610c59f69d87d5a39e3113 && cd ..
+cd llvm-project && git checkout 4acc3ffbb0af5631bc7916aeff3570f448899647 && cd ..
 ```
 
 [same-as-file]: <> (utils/build-mlir.sh)

docs/BuildOnWindows.md

@@ -52,7 +52,7 @@ Install MLIR (as a part of LLVM-Project):
 ```shell
 git clone -n https://github.com/llvm/llvm-project.git
 # Check out a specific branch that is known to work with ONNX-MLIR.
-cd llvm-project && git checkout 91088978d712cd7b33610c59f69d87d5a39e3113 && cd ..
+cd llvm-project && git checkout 4acc3ffbb0af5631bc7916aeff3570f448899647 && cd ..
 ```
 
 [same-as-file]: <> (utils/build-mlir.cmd)

docs/SupportedONNXOps-cpu.md

@@ -26,7 +26,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 19. Limitatio
 | **Asinh** |9 - * | | |
 | **Atan** |7 - * | | |
 | **Atanh** |9 - * | | |
-| **AveragePool** |6 - 18 | | |
+| **AveragePool** |6 - * | | |
 | **BatchNormalization** |6 - * |Training not supported. | |
 | **Bernoulli** |none | | | |
 | **Binarizer** |none | | | |

docs/Testing.md

@@ -36,7 +36,7 @@ The all_test_names.txt is automatically generated with command "make check-onnx-
 
 ### Adding ONNX-supported test cases to the current set of backend tests
 
-When the ONNX-to-Krnl conversion of an operator is added, the corresponding backend tests for this operator should be added to test.py. The available test cases can be found in `third_part/onnx/onnx/backend/test/case/node`. You can identify new tests by looking for the new operator in `test/backend/all_test_names.txt`. Once you have located new tests, you may add the new tests in the `test/backend/inference_backend.py.` Please note to add suffix `_cpu` to the onnx test name. Associated with the test, you can define how to run the tests for the new operator. For example:
+When the ONNX-to-Krnl conversion of an operator is added, the corresponding backend tests for this operator should be added to test.py. The available test cases can be found in `third_party/onnx/onnx/backend/test/case/node`. You can identify new tests by looking for the new operator in `test/backend/all_test_names.txt`. Once you have located new tests, you may add the new tests in the `test/backend/inference_backend.py.` Please note to add suffix `_cpu` to the onnx test name. Associated with the test, you can define how to run the tests for the new operator. For example:
 ```
         "test_and2d_cpu": {STATIC_SHAPE:{}, DYNAMIC_SHAPE:{-1:{-1}}, CONSTANT_INPUT:{-1}},
 ```

docs/UpdatingLLVMCommit.md

@@ -2,7 +2,7 @@
 
 # Updating the LLVM commit or MLIR-HLO submodule
 
-ONNX-MLIR depends on `llvm-project` (among various other projects such as `mlir-hlo`). The `llvm-project` dependency is captured in [utils/clone-mlir.sh](clone-mlir.sh). `mlir-hlo` is a submodule found in the `third_party` directory.
+ONNX-MLIR depends on `llvm-project` (among various other projects such as `mlir-hlo`). The `llvm-project` dependency is captured in [../utils/clone-mlir.sh](clone-mlir.sh). `mlir-hlo` is a submodule found in the `third_party` directory.
 
 We plan to update `llvm-project` a couple of times a month in order to keep up-to-date with the advancements made in `mlir`, but also to decrease the complexity of each update. There is currently no plan to update `mlir-hlo` on any given schedule, though for a specific LLVM update it may be necessary to also update the `mlir-hlo` submodule for the build to continue working correctly. This is because `mlir-hlo` itself also has a dependency on `mlir`.
 
@@ -17,7 +17,7 @@ We've started an update rotation that is described [here](https://github.com/onn
 ## What is the update process?
 
 1. **Lookup green commit hashes**: From the Github issue https://github.com/llvm/torch-mlir/issues/1178, find the LLVM and MLIR-HLO green commits for the week when ONNX-MLIR is being updated.
-2. **Update the `llvm-project` commit**: Update the LLVM commit referenced in the source tree to the green commit hash for the LLVM project from Step 1. The current locations that need to be updated are [utils/clone-mlir.sh](clone-mlir.sh), [docs/BuildOnLinuxOSX.md](BuildOnLinuxOSX.md) and  [docs/BuildOnWindows.md](BuildOnWindows.md).
+2. **Update the `llvm-project` commit**: Update the LLVM commit referenced in the source tree to the green commit hash for the LLVM project from Step 1. The current locations that need to be updated are [utils/clone-mlir.sh](../utils/clone-mlir.sh), [docs/BuildOnLinuxOSX.md](BuildOnLinuxOSX.md) and  [docs/BuildOnWindows.md](BuildOnWindows.md).
 3. **Update the `mlir-hlo` submodule**: In the `third-party/mlir-hlo` directory, run `git fetch` followed by `git checkout <mlir-hlo-commit-hash>` (where `<mlir-hlo-commit-hash>` is the green commit hash for the MLIR-HLO project from Step 1).
 4. **Rebuild and test ONNX-MLIR**: This might involve fixing various API breakages introduced upstream (they are likely unrelated to what you are working on).  If these fixes are too complex, please file a work-in-progress PR explaining the issues you are running into asking for help so that someone from the community can help.
 

src/Accelerators/NNPA/Conversion/ONNXToZHigh/ONNXToZHigh.td

@@ -405,7 +405,7 @@ def GetI64ArrayAttrStridesAveragePool: NativeCodeCall<
   "($0.getDefiningOp<ONNXAveragePoolOp>()))">;
 
 def replaceONNXAveragePoolPattern : Pattern<
-  (ONNXAveragePoolOp:$res $x, $_, $_, $_, $_, $_, $_),
+  (ONNXAveragePoolOp:$res $x, $_, $_, $_, $_, $_, $_, $_),
   [
     // Get attributes using shape helper
     (GetStrAttrPaddingtypeAveragePool:$padtype $res),

src/Accelerators/NNPA/Conversion/ONNXToZHigh/RewriteONNXForZHigh.cpp

@@ -29,9 +29,11 @@
 #include "src/Accelerators/NNPA/Pass/NNPAPasses.hpp"
 #include "src/Conversion/ONNXToKrnl/ONNXToKrnlCommon.hpp"
 #include "src/Dialect/ONNX/DialectBuilder.hpp"
+#include "src/Dialect/ONNX/ElementsAttr/WideNum.hpp"
 #include "src/Dialect/ONNX/ONNXDimAnalysis.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"
 #include "src/Dialect/ONNX/ONNXOps/ShapeHelper.hpp"
+#include "src/Dialect/ONNX/OnnxElementsAttrBuilder.hpp"
 #include "src/Support/TypeUtilities.hpp"
 
 using namespace mlir;
@@ -388,6 +390,62 @@ class SplitLargeMatMulPattern : public OpRewritePattern<ONNXMatMulOp> {
   }
 };
 
+/// This pattern is to replace `C = add/sub(A, B)` by `A` when B is a zero
+/// defined by Expand of scalar constant and C's shape is the same as A's shape.
+/// In other words, the output does not depend on the second operand.
+/// This pattern is similar to Add/SubZerosOnRhs in ConstProp.td but allows
+/// dynamic shape.
+template <typename OP_TYPE>
+class AddSubWithRHSZeroExpandPattern : public OpRewritePattern<OP_TYPE> {
+public:
+  DimAnalysis *dimAnalysis;
+
+  AddSubWithRHSZeroExpandPattern(MLIRContext *context, DimAnalysis *dimAnalysis)
+      : OpRewritePattern<OP_TYPE>(context, 1001), dimAnalysis(dimAnalysis) {}
+
+  LogicalResult matchAndRewrite(
+      OP_TYPE binaryOp, PatternRewriter &rewriter) const override {
+    // Match
+    if (!canBeRewritten(binaryOp, dimAnalysis))
+      return failure();
+    // Rewrite
+    rewriter.replaceOp(binaryOp.getOperation(), {binaryOp.getA()});
+    return success();
+  }
+
+  static bool canBeRewritten(OP_TYPE binaryOp, DimAnalysis *dimAnalysis) {
+    Value A = binaryOp.getA();
+    Value B = binaryOp.getB();
+    Value C = binaryOp.getC();
+
+    // Match
+    // C's shape is the same as A'shape.
+    if (!dimAnalysis->sameShape(A, C))
+      return false;
+    // B is a zero defined by Expand.
+    if (isa<BlockArgument>(B))
+      return false;
+    bool BIsZero = false;
+    if (auto expandOp = dyn_cast<ONNXExpandOp>(B.getDefiningOp())) {
+      Value input = expandOp.getInput();
+      if (isDenseONNXConstant(input)) {
+        // Expand's input is 0?
+        ElementsAttr constElements = getElementAttributeFromONNXValue(input);
+        Type elemType = constElements.getElementType();
+        if (!elemType.isInteger(1)) { // Booleans are not supported.
+          WideNum zeroWN = wideZeroDispatch(elemType, [](auto wideZero) {
+            using cpptype = decltype(wideZero);
+            constexpr BType TAG = toBType<cpptype>;
+            return WideNum::widen<TAG>(static_cast<cpptype>(0.0));
+          });
+          BIsZero = ElementsAttrBuilder::allEqual(constElements, zeroWN);
+        }
+      }
+    }
+    return BIsZero;
+  }
+};
+
 //===----------------------------------------------------------------------===//
 // Rewrite ONNX ops to ZHigh ops and ONNX ops for ZHigh.
 //===----------------------------------------------------------------------===//
@@ -441,11 +499,13 @@ void RewriteONNXForZHighPass::runOnOperation() {
   //
   // This is preferred for NNPA because NNPA BinaryOp does not support
   // broadcasting.
-  target.addDynamicallyLegalOp<ONNXAddOp>([](ONNXAddOp op) {
+  target.addDynamicallyLegalOp<ONNXAddOp>([&dimAnalysis](ONNXAddOp op) {
     return !((isDefinedByONNXConstantOp(op.getA()) &&
                  isUniBroadcatableFirstToSecond(op.getA(), op.getB())) ||
              (isDefinedByONNXConstantOp(op.getB()) &&
-                 isUniBroadcatableFirstToSecond(op.getB(), op.getA())));
+                 isUniBroadcatableFirstToSecond(op.getB(), op.getA())) ||
+             AddSubWithRHSZeroExpandPattern<ONNXAddOp>::canBeRewritten(
+                 op, &dimAnalysis));
   });
   target.addDynamicallyLegalOp<ONNXDivOp>([](ONNXDivOp op) {
     return !((isDefinedByONNXConstantOp(op.getA()) &&
@@ -459,11 +519,13 @@ void RewriteONNXForZHighPass::runOnOperation() {
              (isDefinedByONNXConstantOp(op.getB()) &&
                  isUniBroadcatableFirstToSecond(op.getB(), op.getA())));
   });
-  target.addDynamicallyLegalOp<ONNXSubOp>([](ONNXSubOp op) {
+  target.addDynamicallyLegalOp<ONNXSubOp>([&dimAnalysis](ONNXSubOp op) {
     return !((isDefinedByONNXConstantOp(op.getA()) &&
                  isUniBroadcatableFirstToSecond(op.getA(), op.getB())) ||
              (isDefinedByONNXConstantOp(op.getB()) &&
-                 isUniBroadcatableFirstToSecond(op.getB(), op.getA())));
+                 isUniBroadcatableFirstToSecond(op.getB(), op.getA())) ||
+             AddSubWithRHSZeroExpandPattern<ONNXSubOp>::canBeRewritten(
+                 op, &dimAnalysis));
   });
 
   // Determine if MatMulOp is already legal (no need to rewrite) or need to
@@ -536,6 +598,10 @@ void RewriteONNXForZHighPass::runOnOperation() {
   RewritePatternSet patterns(&getContext());
   populateWithGenerated(patterns);
   patterns.insert<SplitLargeMatMulPattern>(&getContext());
+  patterns.insert<AddSubWithRHSZeroExpandPattern<ONNXAddOp>>(
+      &getContext(), &dimAnalysis);
+  patterns.insert<AddSubWithRHSZeroExpandPattern<ONNXSubOp>>(
+      &getContext(), &dimAnalysis);
 
   // With the target and rewrite patterns defined, we can now attempt the
   // conversion. The conversion will signal failure if any of our `illegal`

src/Accelerators/NNPA/Dialect/ZHigh/ZHigh.td

@@ -29,6 +29,7 @@ def ZHigh_Dialect : Dialect {
   let summary = "A high-level dialect for the ONNX NNPA accelerator ISA.";
   let cppNamespace = "::onnx_mlir::zhigh";
   let useDefaultAttributePrinterParser = 1;  
+  let usePropertiesForAttributes = 0;
 }
 
 //===----------------------------------------------------------------------===//

src/Accelerators/NNPA/Dialect/ZLow/ZLow.td

@@ -20,6 +20,7 @@ def ZLow_Dialect : Dialect {
   let name = "zlow";
   let summary = "A low-level dialect for the ONNX NNPA accelerator ISA.";
   let cppNamespace = "::onnx_mlir::zlow";  
+  let usePropertiesForAttributes = 0;
 }
 
 // Base class for ZLow dialect operations. This operation inherits from the

src/Accelerators/NNPA/Transform/ZLow/ZLowRewrite.cpp

@@ -133,6 +133,12 @@ class StickViewUnstickRemovalPattern : public OpRewritePattern<ZLowStickOp> {
       ZLowStickOp stickOp, PatternRewriter &rewriter) const override {
     Value stickInput = stickOp.getX();
 
+    // Do not handle NCHW layout stickification that transposes data
+    // internally.
+    std::string stickLayout = stickOp.getLayout().value().str();
+    if (stickLayout == LAYOUT_NCHW)
+      return failure();
+
     // Input is a block argument, ignore it.
     if (stickInput.dyn_cast<BlockArgument>())
       return failure();
@@ -157,6 +163,11 @@ class StickViewUnstickRemovalPattern : public OpRewritePattern<ZLowStickOp> {
       ZLowUnstickOp userOp = llvm::dyn_cast<ZLowUnstickOp>(user);
       if (!userOp)
         continue;
+      // Do not handle NCHW layout stickification that transposes data
+      // internally.
+      std::string unstickLayout = userOp.getLayout().value().str();
+      if (unstickLayout == LAYOUT_NCHW)
+        continue;
       // UnstickOp must be before the view operation.
       if (userOp.getOut() == viewSource &&
           user->isBeforeInBlock(viewOp.getOperation())) {

src/Builder/CMakeLists.txt

@@ -11,7 +11,6 @@ add_onnx_mlir_library(OMBuilder
   ModelInputShaper.cpp
 
   LINK_LIBS PUBLIC
-  OMCompilerOptions
   OMHasOnnxSubgraphOpInterface
   OMONNXOps
   OMResultTypeInferenceOpInterface

src/Builder/FrontendDialectTransformer.cpp

@@ -31,7 +31,6 @@
 #include "src/Builder/ImportONNXUtils.hpp"
 #include "src/Builder/ModelInputShaper.hpp"
 #include "src/Builder/SymbolTable.hpp"
-#include "src/Compiler/CompilerOptions.hpp"
 #include "src/Dialect/ONNX/DialectBuilder.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"
 #include "src/Dialect/ONNX/ONNXOps/OpHelper.hpp"
@@ -165,7 +164,7 @@ class FrontendGenImpl {
     opset_map_ = GetOpsetImportsFromProto(model); // Which opsets to use.
     in_model_functions_ = GetModelLocalFunctions(model);
     importGraph(model.graph());
-    if (VerboseOutput) {
+    if (options_.verboseOutput) {
       llvm::outs()
           << "The ONNX model has " << num_of_parameters_
           << " elements in its initializers. This value would be close to and "
@@ -1235,7 +1234,10 @@ class FrontendGenImpl {
       }
 
       for (auto &name : functionProto.output()) {
-        outputs.push_back(LookupOnnxName(name));
+        // Skip missing optional outputs: they are not mapped.
+        if (const Value *valuePtr = frontend_symbols_.GetByOnnxName(name)) {
+          outputs.push_back(*valuePtr);
+        }
       }
 
       frontend_symbols_.popScope(scopeName);

src/Builder/FrontendDialectTransformer.hpp

@@ -39,6 +39,7 @@ namespace onnx_mlir {
  * Options to control the translation of an ONNX model to ONNX-MLIR.
  */
 struct ImportOptions {
+  bool verboseOutput = false;
   // Use types/shapes in the input-model for translation (for intermediate
   // variables)
   bool useOnnxModelTypes = false;

src/Builder/OpBuildTable.inc

@@ -18,7 +18,7 @@ op_dialect_version_map_["Asin"] = {7};
 op_dialect_version_map_["Asinh"] = {9};
 op_dialect_version_map_["Atan"] = {7};
 op_dialect_version_map_["Atanh"] = {9};
-op_dialect_version_map_["AveragePool"] = {11};
+op_dialect_version_map_["AveragePool"] = {19};
 op_dialect_version_map_["BatchNormalization"] = {15};
 op_dialect_version_map_["Bernoulli"] = {15};
 op_dialect_version_map_["Binarizer"] = {1};
@@ -157,7 +157,7 @@ op_dialect_version_map_["ReduceSum"] = {13, 11};
 op_dialect_version_map_["ReduceSumSquare"] = {18, 13};
 op_dialect_version_map_["Relu"] = {14};
 op_dialect_version_map_["Reshape"] = {19};
-op_dialect_version_map_["Resize"] = {18, 13, 11, 10};
+op_dialect_version_map_["Resize"] = {19, 13, 11, 10};
 op_dialect_version_map_["ReverseSequence"] = {10};
 op_dialect_version_map_["RoiAlign"] = {16};
 op_dialect_version_map_["Round"] = {11};

src/Compiler/CMakeLists.txt

@@ -68,6 +68,7 @@ add_onnx_mlir_library(OMCompilerDialects
   OMKrnlOps
   OMONNXOps
   MLIRIR
+  MLIROpenMPDialect
   )
 
 add_onnx_mlir_library(OMCompilerPasses

src/Compiler/CompilerPasses.cpp

@@ -90,7 +90,7 @@ void addONNXToMLIRPasses(mlir::PassManager &pm, bool targetCPU) {
     // Dynamic iterate in ONNXOpTransformPass
     pm.addPass(onnx_mlir::createONNXOpTransformPass(onnxOpTransformThreshold,
         onnxOpTransformReport, targetCPU,
-        enableSimdDataLayout && !disableSimdOption));
+        enableSimdDataLayout && !disableSimdOption, enableConvOptPass));
   } else {
     // Statically add extra passes
     for (int i = 0; i < repeatOnnxTransform; i++) {

src/Compiler/CompilerUtils.cpp

@@ -14,6 +14,7 @@
 
 #include "CompilerUtils.hpp"
 
+#include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Parser/Parser.h"
 #include "mlir/Pass/PassManager.h"
@@ -35,6 +36,7 @@
 
 #include "src/Accelerators/Accelerator.hpp"
 #include "src/Builder/FrontendDialectTransformer.hpp"
+#include "src/Builder/ModelInputShaper.hpp"
 #include "src/Compiler/CompilerDialects.hpp"
 #include "src/Compiler/CompilerOptions.hpp"
 #include "src/Compiler/CompilerPasses.hpp"
@@ -183,6 +185,35 @@ static void loadMLIR(std::string inputFilename, mlir::MLIRContext &context,
     llvm::errs() << "Error can't load file " << inputFilename << "\n";
     exit(1);
   }
+
+  // Set shape information if required.
+  // Only set shape if the module has a single function.
+  uint64_t numOfFuncOp = 0;
+  func::FuncOp funcOp;
+  module->walk([&](func::FuncOp f) {
+    funcOp = f;
+    numOfFuncOp++;
+  });
+  if ((numOfFuncOp == 1) && (!shapeInformation.empty())) {
+    ModelInputShaper modelInputShaper_;
+    modelInputShaper_.setShapeInformation(shapeInformation);
+    auto funcType = dyn_cast<FunctionType>(funcOp.getFunctionType());
+    ArrayRef<Type> argTypes = funcType.getInputs();
+    SmallVector<Type, 4> newArgTypes;
+    for (uint64_t i = 0; i < argTypes.size(); ++i) {
+      Type argTy = argTypes[i];
+      // Get user's shape information.
+      argTy = modelInputShaper_.reshape(i, argTy);
+      // Update the arguments.
+      funcOp.getBody().back().getArgument(i).setType(argTy);
+      newArgTypes.emplace_back(argTy);
+    }
+    // Update the function type.
+    FunctionType newType =
+        FunctionType::get(&context, newArgTypes, funcType.getResults());
+    ConversionPatternRewriter rewriter(&context);
+    rewriter.updateRootInPlace(funcOp, [&] { funcOp.setType(newType); });
+  }
 }
 
 // Tailor LLVMIR to add features that cannot be done with MLIR LLVMIR.
@@ -589,6 +620,7 @@ int processInputFile(StringRef inputFilename, mlir::MLIRContext &context,
 
   if (inputIsSTDIN || inputIsONNX || inputIsONNXText || inputIsJSON) {
     ImportOptions options;
+    options.verboseOutput = VerboseOutput;
     options.useOnnxModelTypes = useOnnxModelTypes;
     options.invokeOnnxVersionConverter = invokeOnnxVersionConverter;
     options.shapeInformation = shapeInformation;

src/Conversion/KrnlToAffine/CMakeLists.txt

@@ -13,7 +13,6 @@ add_onnx_mlir_library(OMKrnlToAffine
 
   LINK_LIBS PUBLIC
   OMSpecializedKernelOpInterface
-  OMCompilerOptions
   OMONNXOps
   OMSupport
   MLIRTransforms