Issue with Transforming Transpose and Mul Layers in CNN Model

[Nishith0402]

I am working with a CNN model in ONNX format. However, I am facing an issue with the transformation of a Transpose layer in my CNN model in ONNX format. After applying the streamline and convert_to_hw transformations in FINN, a Transpose layer is automatically added to convert the data format from channels-first (NCHW) to channels-last (NHWC). Even when I modify the ONNX model to use the channels-last (NHWC) format from the beginning, the Transpose layer still gets added during the transformation steps.

Additionally,there is another Transpose layer followed by a Flatten layer, and I was unable to absorb the Transpose layer into the Flatten layer using "AbsorbTransposeIntoFlatten() ".And also I am encountering an issue at the final stage where a Mul layer needs to be converted into a Thresholding layer, but the transformation does not take place as expected.After the convert_to_hw transformation, the create_dataflow_partition function throws an error: "cycle-free graph violated: partition depends on itself."

Initial Transpose Layer:

Transpose layer followed by flatten:

Is there a way to prevent streamline and convert_to_hw from automatically adding this Transpose layer and convert mul layer to thresholding layer at the output?

I’m looking for suggestions on how to manage these transformations and optimize the affected layers more efficiently using FINN.

[fpjentzsch]

Hi,

1. FINN cannot implement the initial Transpose and the input quantization MultiThreshold (I assume it has a floating point input?), so we usually perform them in software before passing the data to the FINN accelerator. You can either remove them manually or leave them in the graph if FINN cannot map them to hw_custom_ops, because FINN will silently cut off unsupported operations at the beginning and end of the graph, while failing with "cycle-free graph violated: partition depends on itself" if unsupported operations sit in between supported operations.
2. For this convolution -> flatten -> dense layer scenario you should try the RemoveCNVtoFCFlatten() transformation. In your case it will probably fail because it requires an MVAU after the Flatten, not a Thresholding. I believe this Thresholding in your graph is redundant, since you already have a Thresholding before the MaxPool. Your first screenshot also shows a series of two redundant Thresholding layers.
3. There is no MultiThreshold layer at the output into which the Mul could be merged. Since it is floating point, you'll need to perform this operation outside of FINN in software or change your model to remove it or add an additional quantizer to the output if possible.

[Nishith0402]

I used quantization aware training using brevitas with input_quant=Int8ActPerTensorFloat in QuantConv2d. So, I assume even giving the input as floating point it will automatically quantize the inputs which I observed when done the verification of onnx model.

The transpose layers are inserted once the step_streamline transformation is applied. The LowCovtoMatMul transformation introduced those transpose and flatten nodes.I checked it's implementation and observed that it converts the pytorch model which is basically the channels first to channels last. And I assume that's the reason for transpose and flatten nodes insertion in th onnx graph. I am not sure why redundant thresholding node is inserted.

For the case of mul node i removed it from the onnx graph node i assume it will work.

[KanishR1]

@fpjentzsch I am @Nishith0402 's teammate in the project. As you suggested, I have modified my CNN. Now for estimating the reports, all the transformations are working. But I observed that the input and output layers are removed when the onnx transformation happens as mentioned by you earlier.

After tidy up transformation my onnx has an input layer

After streamline transformation transpose is introduced in the input. Could you explain why this is happening and suggest any modifications to prevent this.

Also after creating the dataflow partition my input and output layer are removed as shown below: (I suppose it is because of the transpose in the input and add matmul and mul in the output which cause this removal. Any suggestions to prevent this)

So, I don't understand why my input and output layers are removed. Please help us out in understanding why these are happening.

Also, when I try to generate hardware, I get an hls error

[fpjentzsch]

FINN removes all layers at the beginning/end of the graph which it cannot map to a hw backend layer, such as floating point operations or operations with unsupported hyperparameters.

As for the initial Transpose. Normally you want the Transpose here, because you train in PyTorch with the default NCHW data layout but FINN uses NHWC internally. You simply leave the Transpose, it doesn't get implemented by FINN, and you transpose the data yourself before you feed it to the accelerator.

Now if you actually trained/exported in NHWC already, you might need to manually annotate the data layout of the input tensor. By default, FINN will always assume the input is NCHW and you will see the following warning in your build_dataflow.log:

[2022-07-21 09:47:18,588] /home/felixj/WD/finn/deps/qonnx/src/qonnx/transformation/infer_data_layouts.py:119: UserWarning: Assuming 4D input is NCHW
[2022-07-21 09:47:18,588] warnings.warn("Assuming 4D input is NCHW")

[KanishR1]

Finn removes transpose as well as the input conv2d layer. How do I prevent this ?