How MultiThreshold works

[wanglu4042]

Hi, I have a question about how MultiThreshold works, I set the bit width of Con2d to 2, and QuantHardTanh to 8.
The weight is right, they are all 2 bit, but the parameters for QuantHardTanh are hard to understand, QuantHardTanh was also changed
in to a MutiThreshold and two Mul and one Add.
Could you tell how this part works? How are these numbers calculated?



Thanks!!

[fpjentzsch]

Hi,
the Muls and Adds correspond to the quantizer's scale value and zero point and should be merged into the MultiThreshold node by the streamlining transformations in FINN. This is possible because the MultiThreshold can implement any monotonically non-decreasing function. Streamlining simply updates the stored threshold values by absorbing the Mul/Add nodes. You can see the operating principle here:

The threshold initializer in your 2nd picture has shape [8,255]:

• 8 is the number of channels; you can have different scaling factors (and thus thresholds) for each channel if you allow it in Brevitas
• 255 is (2^8)-1 because your output activation bitwidth is 8
• Each of the threshold values is stored in a float container in QONNX, but will be stored in hardware with a bitwidth of a = input bitwidth. This is usually the accumulator width of the preceding MVU, which is determined by FINN depending on input activation width and weight value ranges.

[wanglu4042]

Hi, thank you for your answer!
If I want to realize it on FPGA, can I get the INT8 parameters which work in Hardware?
If I want to compare parameters with threshold on FPGA, I can only use this Threshold parameters in INT8.
Looking forward to your reply!!
Thanks

[fpjentzsch]

I'm not sure I understand what you mean.

You can set the activation bitwidth to any value, but I would generally recommend to stay at or below 4 bits because the current MultiThreshold implementation is not exactly resource efficient beyond that.

If you want to see what the final threshold values (used during synthesis) look like, you'll find a "thresh.h" file in the "code_gen_ipgen_*" temporary build directories of your MVAU/MT layers.

[wanglu4042]

Oh! So the bit_width in QuantRelu and QuantHardTanh is the bit_width of the activation, and the output are still INT8, if I use INT8 as the input.

[fpjentzsch]

Yes, QuantRelu applies the ReLU activation function and quantizes to whatever bit_width you set. QuantHardTanh is just a linear quantization without a non-linear activation function.

[wanglu4042]

Hi, thanks again for your answer!
But I am sorry that I still have some question about how QuantReLU works in Hardware.
What I want to do is to build this net with verilog, so I am trying to figture out how to write the code for the Activation layer.
This is the folded onnx module, and I'm confused that why the input data is changed from [1,64,64] to [1,64,64,25]

And what are the meanings of parameters in the MatrixVectorActivation,

parameters in MultiThreshold_1_param0 are the same as thresh.h, what does [54, 161, 268] mean?
If there are numbers like 40, 159, 200 as input data, what will they be changed to?
And why MultiThreshold_1_param0 is [8,3] ?
Looking forward to your reply!
I hope I made my point clear.

[fpjentzsch]

I guess that you get [1,64,64,25] because 25 is your kernel dimension = kernel_width * kernel_height = 5 * 5.

The MultiThreshold_1_param0 is [8,3] because 8 = number of output channels of the convolution (quantization can be "per-channel", although it is "per-tensor" in your case as all 8 dimensions have the same threshold values). 3 = (2 ^ output_bitwidth) - 1 = (2^2)-1 = 3.

With [54, 161, 268] as thresholds and assuming the zero-point (ActVal) is 0:
Input 40 will become 0
Input 159 will become 1
Input 200 will become 2

[wanglu4042]

Thanks for your reply!
You helps me a lot, thanks again.
A problem is that the zero-point (ActVal) is 0, and the scale is defined as 1? so it will be 0, 1, 2 and 3 after the QuantReLU?
The scale of this layer are 0 1 2 3 because width is set as 2?
Looking forward to your reply!