array2hexstring This value is not permitted by chosen dtype.

[MatteoAbrate]

Hi everyone, it is me, again, Matteo Abrate Physics Department Torino, i'm struggling with FINN to implement a very simple ConvNet for image recognition, the images I'm trying to classify are 48x48 int8 8that could be generated in the 0-255 range or in the 0-1) images. I tried to develop the net following the cybercecurity example without success because the implementation fail. So i try again now following the cnv_end2end:ecample but I arrive at the end and there is an error regarding dtype. Please help me, i investigated the ONNX models and it appears to be a mix of int8, int32 and uint8 dtype inside the models, i can't understand from where they are defined, the only thing I defined are the Int8Bias in the model. please help me understand why every try of implementation of this simple model is failing, i also have done quantization aware training and i have the tools to modify the range of input of the net from 0-255 or 0-1 or everything i want because i'm generating the dataset for the task. I'm attaching also the ONNX files if someone will take some time to help me out, this work will probably lead to a publication so if someone of you developers of FINN could partecipate and help it will be mentioned.

Matteo Abrate
Researcher Physics UNITO (Italy)

onnx.zip

 from finn.util.basic import make_build_dir
    from finn.util.visualization import showInNetron
    import os
    import torch
    import onnx
    from torch import nn
    from finn.util.test import get_test_model_trained
    from brevitas.export import export_qonnx
    from qonnx.util.cleanup import cleanup as qonnx_cleanup
    from qonnx.core.modelwrapper import ModelWrapper
    from finn.transformation.qonnx.convert_qonnx_to_finn import ConvertQONNXtoFINN
    from qonnx.transformation.infer_shapes import InferShapes
    from qonnx.transformation.fold_constants import FoldConstants
    import numpy as np
    import brevitas.nn as qnn
    from brevitas.nn import QuantReLU, QuantConv2d, QuantLinear, QuantIdentity
    from brevitas.quant import Int8Bias
    from torch.nn import Module
    from qonnx.util.cleanup import cleanup as qonnx_cleanup
    from qonnx.core.modelwrapper import ModelWrapper
    from qonnx.core.datatype import DataType
    from finn.transformation.qonnx.convert_qonnx_to_finn import ConvertQONNXtoFINN
    import torch.nn.functional as F
    from qonnx.transformation.general import GiveReadableTensorNames, GiveUniqueNodeNames, RemoveStaticGraphInputs
    from finn.util.pytorch import ToTensor
    from qonnx.transformation.merge_onnx_models import MergeONNXModels
    from qonnx.core.datatype import DataType
    from qonnx.transformation.insert_topk import InsertTopK
    from qonnx.transformation.infer_datatypes import InferDataTypes

class brevitas_model(Module):
    def __init__(self):
        super(brevitas_model, self).__init__()
        self.quant_inp = qnn.QuantIdentity(bit_width=8, return_quant_tensor=True)
        self.conv1 = qnn.QuantConv2d(in_channels=1, out_channels=10, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu1 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.conv2 = qnn.QuantConv2d(in_channels=10, out_channels=5, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu2 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.conv3 = qnn.QuantConv2d(in_channels=5, out_channels=1, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu3 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.flatten = nn.Flatten()
        self.fc1 = qnn.QuantLinear(144, 72, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu4 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.fc2 = qnn.QuantLinear(72, 72, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu5 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.fc3 = qnn.QuantLinear(72, 1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
      

def forward(self, x): 
    out = self.quant_inp(x)
    out = self.conv1(out)
    out = self.relu1(out)
    out = F.max_pool2d(out, kernel_size=(2, 2), stride=(2, 2))
    out = self.conv2(out)
    out = self.relu2(out)
    out = F.max_pool2d(out, kernel_size=(2, 2), stride=(2, 2))
    out = self.conv3(out)
    out = self.relu3(out)
    out = self.flatten(out)
    out = self.fc1(out)
    out = self.relu4(out)
    out = self.fc2(out)
    out = self.relu5(out)
    out = self.fc3(out)
    return out

model = brevitas_model()
model.load_state_dict(torch.load("./pth/lr_0.001_epochs_100_batch_32_16frame_trace_correction.pth", map_location=torch.device('cpu')))

export_onnx_path ="./onnx/stack_cnn_48x48_export.onnx"

input_t=torch.randn(1, 1, 48, 48)
export_qonnx(model, export_path=export_onnx_path, input_t=input_t)
qonnx_cleanup(export_onnx_path, out_file=export_onnx_path)
model = ModelWrapper(export_onnx_path)
model = model.transform(ConvertQONNXtoFINN())
model = model.transform(InferShapes())
model = model.transform(FoldConstants())
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(RemoveStaticGraphInputs())
model.save("./onnx/stack_cnn_48x48_tidy.onnx")

model = ModelWrapper("./onnx/stack_cnn_48x48_tidy.onnx")
global_inp_name = model.graph.input[0].name
ishape = model.get_tensor_shape(global_inp_name)
# preprocessing: torchvision's ToTensor divides uint8 inputs by 255
totensor_pyt = ToTensor()
chkpt_preproc_name = "./onnx/stack_cnn_16x16_preproc.onnx"
export_qonnx(totensor_pyt, torch.randn(ishape), chkpt_preproc_name)
qonnx_cleanup(chkpt_preproc_name, out_file=chkpt_preproc_name)
pre_model = ModelWrapper(chkpt_preproc_name)
pre_model = pre_model.transform(ConvertQONNXtoFINN())

# join preprocessing and core model
model = model.transform(MergeONNXModels(pre_model))
# add input quantization annotation: UINT8 for all BNN-PYNQ models
global_inp_name = model.graph.input[0].name
model.set_tensor_datatype(global_inp_name, DataType["UINT8"])

# postprocessing: insert Top-1 node at the end
model = model.transform(InsertTopK(k=1))
chkpt_name = "./onnx/stack_cnn_48x48_pre_post.onnx"
# tidy-up again
model = model.transform(InferShapes())
model = model.transform(FoldConstants())
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(InferDataTypes())
model = model.transform(RemoveStaticGraphInputs())
model.save(chkpt_name)

from finn.transformation.streamline import Streamline
from qonnx.transformation.lower_convs_to_matmul import LowerConvsToMatMul
from qonnx.transformation.bipolar_to_xnor import ConvertBipolarMatMulToXnorPopcount
import finn.transformation.streamline.absorb as absorb
from finn.transformation.streamline.reorder import MakeMaxPoolNHWC, MoveScalarLinearPastInvariants
from qonnx.transformation.infer_data_layouts import InferDataLayouts
from qonnx.transformation.general import RemoveUnusedTensors

model = ModelWrapper("./onnx/stack_cnn_48x48_pre_post.onnx")
model = model.transform(MoveScalarLinearPastInvariants())
model = model.transform(Streamline())
model = model.transform(LowerConvsToMatMul())
model = model.transform(MakeMaxPoolNHWC())
model = model.transform(absorb.AbsorbTransposeIntoMultiThreshold())
model = model.transform(ConvertBipolarMatMulToXnorPopcount())
model = model.transform(Streamline())
# absorb final add-mul nodes into TopK
model = model.transform(absorb.AbsorbScalarMulAddIntoTopK())
model = model.transform(InferDataLayouts())
model = model.transform(RemoveUnusedTensors())
model.save("./onnx/stack_cnn_48x48_streamlined.onnx")

from finn.util.basic import pynq_part_map
# change this if you have a different PYNQ board, see list above
pynq_board = "ZCU104"
fpga_part = pynq_part_map[pynq_board]
target_clk_ns = 10

import finn.transformation.fpgadataflow.convert_to_hw_layers as to_hw
from finn.transformation.fpgadataflow.create_dataflow_partition import (
    CreateDataflowPartition,
)
from finn.transformation.move_reshape import RemoveCNVtoFCFlatten
from finn.transformation.fpgadataflow.specialize_layers import SpecializeLayers
from qonnx.custom_op.registry import getCustomOp
from qonnx.transformation.infer_data_layouts import InferDataLayouts

model = ModelWrapper("./onnx/stack_cnn_48x48_streamlined.onnx")
model = model.transform(to_hw.InferBinaryMatrixVectorActivation())
model = model.transform(to_hw.InferQuantizedMatrixVectorActivation())
# TopK to LabelSelect
model = model.transform(to_hw.InferLabelSelectLayer())
# input quantization (if any) to standalone thresholding
model = model.transform(to_hw.InferThresholdingLayer())
model = model.transform(to_hw.InferConvInpGen())
model = model.transform(to_hw.InferStreamingMaxPool())
# get rid of Reshape(-1, 1) operation between hw nodes
model = model.transform(RemoveCNVtoFCFlatten())
# get rid of Tranpose -> Tranpose identity seq
model = model.transform(absorb.AbsorbConsecutiveTransposes())
# infer tensor data layouts
model = model.transform(InferDataLayouts())
parent_model = model.transform(CreateDataflowPartition())
parent_model.save("./onnx/stack_cnn_48x48_parent.onnx")
sdp_node = parent_model.get_nodes_by_op_type("StreamingDataflowPartition")[0]
sdp_node = getCustomOp(sdp_node)
dataflow_model_filename = sdp_node.get_nodeattr("model")
# save the dataflow partition with a different name for easier access
# and specialize the layers to HLS variants
dataflow_model = ModelWrapper(dataflow_model_filename)
dataflow_model = dataflow_model.transform(SpecializeLayers(fpga_part))
dataflow_model.save("./onnx/stack_cnn_48x48_model.onnx")

model = ModelWrapper("./onnx/stack_cnn_48x48_model.onnx")
fc_layers = model.get_nodes_by_op_type("MVAU_hls")


# Each tuple is (PE, SIMD, in_fifo_depth) for a layer
folding = [
    (8, 1, [144]),    # fc1
    (4, 1, [72]),    # fc2
    (1, 1, [1]),     # fc3
]


for fcl, (pe, simd, ififodepth) in zip(fc_layers, folding):
    fcl_inst = getCustomOp(fcl)
    fcl_inst.set_nodeattr("PE", pe)
    fcl_inst.set_nodeattr("SIMD", simd)
    fcl_inst.set_nodeattr("inFIFODepths", ififodepth)

# use same SIMD values for the sliding window operators
swg_layers = model.get_nodes_by_op_type("ConvolutionInputGenerator_rtl")
for i in range(len(swg_layers)):
    swg_inst = getCustomOp(swg_layers[i])
    simd = folding[i][1]
    swg_inst.set_nodeattr("SIMD", simd)

model = model.transform(GiveUniqueNodeNames())
model.save("./onnx/stack_cnn_48x48_folded.onnx")

from finn.transformation.fpgadataflow.make_zynq_proj import ZynqBuild
model = ModelWrapper("./onnx/stack_cnn_48x48_folded.onnx")
model = model.transform(ZynqBuild(platform = pynq_board, period_ns = target_clk_ns))

This is the output:

  File /home/abrate/Desktop/finn/deps/qonnx/src/qonnx/core/modelwrapper.py:140, in ModelWrapper.transform(self, transformation, make_deepcopy, cleanup)
      138 model_was_changed = True
      139 while model_was_changed:
  --> 140     (transformed_model, model_was_changed) = transformation.apply(transformed_model)
      141 if cleanup:
      142     transformed_model.cleanup()
  
  File /home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/make_zynq_proj.py:344, in ZynqBuild.apply(self, model)
      342 kernel_model = kernel_model.transform(GiveUniqueNodeNames(prefix))
      343 kernel_model.save(dataflow_model_filename)
  --> 344 kernel_model = kernel_model.transform(PrepareIP(self.fpga_part, self.period_ns))
      345 kernel_model = kernel_model.transform(HLSSynthIP())
      346 kernel_model = kernel_model.transform(
      347     CreateStitchedIP(self.fpga_part, self.period_ns, sdp_node.onnx_node.name, False)
      348 )
  
  File /home/abrate/Desktop/finn/deps/qonnx/src/qonnx/core/modelwrapper.py:140, in ModelWrapper.transform(self, transformation, make_deepcopy, cleanup)
      138 model_was_changed = True
      139 while model_was_changed:
  --> 140     (transformed_model, model_was_changed) = transformation.apply(transformed_model)
      141 if cleanup:
      142     transformed_model.cleanup()
  
  File /home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/prepare_ip.py:94, in PrepareIP.apply(self, model)
       92 for node in model.graph.node:
       93     if is_hls_node(node) or is_rtl_node(node):
  ---> 94         _codegen_single_node(node, model, self.fpgapart, self.clk)
       95 return (model, False)
  
  File /home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/prepare_ip.py:54, in _codegen_single_node(node, model, fpgapart, clk)
       52     inst.set_nodeattr("code_gen_dir_ipgen", code_gen_dir)
       53     # ensure that there is generated code inside the dir
  ---> 54     inst.code_generation_ipgen(model, fpgapart, clk)
       55 else:
       56     warnings.warn("Using pre-existing code for %s" % node.name)
  
  File /home/abrate/Desktop/finn/src/finn/custom_op/fpgadataflow/rtlbackend.py:57, in RTLBackend.code_generation_ipgen(self, model, fpgapart, clk)
       56 def code_generation_ipgen(self, model, fpgapart, clk):
  ---> 57     self.generate_hdl(model, fpgapart, clk)
  
  File /home/abrate/Desktop/finn/src/finn/custom_op/fpgadataflow/rtl/thresholding_rtl.py:322, in Thresholding_rtl.generate_hdl(self, model, fpgapart, clk)
      320 """Prepare HDL files from templates for synthesis"""
      321 # Generate a dictionary of values to put in RTL template
  --> 322 code_gen_dict = self.prepare_codegen_rtl_values(model)
      324 # Retrieve the destination directory for the final RTL files
      325 code_gen_dir = self.get_nodeattr("code_gen_dir_ipgen")
  
  File /home/abrate/Desktop/finn/src/finn/custom_op/fpgadataflow/rtl/thresholding_rtl.py:197, in Thresholding_rtl.prepare_codegen_rtl_values(self, model)
      195 t_expand = np.expand_dims(thresholds, axis=-1)
      196 bw_hexdigit = roundup_to_integer_multiple(wdt.bitwidth(), 4)
  --> 197 t_packed = pack_innermost_dim_as_hex_string(
      198     t_expand,
      199     wdt,
      200     bw_hexdigit,
      201     prefix="",
      202 )
      204 pe = self.get_nodeattr("PE")
      205 num_channels = self.get_nodeattr("NumChannels")  # number of channels
  
  File /home/abrate/Desktop/finn/src/finn/util/data_packing.py:154, in pack_innermost_dim_as_hex_string(ndarray, dtype, pad_to_nbits, reverse_inner, prefix)
      151 def fun(x):
      152     return array2hexstring(x, dtype, pad_to_nbits, reverse=reverse_inner, prefix=prefix)
  --> 154 return np.apply_along_axis(fun, ndarray.ndim - 1, ndarray)
  
  File /usr/local/lib/python3.10/dist-packages/numpy/lib/shape_base.py:402, in apply_along_axis(func1d, axis, arr, *args, **kwargs)
      400 buff[ind0] = res
      401 for ind in inds:
  --> 402     buff[ind] = asanyarray(func1d(inarr_view[ind], *args, **kwargs))
      404 if not isinstance(res, matrix):
      405     # wrap the array, to preserve subclasses
      406     buff = res.__array_wrap__(buff)
  
  File /home/abrate/Desktop/finn/src/finn/util/data_packing.py:152, in pack_innermost_dim_as_hex_string.<locals>.fun(x)
      151 def fun(x):
  --> 152     return array2hexstring(x, dtype, pad_to_nbits, reverse=reverse_inner, prefix=prefix)
  
  File /home/abrate/Desktop/finn/src/finn/util/data_packing.py:84, in array2hexstring(array, dtype, pad_to_nbits, prefix, reverse)
       81     dtype = DataType["INT" + str(bw)]
       82 for val in array:
       83     # ensure that this value is permitted by chosen dtype
  ---> 84     assert dtype.allowed(val), "This value is not permitted by chosen dtype."
       85     if dtype.is_integer():
       86         if dtype.signed():
  
  AssertionError: This value is not permitted by chosen dtype.

[fpjentzsch]

Hi,

your input quantizer/MultiThreshold converts UINT8 inputs to INT8 for the first conv layer. Thus, its thresholds should fit into a UINT8 representation (same as its input). However, the thresholds look like this:
[0, ... 0, 13, 37, 61, 86, 110, 134, 159, 183, 207, 232, 256, ..., 256]

It contains 256 as a maximum value which does not fit into UINT8. The offending thresholds should be clipped during streamlining, but there remains a bug in FINN which prevents clipping in this case. #1030 fixes this. Can you try with this PR applied? It worked for me on your code provided above.

As a side note, your thresholds for the input quantization contain mostly '0' and '256' values, suggesting that your model only "makes use" of a very small part of the 8 bit value range. If your input is UINT8 image data, this could be a scenario where you don't need any additional input quantization as part of the FINN accelerator at all.

[MatteoAbrate]

Hi Felix, thank you so much for your help, after using the #1030 the code finally built a bitfile.
But i can't manage to make it work, i will explain the problem:
The task i'm giving to this CNN is binary classification 0-1 (but i need the output to be in float32 or similar, i need like 0,8 for example as result) of 1x48x48x1 BN images. The bitfile now is created using the following code:

class brevitas_model(Module):
    def __init__(self):
        super(brevitas_model, self).__init__()
        self.quant_inp = qnn.QuantIdentity(bit_width=8, return_quant_tensor=True)
        self.conv1 = qnn.QuantConv2d(in_channels=1, out_channels=10, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu1 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.conv2 = qnn.QuantConv2d(in_channels=10, out_channels=5, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu2 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.conv3 = qnn.QuantConv2d(in_channels=5, out_channels=1, kernel_size=(3,3), stride=(1,1), padding=(1,1), groups=1, dilation=1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu3 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.flatten = nn.Flatten()
        self.fc1 = qnn.QuantLinear(144, 72, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu4 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.fc2 = qnn.QuantLinear(72, 72, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        self.relu5 = qnn.QuantReLU(bit_width=8, return_quant_tensor=True)
        self.fc3 = qnn.QuantLinear(72, 1, bias=True, weight_bit_width=8, bias_quant=Int8Bias)
        

    def forward(self, x): 
        out = self.quant_inp(x)
        out = self.conv1(out)
        out = self.relu1(out)
        out = F.max_pool2d(out, kernel_size=(2, 2), stride=(2, 2))
        out = self.conv2(out)
        out = self.relu2(out)
        out = F.max_pool2d(out, kernel_size=(2, 2), stride=(2, 2))
        out = self.conv3(out)
        out = self.relu3(out)
        out = self.flatten(out)
        out = self.fc1(out)
        out = self.relu4(out)
        out = self.fc2(out)
        out = self.relu5(out)
        out = self.fc3(out)
        return out

model = brevitas_model()
model.load_state_dict(torch.load("./pth/lr_0.001_epochs_10_batch_32_48x48_scheduler.pth", map_location=torch.device('cpu')))
export_onnx_path ="./onnx/stack_cnn_48x48_export.onnx"
input_t=torch.randn(1, 1, 48, 48)
export_qonnx(model, export_path=export_onnx_path, input_t=input_t)
qonnx_cleanup(export_onnx_path, out_file=export_onnx_path)
model = ModelWrapper(export_onnx_path)
model = model.transform(ConvertQONNXtoFINN())
model = model.transform(InferShapes())
model = model.transform(FoldConstants())
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(RemoveStaticGraphInputs())
model.save("./onnx/stack_cnn_48x48_tidy.onnx")
model = ModelWrapper("./onnx/stack_cnn_48x48_tidy.onnx")

This the output of this cell:
[W NNPACK.cpp:53] Could not initialize NNPACK! Reason: Unsupported hardware.
/home/abrate/Desktop/finn/deps/qonnx/src/qonnx/transformation/extract_conv_bias.py:52: UserWarning: Could not extract bias from node input: "Quant_0_out0"
input: "Quant_1_out0"
input: "Quant_2_out0"
output: "Conv_0_out0"
name: "Conv_0"
op_type: "Conv"
attribute {
  name: "dilations"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides"
  ints: 1
  ints: 1
  type: INTS
}

  warnings.warn(f"Could not extract bias from node {n}")
/home/abrate/Desktop/finn/deps/qonnx/src/qonnx/transformation/extract_conv_bias.py:52: UserWarning: Could not extract bias from node input: "MaxPool_0_out0"
input: "Quant_3_out0"
input: "Quant_4_out0"
output: "Conv_1_out0"
name: "Conv_1"
op_type: "Conv"
attribute {
  name: "dilations"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides"
  ints: 1
  ints: 1
  type: INTS
}

  warnings.warn(f"Could not extract bias from node {n}")
/home/abrate/Desktop/finn/deps/qonnx/src/qonnx/transformation/extract_conv_bias.py:52: UserWarning: Could not extract bias from node input: "MaxPool_1_out0"
input: "Quant_5_out0"
input: "Quant_6_out0"
output: "Conv_2_out0"
name: "Conv_2"
op_type: "Conv"
attribute {
  name: "dilations"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides"
  ints: 1
  ints: 1
  type: INTS
}

  warnings.warn(f"Could not extract bias from node {n}")
/home/abrate/Desktop/finn/deps/qonnx/src/qonnx/transformation/gemm_to_matmul.py:57: UserWarning: The GemmToMatMul transformation only offers explicit support for version 9 of the Gemm node, but the ONNX version of the supplied model is 14. Thus the transformation may fail or return incomplete results.
  warnings.warn(



#The following is commented because i had only 1's as output when i was doing inference and i supposed that the top1k module wasn't necessary.

#global_inp_name = model.graph.input[0].name
#ishape = model.get_tensor_shape(global_inp_name)
# preprocessing: torchvision's ToTensor divides uint8 inputs by 255
#totensor_pyt = ToTensor()
#chkpt_preproc_name = "./onnx/stack_cnn_48x48_preproc.onnx"
#export_qonnx(totensor_pyt, torch.randn(ishape), chkpt_preproc_name)
#qonnx_cleanup(chkpt_preproc_name, out_file=chkpt_preproc_name)
#pre_model = ModelWrapper(chkpt_preproc_name)
#pre_model = pre_model.transform(ConvertQONNXtoFINN())
# join preprocessing and core model
#model = model.transform(MergeONNXModels(pre_model))
# add input quantization annotation: UINT8 for all BNN-PYNQ models
global_inp_name = model.graph.input[0].name
model.set_tensor_datatype(global_inp_name, DataType["UINT8"])
# postprocessing: insert Top-1 node at the end
#model = model.transform(InsertTopK(k=1))
chkpt_name = "./onnx/stack_cnn_48x48_pre_post.onnx"
# tidy-up again
model = model.transform(InferShapes())
model = model.transform(FoldConstants())
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(InferDataTypes())
model = model.transform(RemoveStaticGraphInputs())
model.save(chkpt_name)


from finn.transformation.streamline import Streamline
from qonnx.transformation.lower_convs_to_matmul import LowerConvsToMatMul
from qonnx.transformation.bipolar_to_xnor import ConvertBipolarMatMulToXnorPopcount
import finn.transformation.streamline.absorb as absorb
from finn.transformation.streamline.reorder import MakeMaxPoolNHWC, MoveScalarLinearPastInvariants
from qonnx.transformation.infer_data_layouts import InferDataLayouts
from qonnx.transformation.general import RemoveUnusedTensors
model = ModelWrapper("./onnx/stack_cnn_48x48_pre_post.onnx")
model = model.transform(MoveScalarLinearPastInvariants())
model = model.transform(Streamline())
model = model.transform(LowerConvsToMatMul())
model = model.transform(MakeMaxPoolNHWC())
model = model.transform(absorb.AbsorbTransposeIntoMultiThreshold())
model = model.transform(ConvertBipolarMatMulToXnorPopcount())
model = model.transform(Streamline())
# absorb final add-mul nodes into TopK
#model = model.transform(absorb.AbsorbScalarMulAddIntoTopK())
model = model.transform(InferDataLayouts())
model = model.transform(RemoveUnusedTensors())
model.save("./onnx/stack_cnn_48x48_streamlined.onnx")
from finn.util.basic import pynq_part_map
# change this if you have a different PYNQ board, see list above
pynq_board = "ZCU104"
fpga_part = pynq_part_map[pynq_board]
target_clk_ns = 10

import finn.transformation.fpgadataflow.convert_to_hw_layers as to_hw
from finn.transformation.fpgadataflow.create_dataflow_partition import (
    CreateDataflowPartition,
)
from finn.transformation.move_reshape import RemoveCNVtoFCFlatten
from finn.transformation.fpgadataflow.specialize_layers import SpecializeLayers
from qonnx.custom_op.registry import getCustomOp
from qonnx.transformation.infer_data_layouts import InferDataLayouts

model = ModelWrapper("./onnx/stack_cnn_48x48_streamlined.onnx")
model = model.transform(to_hw.InferBinaryMatrixVectorActivation())
model = model.transform(to_hw.InferQuantizedMatrixVectorActivation())
# TopK to LabelSelect
#model = model.transform(to_hw.InferLabelSelectLayer())
# input quantization (if any) to standalone thresholding
model = model.transform(to_hw.InferThresholdingLayer())
model = model.transform(to_hw.InferConvInpGen())
model = model.transform(to_hw.InferStreamingMaxPool())
# get rid of Reshape(-1, 1) operation between hw nodes
model = model.transform(RemoveCNVtoFCFlatten())
# get rid of Tranpose -> Tranpose identity seq
model = model.transform(absorb.AbsorbConsecutiveTransposes())
# infer tensor data layouts
model = model.transform(InferDataLayouts())
parent_model = model.transform(CreateDataflowPartition())
parent_model.save("./onnx/stack_cnn_48x48_parent.onnx")
sdp_node = parent_model.get_nodes_by_op_type("StreamingDataflowPartition")[0]
sdp_node = getCustomOp(sdp_node)
dataflow_model_filename = sdp_node.get_nodeattr("model")
# save the dataflow partition with a different name for easier access
# and specialize the layers to HLS variants
dataflow_model = ModelWrapper(dataflow_model_filename)
dataflow_model = dataflow_model.transform(SpecializeLayers(fpga_part))
dataflow_model.save("./onnx/stack_cnn_48x48_model.onnx")
model = ModelWrapper("./onnx/stack_cnn_48x48_model.onnx")
fc_layers = model.get_nodes_by_op_type("MVAU_hls")


# Each tuple is (PE, SIMD, in_fifo_depth) for a layer
folding = [
    (8, 1, [144]),    # fc1
    (4, 1, [72]),    # fc2
    (1, 1, [1]),     # fc3
]


for fcl, (pe, simd, ififodepth) in zip(fc_layers, folding):
    fcl_inst = getCustomOp(fcl)
    fcl_inst.set_nodeattr("PE", pe)
    fcl_inst.set_nodeattr("SIMD", simd)
    fcl_inst.set_nodeattr("inFIFODepths", ififodepth)

# use same SIMD values for the sliding window operators
swg_layers = model.get_nodes_by_op_type("ConvolutionInputGenerator_rtl")
for i in range(len(swg_layers)):
    swg_inst = getCustomOp(swg_layers[i])
    simd = folding[i][1]
    swg_inst.set_nodeattr("SIMD", simd)

model = model.transform(GiveUniqueNodeNames())
model.save("./onnx/stack_cnn_48x48_folded.onnx")

from finn.transformation.fpgadataflow.make_zynq_proj import ZynqBuild
model = ModelWrapper("./onnx/stack_cnn_48x48_folded.onnx")
model = model.transform(ZynqBuild(platform = pynq_board, period_ns = target_clk_ns))


This the output of this cell:

/home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/floorplan.py:107: UserWarning: 26 nodes have no entry in the provided floorplan, SLR was set to -1
  warnings.warn(
/home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/insert_fifo.py:234: UserWarning: Input FIFO for IODMA_hls_0_out0 has depth 2 and won't
                        be created. This may cause RTL simulation issues.
                        
  warnings.warn(
/home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/insert_fifo.py:294: UserWarning: Output FIFO for MVAU_rtl_3_out0 has depth 2 and won't
                        be created. This may cause RTL simulation issues.
                        
  warnings.warn(
/home/abrate/Desktop/finn/src/finn/transformation/fpgadataflow/create_stitched_ip.py:290: UserWarning: First node is not StreamingFIFO or IODMA.
                You may experience incorrect stitched-IP rtlsim or hardware
                behavior. It is strongly recommended to insert FIFOs prior to
                calling CreateStitchedIP.
  warnings.warn(
  
  from finn.transformation.fpgadataflow.make_pynq_driver import MakePYNQDriver
model = model.transform(MakePYNQDriver("zynq-iodma"))

model.save("./onnx/stack_cnn_48x48_synth.onnx")

from shutil import copy
from distutils.dir_util import copy_tree

# create directory for deployment files
deployment_dir = make_build_dir(prefix="pynq_deployment_")
model.set_metadata_prop("pynq_deployment_dir", deployment_dir)

# get and copy necessary files
# .bit and .hwh file
bitfile = model.get_metadata_prop("bitfile")
hwh_file = model.get_metadata_prop("hw_handoff")
deploy_files = [bitfile, hwh_file]

for dfile in deploy_files:
    if dfile is not None:
        copy(dfile, deployment_dir)

# driver.py and python libraries
pynq_driver_dir = model.get_metadata_prop("pynq_driver_dir")
copy_tree(pynq_driver_dir, deployment_dir)

So the problem now is this, i flashed the bitfile on my zcu104 and when i try to do inference using this command:

sudo python3 driver.py --exec_mode=execute --batchsize=100 --bitfile=resizer.bit --inputfile=input.npy

input.npy is a 100x48x48x1 tensor of int8 as dtype.
The output that the net gives is always 2.0, the first attempt has also the preprocessing of dividing the inputs by 255 and the topk node at the end but the result in that case was always 0. I don't understand what i'm doing wrong, because if i try to do inference on the same net (not implemented) on pytorch on my pc with the same exact data it works. Is it better to train and implement it on float32 input and then quantize them? I also tried this route but it ends up in incompatibility with the finn dtype it is expecting, now i managed to make it works (as work i intended accept input and generating output.npy, not working properly) but i need to fed to the net int8 inputs.

Please help me further, i also attach onnx models and deploy pkg of this implementation.

onnx-cnv-2.zip
deploy-on-pynq-cnv-int8-2.zip

Thank you so much for your help, it is precious.

Matteo

[fpjentzsch]

If you train on UINT8 inputs (0-255) and also want to run inference with these UINT8 inputs, maybe you could remove the input quantization (self.quant_inp = qnn.QuantIdentity(bit_width=8, return_quant_tensor=True)) from your model entirely.

You would not need the "divide by 255" pre-processing. The TopK node doesn't make sense for binary classification. Both of this you already commented out.

The model.set_tensor_datatype(global_inp_name, DataType["UINT8"]) you'll definitely still need, so that FINN does not treat the input as float32.

Another possibility why your accelerator gives a different output from the original model: If there are layers at the beginning and end of the graph that FINN cannot implement, these will be excluded from the main dataflow partition without an error being thrown. You can see in your "..._parent.onnx" that FINN leaves out a scalar Mul and Add (in addition to the input transpose from NCHW to NHWC). You'll need to apply these operations yourself in software. Disabling the bias on the last layer could make this a bit easier (you won't have to deal with the Add node).

[MatteoAbrate]

Do you mean that i need to add those layers manually through FINN commands during the implementation flow or that i should implement those outside the implementation in the Python code i will run on the FPGA? Where can i find the documentation for doing such operations?
Removing the input quantization (self.quant_inp = qnn.QuantIdentity(bit_width=8, return_quant_tensor=True) doesn't seem duable bacause that layer is needed for the subsequent quantconv2d layer, the only way around is to use a normal conv2d layer as the first layer but i don't think that this will be implemented in FINN.

[fpjentzsch]

Hi @MatteoAbrate, sorry for the delay, I was not available for some time. Did you manage to solve this issue?

For the operations outside the StreamingDataflowPartition in above picture, I meant that they will need to be implemented in software outside of FINN, e.g., in the Python code around the calls to the FINN Example Overlay driver if you are running this on a Zynq SoC.

By the way, we will host a FINN workshop/tutorial next week in Turin: #1144
Since you are also located in Turin, this could be an opportunity to meet in person and discuss your use case :-)