First Dense Layer not created by Finn

[brigilda98]

Hi everyone,

We are trying to transform an encoder model for Pynq-Z2 using Finn framework. However, we have a drawback that we are not able to solve.

We are able to build the bitfile for our model but the first layer is discarded and we cannot understand why. Below the steps we follow to transform it:

1. Model creation and training

(we flatten the features before the first Dense layer)

2. Export to onnx

3. Hardware build

[mmrahorovic]

Hi Brigilda,

Great to hear that you are using FINN for your project and my apologies for the delayed response to your question.

I hope this explanation helps in understanding the observed problem. The reason for the first layer being discarded has to do with the input datatype of the first layer. Since the FINN-backend provides support for quantized layers, each of the ONNX layers must be operating on quantized (integer) tensors. In your definition of the quantized input tensor (input_qt), you have not supplied the bit_width parameter. As a consequence, the FINN-ONNX export from Brevitas does not annotate the input tensor, meaning that the FINN compiler cannot tell whether the input tensor is quantized or not. Hence, the input tensor is assumed to be a floating point tensor and since the input to the first layer is floating point, this layer is discarded.

In more detail:

• You can see here in the export_finn_onnx function that a check is performed on the bit width of the supplied (quantized) input tensor.
• If you would have supplied the bit_width as well as the signed parameter, a quantization annotation would be added (here) to the model description telling the FINN compiler that this input tensor is quantized.
• Since the input tensor is not specified to be a quantized tensor, the FINN transformation responsible for converting the first layer is not triggered as explained above. In particular, you can see here that a check is performed on the idt (input datatype) to see if it is an integer.

In order to solve this, you should add the bit_width parameter to the quantized input tensor definition. In particular, change the line:

• input_qt = QuantTensor(input_t, scale=torch.tensor(scale), signed=False)
  To:
• input_qt = QuantTensor(input_t, scale=torch.tensor(scale), bit_width=torch.tensor(1.0), signed=True)
  or any other type of bit_width you intend to have.

For more examples on the FINN compiler and typical end-to-end flows, you could have a look at our extensive notebooks: https://github.com/Xilinx/finn/tree/main/notebooks/end2end_example.
If you have any other questions, I'd be happy to answer them. I hope this helps you to proceed further.

[mmrahorovic]

I managed to reproduce your error. The error (from my experiments) seems to happen at exporting the RTL as Vivado IP for the StreamingFCLayer_Batch_0 layer, most likely an out-of-memory issue. For example, generating the RTL models took 67 hours and ~125 GB of memory, which explains why your pc froze.

The work-around for this issue is to set default_mem_mode to build_cfg.ComputeEngineMemMode.DECOUPLED.

In short, with default_mem_mode = build_cfg.ComputeEngineMemMode.CONST, the weights are 'baked in' into the StreamingFCLayer_Batch. This causes the long synthesis time and memory consumption. For the decoupled memory mode, the weights get streamed in. A more detailed description of the difference can be found here: https://finn.readthedocs.io/en/latest/internals.html#streamingfclayer-mem-mode.

I hope that this resolves your issue. Please let me know if you have any further problems.

[brigilda98]

I tried setting the default_mem_mode as decoupled but I get this error:

Traceback (most recent call last):
File "/workspace/finn/src/finn/builder/build_dataflow.py", line 166, in build_dataflow_cfg
model = transform_step(model, cfg)
File "/workspace/finn/src/finn/builder/build_dataflow_steps.py", line 450, in step_set_fifo_depths
model = model.transform(
File "/workspace/finn-base/src/finn/core/modelwrapper.py", line 141, in transform
(transformed_model, model_was_changed) = transformation.apply(
File "/workspace/finn/src/finn/transformation/fpgadataflow/set_fifo_depths.py", line 306, in apply
sim = pyverilate_stitched_ip(model)
File "/workspace/finn-base/src/finn/util/pyverilator.py", line 167, in pyverilate_stitched_ip
with open(vivado_stitch_proj_dir + "/all_verilog_srcs.txt", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/finn_dev_mklabs/vivado_stitch_proj_l4h_yvt0/all_verilog_srcs.txt'

[mmrahorovic]

I cannot reproduce that error unfortunately. Are you on the main-branch of FINN?

The steps I did are (using Vivado 2020.1):

• Clone FINN branch
• Create a build.py script (below you can find the one I used -- you'll have to replace final_output_dir)
• From the FINN-repo, run ./run-docker.sh build_custom <path_to_build.py>

Please let me know if this works for you.

Build.py

import torch.nn as nn
import brevitas.nn as qnn
import numpy as np
import torch
from brevitas.quant_tensor import QuantTensor
import brevitas.onnx as bo
from finn.core.modelwrapper import ModelWrapper
import os
import finn.builder.build_dataflow as build
import finn.builder.build_dataflow_config as build_cfg
import brevitas.nn as qnn

input_size = 64*64
hidden = 64
class Autoencoder(nn.Module):
    def __init__(self, input_size, hidden=64):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            qnn.QuantLinear(input_size, 512, weight_bit_width=2, bias=True),
            qnn.QuantReLU(bit_width=2, max_val=1.0),
            qnn.QuantLinear(512, 128, weight_bit_width=2, bias=True),
            qnn.QuantReLU(bit_width=2, max_val=1.0),
            qnn.QuantLinear(128, hidden, weight_bit_width=2, bias=True)
        )
    
    def forward(self, x):
        out = x.view(x.size(0), -1)
        return self.encoder(out)

model = Autoencoder(64*64).to("cpu")
ready_model_filename = "predictive.onnx"
input_shape = (1, 1, 64, 64)
input_a = np.random.randint(0, 255, size=input_shape).astype(np.float32)
scale = 1.0
input_t = torch.from_numpy(input_a * scale).float()
input_qt = QuantTensor(input_t, bit_width=8, scale=torch.tensor(scale), signed=False)
bo.export_finn_onnx(model, export_path=ready_model_filename, input_t=input_qt)

#### Replace <path_to_final_output_dir> ####
final_output_dir = <path_to_final_output_dir>

cfg = build_cfg.DataflowBuildConfig(
    output_dir = final_output_dir,
    mvau_wwidth_max = 80,
    target_fps = 10000,
    synth_clk_period_ns = 10.0,
    board = "Pynq-Z2",
    default_mem_mode = build_cfg.ComputeEngineMemMode.DECOUPLED,
    shell_flow_type = build_cfg.ShellFlowType.VIVADO_ZYNQ,
    generate_outputs=[
    build_cfg.DataflowOutputType.BITFILE,
    build_cfg.DataflowOutputType.PYNQ_DRIVER,
    build_cfg.DataflowOutputType.DEPLOYMENT_PACKAGE,
    ]
)

build.build_dataflow_cfg(ready_model_filename, cfg) 

[brigilda98]

I tried as you said and was still getting the same error. However, I was able to resolve it by following this article: https://support.xilinx.com/s/article/76960?language=en_US

Now it works. Thank you for your support!

[mmrahorovic]

Ah I see, glad to hear you managed to resolve it.

If you run into any other issues, please let us know!