Correct Usage of Brevitas for Quantized Model Inference and Input Tensor Quantization

[GiulioMa]

Hello Brevitas team,

I have been working on a project involving quantized neural network models using Brevitas, and I have encountered some discrepancies between the model outputs and my simulated software implementation. I modified my approach to achieve more consistent results, and I would like to verify if I am using the library correctly. Below, I detail the original code, the modifications I made, and my specific questions.

Original Code

Here is the original code snippet I was using:

import core_FPGA as core
import torch
import numpy as np
from utils import create_input_tensor

# Initialize model and set device
hidden_sizes = [core.N_HIDDEN_1, core.N_HIDDEN_2]
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model = core.Digital_twin(core.N_INPUT, 1, hidden_sizes).to(device)

# Create input tensor
in_float = create_input_tensor(core.N_INPUT, device)

# Evaluate the model
model.eval()
out_brevitas = model(in_float)

# Get scale value
scale = float(out_brevitas.scale[0, 0])

# Extract and convert weights and biases to numpy arrays
W1_sw = model.fc1.quant_weight().int().cpu().numpy().astype(np.int64)
b1_sw = model.fc1.quant_bias().int().cpu().numpy().astype(np.int64).reshape(-1, 1)
W2_sw = model.fc2.quant_weight().int().cpu().numpy().astype(np.int64)
b2_sw = model.fc2.quant_bias().int().cpu().numpy().astype(np.int64).reshape(-1, 1)
W3_sw = model.fc3.quant_weight().int().cpu().numpy().astype(np.int64)
b3_sw = model.fc3.quant_bias().int().cpu().numpy().astype(np.int64).reshape(-1, 1)

# Initialize variables
N_RUNS = 100  # Number of runs for comparison
errors = []

# Matrix Multiplication
def matrix_multiply_sw(A, B):
    return np.dot(A, B).astype(np.int64)  # Ensure result is int64

# Matrix Addition
def matrix_add_sw(A, B):
    return (A + B).astype(np.int64)  # Ensure result is int64

# ReLU Activation
def apply_relu_sw(matrix):
    return np.maximum(matrix, 0).astype(np.int64)  # Ensure result is int64

# Forward Pass
def forward_pass_sw(X, W1, b1, W2, b2, W3, b3):
    # Layer 1
    Z1 = matrix_multiply_sw(W1, X)
    A1 = matrix_add_sw(Z1, b1)
    A1 = apply_relu_sw(A1)
    
    # Layer 2
    Z2 = matrix_multiply_sw(W2, A1)
    A2 = matrix_add_sw(Z2, b2)
    A2 = apply_relu_sw(A2)
    
    # Output Layer
    Y_pred = matrix_multiply_sw(W3, A2)
    Y_pred = matrix_add_sw(Y_pred, b3)
    
    Y_pred_64 = Y_pred.astype(np.int64)

    return Y_pred_64

# Perform multiple runs to compare results
for _ in range(N_RUNS):
    in_float = create_input_tensor(core.N_INPUT, device)
    out_brevitas = model(in_float)
    scale = float(out_brevitas.scale[0, 0])
    
    # Simulated forward pass
    X_sw = in_float.view(-1, 1).cpu().numpy().astype(np.int64)
    Y_pred_64 = forward_pass_sw(X_sw, W1_sw, b1_sw, W2_sw, b2_sw, W3_sw, b3_sw)
    
    # Convert out_brevitas to int
    out_brevitas_int = out_brevitas.int().cpu().numpy().astype(np.int64)

    print(out_brevitas_int, Y_pred_64)
    
    # Calculate the error
    error = np.abs(out_brevitas_int - Y_pred_64)
    errors.append(error)

# Convert list of errors to numpy array for statistical calculations
errors = np.array(errors)

# Calculate max error, average error, and standard deviation of the errors
max_error = np.max(errors)
avg_error = np.mean(errors)
std_error = np.std(errors)

# Print the error metrics
print(f"Max error: {max_error}")
print(f"Average error: {avg_error}")
print(f"Standard deviation of error: {std_error}")

Here is the original implementation of the Digital_twin class:

class Digital_twin(nn.Module):

    def __init__(self, obs_dim, act_dim, hidden_sizes):
        super(Digital_twin, self).__init__()
        self.quant_inp = QuantIdentity(bit_width=14, return_quant_tensor=True)
        self.fc1 = QuantLinear(obs_dim, hidden_sizes[0], bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc1.cache_inference_quant_bias=True
        self.relu1 = QuantReLU(act_quant=None, return_quant_tensor=True)
        self.fc2 = QuantLinear(hidden_sizes[0], hidden_sizes[1], bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc2.cache_inference_quant_bias=True
        self.relu2 = QuantReLU(act_quant=None, return_quant_tensor=True)
        self.fc3 = QuantLinear(hidden_sizes[1], 2 * act_dim, bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc3.cache_inference_quant_bias=True
        
    def forward(self, obs):
        obs = self.quant_inp(obs)
        net_out = self.fc1(obs)
        net_out = self.relu1(net_out)
        net_out = self.fc2(net_out)
        net_out = self.relu2(net_out)
        net_out = self.fc3(net_out)
        return net_out

In this original code, the out_brevitas_int and Y_pred_64 do not match, leading to discrepancies in the error calculations.

Modified Code

To address these discrepancies, I made the following modifications:

from brevitas.quant_tensor import QuantTensor

# Function to create input tensor
def quantize_input_tensor(in_float, scale=32767.0, bit_width=14, zero_point=0.0, training=False):
    int_value = in_float
    quant_value = (int_value - zero_point) * scale
    quant_tensor_input = QuantTensor(
        quant_value,
        scale=torch.tensor(scale),
        zero_point=torch.tensor(zero_point),
        bit_width=torch.tensor(float(bit_width)),
        signed=True,
        training=training)
    return quant_tensor_input

class Digital_twin(nn.Module):

    def __init__(self, obs_dim, act_dim, hidden_sizes):
        super(Digital_twin, self).__init__()
        self.fc1 = QuantLinear(obs_dim, hidden_sizes[0], bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc1.cache_inference_quant_bias = True
        self.relu1 = QuantReLU(act_quant=None, return_quant_tensor=True)
        self.fc2 = QuantLinear(hidden_sizes[0], hidden_sizes[1], bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc2.cache_inference_quant_bias = True
        self.relu2 = QuantReLU(act_quant=None, return_quant_tensor=True)
        self.fc3 = QuantLinear(hidden_sizes[1], 2 * act_dim, bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc3.cache_inference_quant_bias = True
        
    def forward(self, obs):
        obs = quantize_input_tensor(obs, scale=1.0)
        net_out = self.fc1(obs)
        net_out = self.relu1(net_out)
        net_out = self.fc2(net_out)
        net_out = self.relu2(net_out)
        net_out = self.fc3(net_out)
        return net_out

In the modified code, the quantize_input_tensor function is used to quantize the input tensor before passing it to the model.

Questions

1. Library Usage:

   • Am I using the QuantLinear and QuantReLU layers correctly in conjunction with the QuantTensor for input quantization?

2. Input Tensor Quantization:

   • Is the quantize_input_tensor function correctly implemented to quantize input tensors before passing them to the Brevitas model?
   • Are there any additional steps or considerations I should take into account to ensure accurate input quantization?
   • Does this implementation imply that scale is constant as I fix it in quantize_input_tensor? how can I properly tune scale?

3. Handling Data Types:

   • I had to modify the library so that the value returned from .int() is int64 to avoid overflow issues. Is this an appropriate modification, or is there a better way to handle potential overflow in quantized models?

Why do I not get the same results from Brevitas and the software version when using QuantIdentity, while I do when using quantize_input_tensor?

Thank you for your time and assistance.

Best regards,
Giulio.

[Giuseppe5]

Hey,
Thanks for the detailed description.
There are a few things going on, so I'm going to give a few suggestion (a bit in random order) to try to simplify the problem and reduce the surface area for errors. More complexity can be added later on I guess.

• To quantize an input, we have a QuantIdentity activation function, which has a similar interface to QuantReLU, but only applies quantization
• Passing act_quant=None to a quantized activation function (e.g., QuantReLU) means that no quantization will be performed, and that would be equivalent to a normal ReLU
• I would recommend starting with less layers (maybe just input, conv, relu?) with no bias quantization, and then slowly adding elements back in
• There are numerical discrepancies between the output of the model and its integer executed counterpart. This is mostly due to floating point arithmetics that can lead to different roundings. These errors are generally very small though, but non zero.
• Brevitas supports many different ways to compute scales. For weights, the default way is based on statistics on the weights themselves. For activations, it is expected that some sort of calibration/training procedure is used before using/exporting the scales.
• The output of int() for a properly quantized model shouldn't overflow on its own. If you want to combine it with something else, I see the point but that's not what Brevitas was designed for

[GiulioMa]

Hi Giuseppe, thank you for your response.

I have a couple of follow-up questions based on your feedback:

1. Input Quantization Discrepancy:
   I've noticed a significant difference in results when using my custom quantizer (quantize_input_tensor) versus using QuantIdentity for input quantization. With my custom quantizer, I get results that closely match my FPGA implementation (with only minor discrepancies). However, when using QuantIdentity, the results are substantially different. Could you help me understand why this might be happening? Are there specific parameters or configurations for QuantIdentity that I should be using to achieve results more consistent with my FPGA implementation?

2. Integer Overflow and Model Output:
   I encountered an issue where calling .int() on the model output was resulting in the maximum value representable by int32, due to the large values (float/scale) involved. This led me to modify the library to use int64. However, I'm concerned this might not be the correct approach. Is there a recommended way to handle this situation within the Brevitas framework?

   Additionally, I've noticed that the output values of my model keep increasing from one layer to another, and so do the bitwidths. Is this expected behavior, or should I be implementing some form of normalization or scaling between layers to prevent this escalation?

Best regards,
Giulio

[Giuseppe5]

Would you be able to create a simple script that shows the discrepancies/overflow that you're mentioning?

With respect to output values, usually batch norm for convolutional networks is a common choice for normalization.

[Giuseppe5]

As a side note, since you're working on deploying your networks on FPGA, I might recommend taking a look at the FINN project if you haven't already. It can consume directly Brevitas-quantized networks and there are tutorials to show how to do so!

[GiulioMa]

Would you be able to create a simple script that shows the discrepancies/overflow that you're mentioning?

Hello Giuseppe,

Thank you for your response. I've created a script that demonstrates the overflow issue and the discrepancies between the Brevitas model and a software-only implementation:

import torch
import torch.nn as nn
import numpy as np
from brevitas.nn import QuantIdentity, QuantLinear, QuantReLU
from brevitas.quant import Int8WeightPerTensorFloat, Int8Bias
from brevitas.quant_tensor import QuantTensor
import brevitas

# Print Brevitas version
print(f"Brevitas version: {brevitas.__version__}")

# Constants
N_INPUT = 2
N_HIDDEN_1 = 64
N_HIDDEN_2 = 80
SCALE = 1/32000.

def quantize_input_tensor(in_float, scale, bit_width=16, zero_point=0.0, training=False, device=torch.device('cpu')):
    int_value = in_float / scale
    quant_value = (int_value - zero_point) * scale
    quant_tensor_input = QuantTensor(
        quant_value,
        scale=torch.tensor(scale),
        zero_point=torch.tensor(zero_point),
        bit_width=torch.tensor(float(bit_width)),
        signed=True,
        training=training).to(device)
    return quant_tensor_input

class Digital_twin(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden_sizes, scale, device):
        super(Digital_twin, self).__init__()
        self.fc1 = QuantLinear(obs_dim, hidden_sizes[0], bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc1.cache_inference_quant_bias=True
        self.relu1 = QuantReLU(act_quant=None, return_quant_tensor=True)
        self.fc2 = QuantLinear(hidden_sizes[0], hidden_sizes[1], bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc2.cache_inference_quant_bias=True
        self.relu2 = QuantReLU(act_quant=None, return_quant_tensor=True)
        self.fc3 = QuantLinear(hidden_sizes[1], 2 * act_dim, bias=True, 
                               weight_quant=Int8WeightPerTensorFloat, bias_quant=Int8Bias, return_quant_tensor=True)
        self.fc3.cache_inference_quant_bias=True
        self.device = device
        self.scale = scale
        
    def forward(self, obs):
        obs = quantize_input_tensor(obs, self.scale, training=self.training, device=self.device)
        net_out = self.fc1(obs)
        net_out = self.relu1(net_out)
        net_out = self.fc2(net_out)
        net_out = self.relu2(net_out)
        net_out = self.fc3(net_out)
        return net_out

def create_input_tensor(dim, scale, device):
    random_numbers = np.random.uniform(low=10., high=16000., size=(1, dim)).astype(np.int16)    
    tensor = torch.tensor(random_numbers, dtype=torch.float32, device=device)
    return tensor * scale

# Software-only forward pass
def software_forward(X, W1, b1, W2, b2, W3, b3):
    def relu(x):
        return np.maximum(x, 0)
    
    A1 = relu(np.dot(W1, X.T) + b1)
    A2 = relu(np.dot(W2, A1) + b2)
    Y = np.dot(W3, A2) + b3
    return Y.T

# Main execution
if __name__ == "__main__":
    device = torch.device('cpu')
    hidden_sizes = [N_HIDDEN_1, N_HIDDEN_2]
    model = Digital_twin(N_INPUT, 1, hidden_sizes, SCALE, device).to(device)
    
    # Set model to evaluation mode
    model.eval()
    
    # Create input tensor
    in_float = create_input_tensor(N_INPUT, SCALE, device)
    
    print("Input tensor:")
    print(in_float)
    
    # Forward pass
    with torch.no_grad():
        out_brevitas = model(in_float)
    
    print("\nModel output (before int conversion):")
    print(out_brevitas.value)
    
    # Int32 conversion (will overflow)
    out_brevitas_int32 = out_brevitas.int()
    print("\nModel output (after int32 conversion):")
    print(out_brevitas_int32)
    
    # Int64 conversion
    out_brevitas_int64 = (out_brevitas.value / out_brevitas.scale[0,0]).to(torch.int64)
    print("\nModel output (after int64 conversion):")
    print(out_brevitas_int64)
    
    # Software-only forward pass
    W1 = model.fc1.quant_weight().int().cpu().numpy().astype(np.int64)
    b1 = model.fc1.quant_bias().int().cpu().numpy().astype(np.int64).reshape(-1, 1)
    W2 = model.fc2.quant_weight().int().cpu().numpy().astype(np.int64)
    b2 = model.fc2.quant_bias().int().cpu().numpy().astype(np.int64).reshape(-1, 1)
    W3 = model.fc3.quant_weight().int().cpu().numpy().astype(np.int64)
    b3 = model.fc3.quant_bias().int().cpu().numpy().astype(np.int64).reshape(-1, 1)
    
    X_sw = (in_float / SCALE).cpu().numpy().astype(np.int64)
    Y_sw = software_forward(X_sw, W1, b1, W2, b2, W3, b3)
    
    print("\nSoftware-only output:")
    print(Y_sw)
    
    # Compare results with int64
    print("\nDifference between int64 Brevitas and software-only:")
    diff_int64 = out_brevitas_int64.cpu().numpy() - Y_sw
    print(diff_int64)
    print(f"Max absolute difference (int64): {np.abs(diff_int64).max()}")
    
    # Compare results with int32
    print("\nDifference between int32 Brevitas and software-only:")
    diff_int32 = out_brevitas_int32.cpu().numpy() - Y_sw
    print(diff_int32)
    print(f"Max absolute difference (int32): {np.abs(diff_int32).max()}")

Here's the output I get:

Brevitas version: 0.10.3
Input tensor:
tensor([[0.1065, 0.0006]])

Model output (before int conversion):
tensor([[ 0.0024, -0.0016]])

Model output (after int32 conversion):
tensor([[-2147483648, -2147483648]], dtype=torch.int32)

Model output (after int64 conversion):
tensor([[ 16148918272, -10562889728]])

Software-only output:
[[ 16148916683 -10562888756]]

Difference between int64 Brevitas and software-only:
[[1589 -972]]
Max absolute difference (int64): 1589

Difference between int32 Brevitas and software-only:
[[-18296400331 8415405108]]
Max absolute difference (int32): 18296400331

[Giuseppe5]

The reason why you see overflow is because as mentioned earlier, QuantReLU(act_quant=None, return_quant_tensor=True) will not requantize your tensor, but simply apply ReLU on the output of the convolution. I understand that this might be the desired behaviour, but it comes with a cost.

If you modify the forward pass of the model to:

    def forward(self, obs):
        obs = quantize_input_tensor(obs, self.scale, training=self.training, device=self.device)
        net_out = self.fc1(obs)
        print(net_out.bit_width)
        net_out = self.relu1(net_out)
        net_out = self.fc2(net_out)
        print(net_out.bit_width)
        net_out = self.relu2(net_out)
        net_out = self.fc3(net_out)
        print(net_out.bit_width)
        return net_out

The output will be:

tensor(26.)                                                                                                                                                                                                                           
tensor(41.)                                                                                                                                                                                                                           
tensor(57.) 

In Brevitas, if the input/weights/bias of a QuantConvolutions are all quantized (and respect certain constraints) we compute the worst case scenario bit-width required to store the output of the operation. Since there is never any re-quantization step to bring down the bitwidth after the convolution, it keeps growing up and then overflowing. When dealing with FPGAs implementation, we try to avoid this behaviour as it reduces the advantages of quantization.

Let me know if this is a desired behaviour, before we continue with the other issues.
A few extra comments I might need help with:

• Why is SCALE = 1/32000.?
• Why the input tensor has values between 10 and 16000?
• Do you need your input tensor to be quantized with a constant scale factor? Quantized activation functions (QuantIdentity, QuantReLU), when correctly initialized, offer several possibilities for picking the scale factor, including setting a constant one. However, that could be not the best one based on the data range of the inputs you have

[GiulioMa]

Dear Giuseppe,

I apologize for the delayed response; the holiday season intervened. Thank you for your explanation.
I'd like to provide some clarifications and seek your advice on a few points:

1. QuantReLU and Requantization:
   I intentionally disabled requantization in ReLU to mirror my FPGA implementation, where I simply set negative values to 0. This approach maintains consistency between the Brevitas model and the FPGA implementation, which uses an increasing bitwidth similar to Brevitas' progression. Are there better alternatives?

2. Concerning the SCALE value:
   The scale value of 1/32000 was indeed chosen as an example for initial testing. In my actual implementation, I've looked for the scale value that makes the output distribution a standard normal distribution (μ = 0, σ = 1).

3. Input Tensor Range:
   The input tensor values between 10 and 16000 reflect the range of my specific sensor data.

4. Constant Scale Factor for Input Quantization:
   I've been using a constant scale factor because it yields results consistent with both the FPGA implementation and the 64-bit integer version. However, I'm open to exploring more sophisticated quantization techniques.

I'm particularly interested in maintaining consistency between the Brevitas model and the FPGA implementation while optimizing for performance and accuracy. Any suggestions you have for achieving this balance would be greatly appreciated.

Thank you for your continued support and expertise.

Best regards,
Giulio.

[Giuseppe5]

I think the better alternative would be to have your FPGA implementation to requantize the values to a lower bit-width to avoid overflow issues.
Although I am aware of some of the challenges that can arise when dealing with a FPGA quantized implementation, I don't have a lot of first-hand experience on that, so I can't help much.

What I can say is that usually FPGA implementations rely on low precision bit-width computations, with more or less clever tricks to compensate for the overhead added by the requantization steps (I think I already mentioned FINN, that uses some of these techniques).
Similarly, it is generally not recommended to use static scale factors unless you are dealing with some specific use cases, otherwise the quantization accuracy might resent of that.

In general, my experience with these tasks is to approach the problem differently, designing first a quantized network with good accuracy (keeping in mind a few good-to-have FPGA constraints, e.g., the requantization step to avoid overflow as I mentioned earlier), and then implement that design on FPGA, and having it match your floating point version. Usually some back and forth is necessary to mantain accuracy and optimize your FPGA implementation.

[GiulioMa]

Hi Giuseppe,

Thanks for the reply.

I understand that FINN would generally help for FPGA implementations. My case though differs from the regular workflow as I need to periodically update the weights (the FPGA communicates via eth to an external server for that).

Also, while I fix the scale for the input quantizer, I see the scale of the output layer changing during training (this is also a parameter I transfer to the FPGA). Finally, not sure about other implementations, but in my case I get a max error (compared to the exact 64bit integer implementation) of 0.01%.

[khoaguin]

I tried the code in the above snippet #995 (reply in thread) using QuantReLU(return_quant_tensor=True). The print(net_out.bit_width) prints out values

tensor(26.)
tensor(23.)
tensor(24.)

However I still got the differences

Software-only output:
[[ 28725415519 158969674579]]

Difference between int64 Brevitas and software-only:
[[ -28725414251 -158969667866]]
Max absolute difference (int64): 158969667866

Difference between int32 Brevitas and software-only:
[[ -28725414251 -158969667866]]
Max absolute difference (int32): 158969667866

[Giuseppe5]

The output of the network will have a larger bit-width as it is the output of a linear layer. Matrix multiplication between two int8 values requires a higher precision accumulation. The bit-width that you see represents a worst case estimation, and if you need tighter bounds, Brevitas supports accumulation aware quantization:
https://arxiv.org/abs/2308.13504

With respect to the difference, I'm not quite sure if the two models you're trying to compare are actually equivalent;
I would start with a slightly simpler example.