Questions on the AWQ

[suhcrates-web]

I’ve thoroughly read your impressive paper on AWQ, and I have a few questions that came up while reviewing the material.

1. In Equation (2), the formula for deriving (\delta') isn’t explicitly provided. How is (\delta') calculated?

2. If the formula is similar to the traditional method of multiplying (w) by (s), like in (\frac{\text{max}(|w_s|)}{2^{N-1}}), wouldn’t (\delta') always differ from (\delta)?

3-1) Do salient weights use the same (2^{N-1}) integer levels as standard quantized weights?

3-2) If not, why is it necessary to express (w) as (w \cdot s) rather than just using a float value for (w)?

3-3) If salient weights also use (2^{N-1}) integer levels, wouldn’t it be possible to improve the performance of standard quantized weights by multiplying them by (s) as well?

I appreciate your time and look forward to your insights.