Add precision conversion and quantization filters #3059
Conversation
There was a problem hiding this comment.
Why not just add an option for compressions to sft_job?
There was a problem hiding this comment.
good point! updated
There was a problem hiding this comment.
Need some unit tests for model compressor & extractor
| else: | ||
| self.compression_type = compression_type | ||
| # compression constants | ||
| self.FP16_MIN = np.finfo(np.float32).min |
There was a problem hiding this comment.
Shouldn't this depend on source_data_type?
There was a problem hiding this comment.
yes, initiate here to avoid multiple finfo calling
| class ModelCompressor(DXOFilter): | ||
| def __init__( | ||
| self, | ||
| source_data_type="float32", |
There was a problem hiding this comment.
in general, why do we need source_data_type? Can we not automatically determine it?
There was a problem hiding this comment.
removed, also removed its counterpart for decompressor
There was a problem hiding this comment.
I would call this "Decompressor". Extraction is more often used for archives with multiple files. Here, we just use data compression/decompression.
There was a problem hiding this comment.
good idea, changed to "quantizer" and "dequantizer"
Fixes # .
Description
Add 16 and 8 bit quantization filters and apply it to LLM-HF example and experiments
Types of changes
./runtest.sh.