Dl/torch experimental statistics #24
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…kit#2059) ### Changes Extended data free int8 weight compression algorithm for OpenVINO backend Example (WeightsModel):  PR to optimum huggingface/optimum-intel#415 ### Reason for changes Optimize the model footprint and performance of large models where the size of weights is relatively larger than the size of activations ### Related tickets 117412 ### Tests `tests/openvino/native/quantization/test_weights_compression.py` swin transformer support verified Results Task: lambada_openai | Model |Metric|Value | |Stderr| |--------------|------|-----:|---|-----:| |dolly-v2-3b_original| ppl |5.0144|± |0.1510| | |acc |0.6297|± |0.0067| |dolly-v2-3b_compressed|ppl |4.9868|± |0.1498| | |acc |0.6313|± |0.0067| |Llama-2-7b-chat-hf_original|ppl |3.2788|± |0.0866| | |acc |0.7058|± |0.0063| |Llama-2-7b-chat-hf_compressed|ppl |3.2856|± |0.0869| | |acc |0.7054|± |0.0064|
### Changes Use num_ranking_processes = 1 for Windows ### Reason for changes num_ranking_processes > 1 leads to the error of nncf.qunatize_with_accuracy_control(...) on Windows: ### Related tickets ref:118321 ### Tests N/A
### Changes Add package constraint `filelock<3.12.3`. ### Reason for changes Installing tensorflow test environment produces the error: ``` ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow 2.12.1 requires typing-extensions<4.6.0,>=3.6.6, but you have typing-extensions 4.7.1 which is incompatible. ``` Starting from `filelock==3.12.3`, it requires `typing-extensions>=4.7.1` so downgrading to `filelock==3.12.2` resolves the issue. This is temporary measure unless we upgrade to TF 2.13 which does not have `typing-extensions<4.6.0` constraint.
### Changes Create a separate module to store metatype groups in one location. ### Reason for changes Ref: 113580 ### Related tickets Ref: 113580 ### Tests N/A
### Changes - Update calibrate.py with custom parameters ### Reason for changes - To support parameters like `smooth_quant_alpha` that affect accuracy ### Related tickets - 105473 ### Tests N/A
### Changes Fixed the package detection to account for cases where backend's `ModuleSpec` origin path can validly contain 'nncf' ### Reason for changes NNCF would report incorrect package availability when the virtual environment has `nncf` in its path, which can actually happen commonplace. ### Related tickets 119563 ### Tests test_frameworks_detected_if_origin_in_nncf
### Changes - Updated MinMax logic to work with quantizers for embedding that may be unified ### Reason for changes - Bugfix ### Related tickets - 119561 ### Tests - Updated graph tests 
### Changes Don't create Smooth Quant algorithm if alpha parameter is negative. ### Reason for changes When running AA with `tune_hyperparams=True` and `smooth_quant_alpha<0`, it fails here https://github.com/openvinotoolkit/nncf/blob/develop/nncf/quantization/algorithms/post_training/algorithm.py#L168 because SQ had been created. This fix makes it so SQ is not created if it won't be used.
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
from
9dd3f85
to
0de0502
Compare
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
from
adf0070
to
4e099c8
Compare
openvinotoolkit#2073) ### Changes * ChannelAlignment algorithm is enabled by default * Biases are added only for operations that are affected by CA algorithm ### Reason for changes * To increase models mertics by using ChannelAlignment algorithm by default ### Related tickets 114328 114583 ### Tests tests/post_training/test_templates/test_channel_alignment.py is updated
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
from
b4aced0
to
967c0a5
Compare
github-actions
bot
added
documentation
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
3 times, most recently
from
681c357
to
79cf566
Compare
### Changes - Added new operation - GroupNormalization ### Reason for changes - Performance degradations that are caused by not correct quantization scheme - New operation support ### Related tickets - 119821 - 119335 ### Tests - TBD
### Changes Disable MaskRCNN and RetinaNet graph tests until ticket 119664 is resolved. The following tests are now excluded: test_compressed_graph.py::TestModelsGraph::test_quantize_network[w_sym_t_a_sym_t-retinanet] test_compressed_graph.py::TestModelsGraph::test_quantize_network[w_sym_t_a_sym_t-mask_rcnn] test_compressed_graph.py::TestModelsGraph::test_quantize_network[w_sym_ch_a_asym_t-retinanet] test_compressed_graph.py::TestModelsGraph::test_quantize_network[w_sym_ch_a_asym_t-mask_rcnn] test_compressed_graph.py::TestModelsGraph::test_magnitude_sparsity_network[retinanet] test_compressed_graph.py::TestModelsGraph::test_magnitude_sparsity_network[mask_rcnn] test_compressed_graph.py::TestModelsGraph::test_rb_sparsity_network[retinanet] test_compressed_graph.py::TestModelsGraph::test_rb_sparsity_network[mask_rcnn] test_compressed_graph.py::TestModelsGraph::test_pruning_network[retinanet] test_compressed_graph.py::test_quantize_outputs[w_sym_t_a_sym_t-retinanet] test_compressed_graph.py::test_quantize_outputs[w_sym_ch_a_asym_t-retinanet]
…toolkit#2118) ### Changes Add message about deprecation of `export_to_onnx_standard_ops` option in NNCFConfig ### Reason Recommended way to export to onnx with QuantizeLinear-DequantizeLinear node pairs is `nncf.strip(quantized_model)`.
openvinotoolkit#2115) ### Changes NNCF should not quantize GRU ops with linear_before_reset set to true, since oneDNN does not support it yet ### Reason for changes To align with POT ### Related bug openvinotoolkit#2105 ### Tests Added `test_ignore_nodes_by_attribues` for OV backend
…oolkit#2123) ### Changes Added Whisper notebook to the list of quantization samples
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
from
e1fbd21
to
e9aaa44
Compare
### Changes - Add marks `nightly` and `weakly` for tests. - Mark sanity tests as `nightly` - Split `test_functions.TestParametrized` to fast for precommit and long for nightly - Time of torch precommit reduced from 60 to 40 mins - Set `xfail` for sanity tests with `--mode train` in case of segment fault. Sporadic segment fault reproduced on torch>=2.0.0 on call `backward` function. ### Related tickets 119128
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
from
884aecb
to
53b2067
Compare
…kit#2119)" (openvinotoolkit#2139) ### Changes This reverts openvinotoolkit#2119 ### Reason for changes It seems that the problem is only reproduced on a single machine. This machine is excluded from test scope. ### Related tickets 119664
### Changes Added `nncf.common.logging.track_progress.track()` method to replace `tqdm` for quantization algorithms bars. This method is an almost exact copy of `rich.progress.track()` method, but with an addition of an iteration counter. This is kind of a hack, but `rich` does not provide a way to add custom `Column` objects to the `track()` method. By default `rich.progress.track()` renders progress bar as  With an addition of customizations this becomes  With this change the quantization pipeline looks like  For iterables without length the progress bar displays as  How it looks in a notebook:  ### Reason for changes - User experience improvement - Avoiding multi-line logs in CI produced by `tqdm`, for example: Statistics collection: 0%| | 0/300 [00:00<?, ?it/s] Statistics collection: 0%| | 1/300 [00:00<01:13, 4.09it/s] Statistics collection: 1%| | 2/300 [00:00<00:50, 5.87it/s] Statistics collection: 1%|▏ | 4/300 [00:00<00:33, 8.79it/s] Statistics collection: 2%|▏ | 6/300 [00:00<00:26, 11.30it/s] Statistics collection: 3%|▎ | 8/300 [00:00<00:24, 11.88it/s]
### Changes - Fixed `pymoo` version. - Added patch for the `IntegerRandomSampling` to avoid errors. ### Reason for changes - Bugfix
### Changes As stated in the title ### Reason for changes We do post-PR merge uploads of coverage reports to codecov, and it seems that codecov will reject reports [older than 12 hours](https://docs.codecov.com/docs/codecov-yaml#expired-reports) at the time of the upload, which does not work with our appoach and makes PRs merged more than 12 hours after the precommit checks passed break the coverage state on the develop. ### Related tickets N/A ### Tests N/A
### Changes - Added LINEAR_SQUEEZE_ARITHMETIC_ACTIVATIONS pattern - Added ONNXGlobalMaxPoolMetatype metatype for ONNX backend ### Reason for changes Fixed FakeQuantize operations placement ### Related tickets ref: 113396 ### Tests N/A
### Changes - Bump release version
### Changes Use mo.convert_model instead of converting via onnx in tests. - Update __version__ to 2.6.0, mo.convert_model contains check `nncf.__version__ >= 2.6.0` - Add dump fp32 models - Disable POT backend for `inception_resnet_v2, levit_128, vgg11` in test_quantize_conformance.py (120012) - Update metrics of test_quantize_conformance.py | Model | 2023.0.1 | 2023.1.0 | Diff | |-----------------------------------------------------|------------|------------|---------| | timm/efficientnet_b0_backend_OLD_TORCH | 0.74976 | 0.75538 | 0.0056 | | timm/efficientnet_b0_BC_backend_POT | 0.76878 | 0.76872 | -0.0001 | | timm/levit_128_backend_ONNX | 0.73082 | 0.73184 | 0.0010 | | timm/mobilenetv2_050_backend_POT | 0.64234 | 0.64362 | 0.0013 | | timm/mobilenetv2_050_BC_backend_POT | 0.64376 | 0.6431 | -0.0007 | | timm/mobilenetv3_small_050_backend_POT | 0.27062 | 0.54104 | 0.2704 | | timm/swin_base_patch4_window7_224_backend_OLD_TORCH | 0.79586 | 0.81376 | 0.0179 | | timm/swin_base_patch4_window7_224_backend_TORCH | 0.78026 | 0.80696 | 0.0267 | ### Reason for changes New api in openvino to convert PT models ### Related tickets 117885
### Changes - Updated OpenVINO version to 23.1
…penvinotoolkit#2146) ### Changes In torch classification sample: `write_metrics` and `mlflow` accuracy logging is called only in main process ### Reason for changes Torch test_compression_training (build 106) failed due to metric collected by `write_metrics` and metric saved in checkpoint with key "best_acc1" were different ### Tests Torch test_compression_training build ~~109~~ 111
### Changes Fix incorrect version of OV ### Reason for changes ### Related tickets N/A ### Tests N/A
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
2 times, most recently
from
ad66ed5
to
00b4dd9
Compare
### Changes - Use multithreading instead of multiprocessing to calculate quantizer ranking score - Rename num_ranking_process to num_ranking_workers ### Reason for changes - Support parallel calculations of quantizer ranking score for Windows - Introducing more general name of parameter of number parallel workers. ### Related tickets ref: 119274 ### Tests N/A
…2101) ### Changes Add support of quantization for OpenVINO models with If operation - https://docs.openvino.ai/2023.0/openvino_docs_ops_infrastructure_If_8.html ### Reason for changes Customer request / Obtain the maximum performance for models with inner subgraphs under If operation ### Related tickets 113826 ### Tests Tested on a model attached to a ticket Add synthetic model with If operation
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
from
f75dbe1
to
bdfd115
Compare
### Changes Use absolute value of avg_total_trace in denominator, as done in [PyHessian:](https://github.com/amirgholami/PyHessian/blob/master/pyhessian/hessian.py#L186C22-L186C22) ### Reason for changes more robust mixed precision algo ### Related tickets resolves openvinotoolkit#2155 ### Tests hawq-related tests
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
3 times, most recently
from
b3fb6b2
to
cca8660
Compare
### Changes - Updated base Python version ### Reason for changes - Drop Python 3.7 support ### Related tickets - 118798
### Changes - Fixed the `astroid` version (`shpinx-autoapi` dependence) ### Reason for changes - The 3.0.0 version breaks our pipeline
### Changes Legacy quantizer storage names in checkpoints were removed ### Reason for changes The format had changed after the NNCFNetwork rework, and the deprecation warning has been in place for two releases now. ### Related tickets 121535 ### Tests test_load_model_state
daniil-lyakhov
force-pushed
the
dl/torch_experimental_statistics
branch
2 times, most recently
from
5e859c2
to
bf9c611
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Reason for changes
Related tickets
Tests