Add Model Zoo testing support #2990

attila-dusnoki-htec · 2024-04-23T10:49:31Z

This change adds 2 "Model Zoo" to test:

ONNX Zoo (https://onnx.ai/models/ and https://github.com/onnx/models)
- The script will use the repository, and does automatically the following steps:
  - Downloads the models one-by-one with the provided test sample
  - Unzips and converts everything to test_runners.py format
  - Runs the test
  - Cleans up after the run
  - Repeats for all selected models
Sample Generator with Datasets
- The provided pyhon modul can download models and generated test samples from real datasets for them
- It is extendable with furter datasets and models
- Current support
  - 3 Datasets for image, text and audio
    - ImageNet 2012 Val, SQuAD v1.1, LibriSpeech ASR
  - Models using those datasets
    - ResNet50 v1, ResNet50 v1.5, TIMM MobileNetv3-large, ViT-base-patch16-224, CLIP-ViT-large-patch14
    - BERT-large-uncased, DistilBERT-base-cased-distilled, Roberta-base, GPT-J, Llama2-7b-chat-hf, T5-base, Gemma-2b-it
    - Wav2Vec2-base-960h, Whisper-small-en

A very WIP version. Uses ResNet50 and Imagenet only.

kahmed10 · 2024-06-11T11:27:37Z

tools/model_zoo/test_generator/requirements.txt

+# THE SOFTWARE.
+#
+#####################################################################################
+-f https://repo.radeon.com/rocm/manylinux/rocm-rel-6.0/


can you bump the version for this?

kahmed10 · 2024-06-11T11:30:48Z

tools/model_zoo/onnx_zoo/README.md

+
+## Getting the repository
+
+*Important: make sure to enable git-lfs*


Suggested change

*Important: make sure to enable git-lfs*

> [!IMPORTANT]

> Make sure to enable git-lfs.

kahmed10 · 2024-06-11T11:31:10Z

tools/model_zoo/onnx_zoo/README.md

+
+## Running the tests
+
+*Important: the argument must point to a folder, not a file*


Suggested change

*Important: the argument must point to a folder, not a file*

> [!IMPORTANT]

> The argument must point to a folder, not a file.

kahmed10 · 2024-06-11T11:40:57Z

tools/model_zoo/test_generator/README.md

+./test_models.sh generated/
+```
+
+Note: `generated` is the default output folder, make sure to match `--output-folder-prefix` name


Suggested change

Note: `generated` is the default output folder, make sure to match `--output-folder-prefix` name

> [!NOTE]

> `generated` is the default output folder, make sure to match `--output-folder-prefix` name.

kahmed10 · 2024-06-11T11:42:02Z

tools/model_zoo/test_generator/README.md

+                        Max number of sum-samples generated for decoder models. Use 0 to ignore it. (Only for decoder models)
+```
+
+Note: Some models require permission to access, use `huggingface-cli login`


Suggested change

Note: Some models require permission to access, use `huggingface-cli login`

> [!NOTE]

> Some models require permission to access, use `huggingface-cli login`.

kahmed10 · 2024-06-11T21:59:04Z

tools/model_zoo/test_generator/README.md

+
+## Adding more datasets
+
+The 3 most common usecase are handled:


Suggested change

The 3 most common usecase are handled:

The 3 most common use cases are handled:

tools/model_zoo/test_generator/README.md

kahmed10

From review side, looks good. Would need someone to test and verify it works. @causten

aarushjain29

It is working without any errors. By default, the numpy 2.0 is there but it requires numpy 1.24.1

attila-dusnoki-htec · 2024-06-26T18:32:02Z

It is working without any errors. By default, the numpy 2.0 is there but it requires numpy 1.24.1

Oh right, it was released not long ago.
Probably everything needs to be pinned to a version, not just the critical part like torch.

tools/model_zoo/test_generator/requirements.txt

Co-authored-by: Chris Austen <[email protected]>

(cherry picked from commit 497c277)

attila-dusnoki-htec added 30 commits April 23, 2024 10:22

Basic test dataset generator

5f12b89

A very WIP version. Uses ResNet50 and Imagenet only.

Refactored to classes, extended with resnet/optimum

35efda4

Add more models

e7e70f0

Create session at start and re-use

85018a0

Add SQuAD dataset. Add distilbert and roberta models

96815f5

Cleanup

1e97979

Cleanup pt2

a6bc373

WIP asr dataset

99a828a

Finish asr dataset, add wav2vec model

f7556b2

WIP GPTJ

7c64686

WIP enc-dec monolith

fb326b8

Add decoder step. Enable Whisper

7ae6a76

move files into modules

18a3455

split classes into more files

b67df1e

more refactor, fix imports, add properties

265d85a

Use cached model if exists

ef48f25

update download logic

6f6b986

Enable GPT-J

ee54bd2

add common output folder

942ff9e

Fix imagenet preprocess logic

a518015

Add hybrid model ClipVit

4e5317c

Fix limit logic

b47c811

fix a bug with initial data

0172a6a

Add T5 base model

b364f84

Skip model if something goes wrong

8883d86

Add gemma-2b-it model

1d79631

Add Llama2-7b-chat-hf model

363b2e3

Move text generation logic into a class

59d00fd

Add BERT-large-uncased model

4299a9a

remove incorrect classmethod decorators

adc3e8a

attila-dusnoki-htec added 6 commits May 7, 2024 08:34

text_preprocess_2 fix

540f9ee

Add COCO2017 dataset

9c564ff

Enable StableDiffusionXL model

23913e8

cleanup

e72d4f0

fix log naming

96fcac8

Catch exceptions during creation, not just during download

56f9551

kahmed10 reviewed Jun 11, 2024

View reviewed changes

tools/model_zoo/test_generator/README.md Outdated Show resolved Hide resolved

attila-dusnoki-htec and others added 4 commits June 13, 2024 07:18

Bump torch version

5b7d9f6

Address review comments

c1355b9

Fix missing license issue

6c1348d

Merge branch 'develop' into test_dataset_generator

3b80f79

kahmed10 approved these changes Jun 19, 2024

View reviewed changes

causten requested a review from aarushjain29 June 26, 2024 02:44

aarushjain29 approved these changes Jun 26, 2024

View reviewed changes

causten reviewed Jun 26, 2024

View reviewed changes

tools/model_zoo/test_generator/requirements.txt Outdated Show resolved Hide resolved

aarushjain29 self-requested a review June 26, 2024 20:37

Update tools/model_zoo/test_generator/requirements.txt

77bec3a

Co-authored-by: Chris Austen <[email protected]>

aarushjain29 approved these changes Jul 1, 2024

View reviewed changes

Merge branch 'develop' into test_dataset_generator

41d9678

causten merged commit 497c277 into develop Jul 2, 2024
41 of 44 checks passed

causten deleted the test_dataset_generator branch July 2, 2024 15:52

umangyadav pushed a commit that referenced this pull request Jul 4, 2024

Add Model Zoo testing support (#2990)

a133e79

(cherry picked from commit 497c277)

TedThemistokleous pushed a commit that referenced this pull request Aug 21, 2024

Add Model Zoo testing support (#2990)

9124be1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Model Zoo testing support #2990

Add Model Zoo testing support #2990

attila-dusnoki-htec commented Apr 23, 2024

kahmed10 Jun 11, 2024

kahmed10 Jun 11, 2024

kahmed10 Jun 11, 2024

kahmed10 Jun 11, 2024 •

edited

Loading

kahmed10 Jun 11, 2024 •

edited

Loading

kahmed10 Jun 11, 2024

kahmed10 left a comment •

edited

Loading

aarushjain29 left a comment

attila-dusnoki-htec commented Jun 26, 2024


		## Getting the repository

		Important: make sure to enable git-lfs

	Important: make sure to enable git-lfs
	> [!IMPORTANT]
	> Make sure to enable git-lfs.


		## Running the tests

		Important: the argument must point to a folder, not a file

	Important: the argument must point to a folder, not a file
	> [!IMPORTANT]
	> The argument must point to a folder, not a file.

	Note: `generated` is the default output folder, make sure to match `--output-folder-prefix` name
	> [!NOTE]
	> `generated` is the default output folder, make sure to match `--output-folder-prefix` name.

	Note: Some models require permission to access, use `huggingface-cli login`
	> [!NOTE]
	> Some models require permission to access, use `huggingface-cli login`.


		## Adding more datasets

		The 3 most common usecase are handled:

	The 3 most common usecase are handled:
	The 3 most common use cases are handled:

Add Model Zoo testing support #2990

Add Model Zoo testing support #2990

Conversation

attila-dusnoki-htec commented Apr 23, 2024

kahmed10 Jun 11, 2024

Choose a reason for hiding this comment

kahmed10 Jun 11, 2024

Choose a reason for hiding this comment

kahmed10 Jun 11, 2024

Choose a reason for hiding this comment

kahmed10 Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

kahmed10 Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

kahmed10 Jun 11, 2024

Choose a reason for hiding this comment

kahmed10 left a comment • edited Loading

Choose a reason for hiding this comment

aarushjain29 left a comment

Choose a reason for hiding this comment

attila-dusnoki-htec commented Jun 26, 2024

kahmed10 Jun 11, 2024 •

edited

Loading

kahmed10 Jun 11, 2024 •

edited

Loading

kahmed10 left a comment •

edited

Loading