Benchmarking

Repository for AI model benchmarking on accelerator hardware.

Performance Table

Model	Input Size	Batch	Grayskull e75	Grayskull e150	Wormhole n150	Wormhole n300 (single-chip)	Wormhole n300 (dual-chip)
BERT-Large (sen/s)	384	64	83	99	118	x	TBD
T5-Large (tok/s)	64	1	24	31	75	65	TBD
FLAN-T5-Large (tok/s)	64	1	9	25	70	65	TBD
Whisper-Small (tok/s)	30s	1	x	x	9	9	TBD
Falcon-7B (tok/s)	128	32	x	x	72	68	TBD
SD v1-4 (s/img)	512x512	1	x	x	100	50	TBD
ResNet50 (img/s)	3x224x224	256	1116	1404	2893	1066	TBD
VoVNet-V2 (img/s)	3x224x224	128	516	819	1604	1199	TBD
MobileNetV1 (img/s)	3x224x224	128	2463	2912	3110	2283	TBD
MobileNetV2 (img/s)	3x224x224	256	1143	1490	2771	2433	TBD
MobileNetV3 (img/s)	3x224x224	64	1191	1267	1986	1632	TBD
HRNet-V2 (img/s)	3x224x224	128	140	231	324	258	TBD
ViT-Base (img/s)	3x224x224	64	301	363	539	446	TBD
DeiT-Base (img/s)	3x224x224	64	302	362	539	446	TBD
YOLOv5-Small (img/s)	3x320x320	128	276	221	1191	1166	TBD
OpenPose-2D (img/s)	3x224x224	64	885	1311	1260	1195	TBD
U-Net (img/s)	3x256x256	48	223	272	494	399	TBD
Inception-v4 (img/s)	3x224x224	128	371	458	1060	1118	TBD

Setup Instructions

Installation

First, create either a Python virtual environment with PyBuda installed or execute from a Docker container with PyBuda installed.

Installation instructions can be found at Install TT-BUDA.

Next, install the model requirements:

pip install -r requirements.txt

Installing on a System with a GPU

If your system contains a GPU device, you can use the requirements-cuda.txt file to install the correct dependencies.

To install packages, make sure your virtual environment is active.

To add additional CUDA packages

pip install -r requirements-cuda.txt

Environment Setup

Setup Access to HuggingFace Hub

To access the benchmarking datasets, follow these steps to set up your access to the HuggingFace Hub:

1. Create a HuggingFace Account:

   • Visit Hugging Face and create an account if you haven't already.

2. Generate User Access Token:

   • Follow the steps outlined in the HuggingFace Docs - Security Tokens to generate a User Access Token.

3. Install huggingface_hub Library:

   • Install the huggingface_hub library by running:

     pip install huggingface_hub

4. Login to HuggingFace CLI:

   • Login to the HuggingFace CLI using your User Access Token:

     huggingface-cli login

   • Enter your User Access Token when prompted.

5. Validate Setup:

   • Run the following command to verify your login status:

     huggingface-cli whoami

   • If your username is displayed, it means you are successfully logged in.

6. Dataset Access:

   • Visit HuggingFace Datasets - ImageNet-1k and follow the instructions to grant access to the ImageNet-1k dataset.

Validation Steps

After completing the setup process, ensure that everything is working correctly:

1. Verify Hugging Face Hub Login:

   • Run the following command to verify that you are logged in to the Hugging Face Hub:

     huggingface-cli whoami

   • If your username is displayed, it means you are successfully logged in.

2. Check Dataset Access:

   • Visit the HuggingFace Datasets - ImageNet-1k page.
   • Make sure you can view and access the dataset details without any permission errors.

3. Accept Dataset Access (If Required):

   • If you encounter any permission errors while accessing the ImageNet-1k dataset, ensure that you follow the instructions provided on the dataset page to grant access.

Benchmarking Datasets Setup

To set up the three required datasets for running benchmarking tests within this repository, follow these steps for each dataset:

1. HuggingFace datasets: will download into ${HF_HOME}/datasets/ once HuggingFace Hub access is set up.

2. COCO Dataset: You can automatically download the COCO dataset from here. No login is required, and the dataset will be cached in ~/.cache/coco.

   To download the COCO dataset, follow these steps:

   # use another location for MLDATA_DIR if desired, below is default
   # Create the `coco` directory inside the cache directory:
   mkdir -p ~/.cache/mldata/coco
   
   # Navigate to the `coco` directory:
   cd ~/.cache/mldata/coco
   
   # Create the `images` directory:
   mkdir images
   cd images
   
   # Download the COCO validation images:
   wget http://images.cocodataset.org/zips/val2017.zip
   
   # Unzip the downloaded file:
   unzip val2017.zip
   
   # Move back to the `coco` directory:
   cd ..
   
   # Download the COCO train/val annotations:
   wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
   
   # Unzip the downloaded file:
   unzip annotations_trainval2017.zip

3. LGG Segmentation Dataset: must be manually downloaded from https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation, this requires a Kaggle login. The commands below are to extract the downloaded archive into the correct location: ~/.cache/mldata/lgg_segmentation/kaggle_3m.

   # use another location for MLDATA_DIR if desired, below is default;
   # Download and move the downloaded archive and unzip within the llgg_segmentation folder.
   mkdir -p ~/.cache/mldata/lgg_segmentation
   cd ~/.cache/mldata/lgg_segmentation
   # download and move the archive here then unzip
   unzip archive.zip
   # the dataset appears to have two copies that are equivalent, remove the extra one
   rm -r lgg-mri-segmentation

Run Benchmarking

As part of the environment setup, you may need to add the root to PYTHONPATH:

export PYTHONPATH=.

Benchmarking Script

benchmark.py allows easy way to benchmark performance of a support model, while varying configurations and compiler options. The script will measure real end-to-end time on host, starting post-compile at first input pushed, and ending at last output received.

To specify the device to run on ("tt", "cpu", or "cuda"), include the -d argument flag.

The script optionally outputs a .json file with benchmark results and options used to run the test. If the file already exists, the results will be appended, allowing the user to run multiple benchmark back-to-back, like:

python benchmark.py -d tt -m bert -c base --task text_classification --save_output

or

python benchmark.py -d cuda -m bert -c base --task text_classification --save_output

To see which models and configurations are currently supported, run:

benchmark.py --list

Run Examples

You can find example commands for various conditions in the file:

• run_benchmark_tt_perf for TT and run_benchmark_cuda GPU & CPU devices

Contributing

We are excited to move our development to the public, open-source domain. However, we are not adequately staffed to review contributions in an expedient and manageable time frame at this time. In the meantime, please review the contributor's guide for more information about contribution standards.

Communication

If you would like to formally propose a new feature, report a bug, or have issues with permissions, please file through GitHub issues.

Please access the Discord community forum for updates, tips, live troubleshooting support, and more!