How to run "onnxruntime_tools.transformers.bert_perf_test" on cloud services

[Matthieu-Tinycoaching]

Hi community,

I have run onnxruntime_tools.transformers.bert_perf_test on my local machine based on an onnx model of a BERT-derived transformer.

I would like to run these performance tests on cloud platforms. I have so built a docker image including both the onnxruntime and onnxruntime_tools libraries and the onnx model of transformer.

How could I based on this docker image run onnxruntime_tools.transformers.bert_perf_test on AWS EC2 ?

Thanks!

[tianleiwu]

The tool is part of onnxruntime package, so you do not need onnxruntime_tools. It can run like

python -m onnxruntime.transformers.bert_perf_test --help

python -m onnxruntime.transformers.bert_perf_test --model path_to_your_model.onnx --batch_size 1 --sequence_length 128 --samples 10 --test_times 100 --use_gpu --intra_op_num_threads 4 --input_ids_name input_ids --segment_ids_name tokent_type_ids --input_mask_name attention_mask

You can follow https://stackoverflow.com/questions/46245844/pass-arguments-to-python-argparse-within-docker-container to run it in docker

[Matthieu-Tinycoaching]

Thanks @tianleiwu but the tool from onnxruntime package lacks the --inclusive --all flags. How could I deal with it without using onnxruntime_tools ?

[tianleiwu]

@Matthieu-Tinycoaching,

When you install onnxruntime 1.6.0, it has --inclusive --all flags. For onnxruntime 1.7.0, openmp is removed from binary, so there is no need to tune openmp related setting and the bert_perf_test tool in 1.7.0 is simplified.