Added prompt based testing of text generation models

[MohitIntel]

What does this PR do?

Enables prompt based tests for text generation models and compares the performance with baseline numbers.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[regisss]

@MohitIntel Thanks for opening this PR as this is truly needed! I think we should measure the throughput and the accuracy in the same test (kind of the same as it is done for training). How long does it take to generate 1024 tokens with this prompt?

[MohitIntel]

@MohitIntel Thanks for opening this PR as this is truly needed! I think we should measure the throughput and the accuracy in the same test (kind of the same as it is done for training). How long does it take to generate 1024 tokens with this prompt?

@regisss , I enabled accuracy check in the latest commit. The total time for running prompt based perf and accuracy tests for both bloomz-7b1 and llama-v2-13b-hf is nearly 26 minutes.

Here is the timing breakdown for generating 1024 tokens:
bloomz-7b1 - bf16 --> 5m:36s
bloomz-7b1 - fp32 --> 8m:19s
llama-2-13b-hf - bf16 --> 4m:39s
llama-2-13b-hf - fp32 --> 7m:12s

[regisss]

@MohitIntel Could we reduce the size of the prompt and the number of tokens to generate? 128 for both could be good enough no?
Asking this because I will duplicate the CI at some point so that it runs on main branches and on stable releases.