Skip to content

Commit

Permalink
add link to benchmarking script
Browse files Browse the repository at this point in the history
  • Loading branch information
cpoile committed May 27, 2024
1 parent b58c23f commit d57f3b9
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ Legend:

### Description of the methodology

We used a 10 minute call between 4 participants. The call script can be found in the load testing subpackage of the calls repository (TODO: link when merged). The script includes moments where two participants talk over one another to provide a realistic stress-test of the live captioning system. Each participant's lines were sent to AWS Polly text-to-speech and then the audio was "said" on the call. The call's audio was processed and live captions were sent to all call participants, as it would be on a call between real people. At the moment we do not have a measure for quality or accuracy of the live transcriptions.
We used a 10 minute call between 4 participants. The call script can be found in the load testing subpackage of the calls repository: [script for benchmarking](https://github.com/mattermost/mattermost-plugin-calls/blob/main/lt/scripts/script-for-benchmarking.txt). The script includes moments where two participants talk over one another to provide a realistic stress-test of the live captioning system. Each participant's lines were sent to AWS Polly text-to-speech and then the audio was "said" on the call. The call's audio was processed and live captions were sent to all call participants, as it would be on a call between real people. At the moment we do not have a measure for quality or accuracy of the live transcriptions.
For the simultaneous calls, the second (and third) calls were started in different channels one minute (and two minutes) after the first call. This prevented the "talking over" portions of the call to line-up on every call and create an unrealistic amount of load.

### Summary
Expand Down

0 comments on commit d57f3b9

Please sign in to comment.