Include transcriptions performance test results (#47)

mattermost · Nov 21, 2023 · e8e15bd · e8e15bd
1 parent 815b8d7
commit e8e15bd
Showing 1 changed file with 36 additions and 0 deletions.
diff --git a/docs/performance.md b/docs/performance.md
@@ -30,6 +30,42 @@ On the Mattermost side, it may also be necessary to tune the [`FileSettings.MaxF
 > If a load-balancer or proxy is in front of Mattermost, extra configuration may be necessary. 
 > As an example, `nginx` would likely require `client_max_body_size` to be set accordingly.
 
+### Calls Transcriptions
+
+#### Deployment
+
+- `calls-offloader` `v0.5.0`
+- `calls-transcriber` `v0.1.0`
+- `c6i.2xlarge` EC2 instance
+	- 8vCPU / 16GB RAM
+
+#### Model sizes
+
+The transcriber's model size can be configured through the [Calls plugin](https://docs.mattermost.com/configure/plugins-configuration-settings.html#transcriber-model-size). At this time we support the following [Whisper.cpp](https://huggingface.co/ggerganov/whisper.cpp) models:
+
+| Model | File size | Memory |
+|------:|----------:|-------:|
+| tiny  | 75MB      |~273MB  |
+| base  | 142MB     |~388MB  |
+| small | 466MB     |~852MB  |
+
+#### Benchmarks
+
+| Model | Threads | CPU (avg) | Memory (avg) | Call duration | Processing time |
+|-------|---------|-----------|--------------|---------------|-----------------|
+| tiny  | 1       | 13.5%     | 1.20GB       | 10m           | 2m20s (4.28x)   |
+| base  | 1       | 13.0%     | 1.23GB       | 10m           | 4m45s (2.10x)   |
+| small | 1       | 12.8%     | 1.67GB       | 10m           | 16m50s (0.59x)  |
+| tiny  | 2       | 25.0%     | 1.20GB       | 10m           | 1m17s (7.79x)   |
+| base  | 2       | 25.5%     | 1.27GB       | 10m           | 2m41s (3.73x)   |
+| small | 2       | 25.3%     | 1.68GB       | 10m           | 9m23s (1.07x)   |
+| tiny  | 4       | 49.4%     | 1.20GB       | 10m           | 45s (13.33x)    |
+| base  | 4       | 49.8%     | 1.27GB       | 10m           | 1m32s (6.52x)   |
+| small | 4       | 49.6%     | 1.71GB       | 10m           | 5m27s (1.84x)   |
+| tiny  | 4       | 48.7%     | 1.85GB       | 60m           | 3m38s (16.51x)  |
+| base  | 4       | 49.5%     | 1.70GB       | 60m           | 7m6s (8.45x)    |
+| small | 4       | 50.0%     | 1.99GB       | 60m           | 22m50s (2.63x)  |
+
 ## Scalability
 
 Starting in version `v0.3.2`, this service includes support for horizontal scalability. This can be achieved by adding an HTTP load balancer in front of multiple `calls-offloader` instances, and configuring the [Job Service URL](https://docs.mattermost.com/configure/plugins-configuration-settings.html#job-service-url) setting accordingly to point to the balancer's host.