OpenAI Batch Requests

[Clashsoft]

Batch requests (https://platform.openai.com/docs/api-reference/batch) are a new OpenAI API feature. They will allow us to avoid rate limits and reduce token costs, at the expense of up to 24 hours of waiting time. For some assignments this might be acceptable, so we can add a toggle to allow the use of the batch API.

Currently only /v1/chat/completions is supported.

This is on hold however, until embeddings are supported as well.

Embeddings are supported as of 2024-05.