Skip to content

Commit

Permalink
Big perf improvement on loading.
Browse files Browse the repository at this point in the history
Our OpenAI "is server up" check was retrying (default openai client behaviour), with sleeps between retries! Really slowed things down. Especially since local servers (LiteLLM, etc) are often down if app closed.

Now with retry=0 it's speedy
  • Loading branch information
scosman committed Jan 10, 2025
1 parent 7430759 commit c23ae14
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions app/desktop/studio_server/provider_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -633,6 +633,10 @@ def openai_compatible_providers_load_cache() -> OpenAICompatibleProviderCache |
openai_client = openai.OpenAI(
api_key=api_key,
base_url=base_url,
# Important: max_retries must be 0 for performance.
# It's common for these servers to be down sometimes (could be local app that isn't running)
# OpenAI client will retry a few times, with a sleep in between! Big loading perf hit.
max_retries=0,
)

try:
Expand Down

0 comments on commit c23ae14

Please sign in to comment.