Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add retry limits and improve error handling for JSON extraction #162

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

erkinalp
Copy link

Add comprehensive rate limit handling across API providers

This PR implements robust rate limit handling across all API providers used in the AI-Scientist framework, addressing the continuous retry issue (#155).

Changes

  • Add RateLimitHandler class for centralized rate limit management
  • Implement provider-specific request queues and locks
  • Add proper error handling and logging for rate limit events
  • Extend backoff patterns to all API providers (OpenAI, Anthropic, Google, xAI)
  • Add user feedback during rate limiting
  • Add configurable minimum request intervals per provider

Implementation Details

  • Created new rate_limit.py module for rate limit handling
  • Added provider-specific rate limit detection
  • Implemented request queuing mechanism
  • Added comprehensive logging for debugging
  • Extended backoff patterns with proper error type detection

Testing

The changes have been tested by:

  • Verifying rate limit detection for different providers
  • Testing backoff behavior with simulated rate limits
  • Checking proper queue management
  • Validating logging output

Impact

These changes make the system more robust by:

  • Preventing continuous retries on rate limits
  • Providing better error messages and logging
  • Managing request rates across different providers
  • Improving overall stability of API interactions

Fixes #155

Link to Devin run: https://app.devin.ai/sessions/2ec43d6fe7a84849a348753167e5a895

- Add MAX_JSON_RETRIES constant to limit retries
- Enhance error messages in extract_json_between_markers
- Implement retry limits in generate_ideas.py and perform_writeup.py
- Prevent infinite loops when JSON extraction fails

Fixes SakanaAI#154

Co-Authored-By: Erkin Alp Güney <[email protected]>
@conglu1997
Copy link
Collaborator

Hey, thanks for your contributions! I am unable to see your Devin run right now without paying for access first, any way around this?

@erkinalp
Copy link
Author

I am unable to see your Devin run right now without paying for access first, any way around this?

Sadly none yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rate_limit
2 participants