Test suite v0.4

[teocns]

Overview

This PR reorganizes the test infrastructure by implementing a comprehensive testing strategy that separates unit and integration tests, with a focus on provider integrations and API functionality testing.

Infrastructure Changes

• Split tests into dedicated unit/ and integration/ directories
• Configure separate CI jobs for unit and integration tests
• Add VCR.py for HTTP interaction recording/replay
• Centralize test fixtures in conftest.py
• Add development tools (pytest-sugar, pdbpp)
• Update pytest configuration for optimal test isolation
• Set up proper test timeouts (5 minutes)

Test Fixtures

• Implement JWT token management
• Add provider response spying capabilities
• Centralize mock_req fixture
• Add session management utilities
• Add package availability control
• Set up VCR ignore hosts and options

Core Functionality Tests

• Improve session handling and teardown
• Enhance async loop lifecycle management
• Add concurrent API request handling tests
• Implement proper singleton cleanup
• Move telemetry tests to unit test directory

Provider Integration Tests

OpenAI

• Basic sync/async completions
• Streaming responses
• Assistants API integration

Mistral

• Sync completions
• Async completions
• Streaming responses

Cohere

• Chat completions (sync)
• Chat completions (async)
• Stream handling
• Instrumentation control

AI21

• Sync completions
• Async completions
• Stream management

Groq

• Basic completions
• Session management
• Stream handling

LlamaStack

• Agent configuration
• Shield management
• Model selection

CrewAI Integration Tests

• Basic Setup
  • Test initialization
  • Test session creation
  • Test auto-end behavior
• Session Management
  • Test crew lifecycle
  • Test task completion tracking
  • Test manual session control
• Agent Monitoring
  • Test single agent tracking
  • Test multi-agent tracking
  • Test task delegation tracking
• Tool Integration
  • Test built-in tools
  • Test custom tools
  • Test tool error handling
• Example Workflows
  • Test job posting flow
  • Test markdown validator
  • Test Instagram post creation

API Server Tests

• Session lifecycle in API context
• Tool recording
• Event validation
• Multi-session handling

Documentation

• Update CONTRIBUTING.md with new test structure
• Document VCR usage and best practices
• Add provider test documentation

Integration

• Adapt and integrate tests from PR Add LLM Integration Tests #603
• Update LangChain handler tests for current API
• Implement remaining provider integration tests

CrewAI Integration Tests

Basic Integration Tests

• Test initialization with CrewAI
  • Verify AgentOps initialization before Crew constructor
  • Test auto_start_session behavior
  • Test skip_auto_end_session parameter
  • Verify proper session creation

Session Management Tests

• Test session lifecycle with CrewAI
  • Verify session starts with Crew initialization
  • Test automatic session ending when tasks complete
  • Verify session state after crew.kickoff()
  • Test manual session ending

Event Recording Tests

• Test LLM event recording
  • Verify LLM calls are properly tracked
  • Test streaming responses tracking
  • Verify async LLM calls tracking
  • Test multiple LLM calls within one session

Multi-Agent Tests

• Test multi-agent scenarios
  • Verify each agent's actions are tracked
  • Test inter-agent communication tracking
  • Verify task delegation and completion tracking
  • Test parallel agent execution monitoring

Tool Usage Tests

• Test tool integration
  • Verify custom tool execution tracking
  • Test built-in tool usage monitoring
  • Verify tool error handling
  • Test tool result recording

Example Workflow Tests

• Test job posting workflow
  • Verify researcher agent tracking
  • Test writer agent monitoring
  • Verify review agent tracking
  • Test complete workflow execution

Error Handling Tests

• Test error scenarios
  • Verify failed task tracking
  • Test exception handling
  • Verify session state after errors
  • Test recovery mechanisms

Integration with Other Tools

• Test CrewAI with other integrations
  • Test OpenAI provider integration
  • Verify LangChain compatibility
  • Test custom tool implementations
  • Verify third-party tool usage

Special Features Tests

• Test CrewAI-specific features
  • Verify task dependency tracking
  • Test sequential vs parallel execution
  • Verify agent role assignments
  • Test custom agent configurations

Known Issues & Mitigations

• Test timing out when running integration and unit tests together
  • Solution: Separated test runs in CI
• Complex patching layers management
  • Solution: Centralized mock configuration
• VCR initialization conflicts
  • Solution: Scoped VCR config to session level

[codecov]

Codecov Report

Attention: Patch coverage is 2.43902% with 40 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
agentops/llms/providers/openai.py	2.43%	40 Missing ⚠️

📢 Thoughts on this report? Let us know!

[teocns]

Some funny behavior with what I believe is the async loop lifecycle management. Tests time out, or maybe there is one particular test making the testing suite time out. Running each individually work, so far. Tried different combos of async event loop configuration without luck

[teocns]

I found out the issue is hit only when vcr is initialized (i.e imported)

Tests keep going until a vcr replay kicks in

I think this has to do with Async httpx

[the-praxs]

So far we have the integration tests for all providers except Llama Stack and the partners (crewAI, Autogen, TaskWeaver, LangChain).

I have Llama Stack configured with Fireworks to make it work on my local machine but that's unreliable. I need to make it work to get the cassette recordings.

Partner integration tests I am working on the comprehensive suite as mentioned above.