Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[generative_ai] Missed regressions in the code samples because of targeted test runs #12990

Open
Valeriy-Burlaka opened this issue Dec 12, 2024 · 1 comment
Assignees
Labels
api: vertex-ai Issues related to the Vertex AI API. samples Issues that are directly related to samples. triage me I really want to be triaged. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@Valeriy-Burlaka
Copy link
Member

Valeriy-Burlaka commented Dec 12, 2024

Describe the issue

We're starting to miss regressions in the code samples for generative_ai because of migrating the CI to do targeted (per feature directory) test runs. Meaning, when there are no modifications done in a feature directory, the tests will not run in this directory and the issue will silently live there, potentially for a long time.

We need to run the entire test suite regularly to ensure all code samples remain functional.

Examples

  1. function_calling/basic_example.py is not working anymore with Gemini Flash model. I discovered this when updated a different sample in this directory (see the comment).
  2. Two function calling with OpenAI samples — function_calling/chat_function_calling_basic.py & function_calling/chat_function_calling_config.py — are not working anymore. Same story here: I discovered the issue only because I updated a different file in this feature directory. See this comment.
@Valeriy-Burlaka Valeriy-Burlaka added triage me I really want to be triaged. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: vertex-ai Issues related to the Vertex AI API. labels Dec 12, 2024
@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Dec 12, 2024
@Valeriy-Burlaka
Copy link
Member Author

Valeriy-Burlaka commented Dec 13, 2024

Created #12996 to track the FC/OpenAI failure (#2).
The #1 is fixed by #12811 and I think is too trivial to track it in a separate ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: vertex-ai Issues related to the Vertex AI API. samples Issues that are directly related to samples. triage me I really want to be triaged. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

2 participants