Roborazzi AI-Powered Image Assertion

[takahirom]

Background

Advancements in AI: AI technologies are becoming increasingly sophisticated and affordable.
Manual Screenshot Testing: Traditional methods require manual verification, limiting scalability.

Problem

Scalability Limitations: Manual checks hinder large-scale screenshot testing.
High Maintenance Costs: Human intervention increases effort and costs.

Proposal

Integrate AI for Image Comparison: Use AI to automate screenshot verification in tests.
Enhance Roborazzi: Add AiOptions to enable AI-based image evaluation.

Current code

https://github.com/takahirom/roborazzi/blob/658bb293e7beb163f5004d1f09987f35f0294064/docs/topics/ai_powered_image_assertion.md

Changes for Existing Projects

We won't introduce any new dependencies for existing projects. I think I'll add the roborazzi-ai module to manage dependencies like Gemini or others, so you don't have to worry about disrupting existing dependencies.

Discussion Points

When We Execute This

The current AI implementation runs only when:

• AiOptions are set
  • Because we can't run without prompts and API keys
• Comparing or verifying tasks
• The image is changed with ImageComparator
  • Because I think we can save a lot of money if we check only when images are changed.

And the test fails only when:

• All above conditions are met
• requiredFulfillmentPercent is set
• It doesn't fulfill
• (These tests fail even if it's a compare task.)

How We Set Prompts

Currently, it is a little difficult to set a prompt because we need to change RoborazziOptions.CompareOptions.AiOptions.prompt.

Things to Do

• Get feedback from someone
• Decide on the name of this feature
• Decide if it is okay to have prompt in RoborazziOptions.CompareOptions. If we create a new function for this, like captureRoboImageWithAI, it will require bigger changes. Alternatively, provide a way to change these options easily for RoborazziOptions.

[takahirom]

TODO:

• Add a test using AiModel.Manual
• remove loadRoboAi()
• Separate roborazzi-ai into roborazzi-ai and roborazzi-ai-gemini

[github-actions]

Snapshot diff report

File name	Image
com.github.takahirom .roborazzi.sample.Ma nualTest.captureWith Ai_compare.png
com.github.takahirom .preview.tests.Previ ewsKt.PreviewWithPro perties2.WITH_SYSTEM UI_WITH_BACKGROUND BG_COLOR_4278190335_ NIGHT_NEXUS_5_WALLPA PER_GREEN_DOMINATED_ compare.png