Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Introduce CLI tests to ensure regression is caught in CI #401

Open
12 tasks
c12i opened this issue Oct 23, 2024 · 0 comments
Open
12 tasks

chore: Introduce CLI tests to ensure regression is caught in CI #401

c12i opened this issue Oct 23, 2024 · 0 comments

Comments

@c12i
Copy link
Collaborator

c12i commented Oct 23, 2024

Currently, the scaffolding CLI testing relies heavily on manual processes, which has led to several challenges:

  1. Issues with file/directory generation are only caught through manual testing
  2. The testing process is time-consuming
  3. Edge cases may be missed due to the manual nature of testing
  4. No automated verification of scaffolding output

Current Test Coverage Limitations

While we have integration tests using tryorama that verify the generated hApp's zome functions work as expected, these tests do not verify the CLI's code generation functionality. This has led to issues:

Recent example: The ui directory was completely omitted due to Crane lib filtering out .hbs files, but existing tests didn't catch this
Current tests focus on zome function behavior but don't verify the actual scaffolding output
No automated verification of template processing and file generation

Proposed Solution

Implement automated CLI tests in our CI pipeline to:

  • Verify correct file/directory generation
  • Ensure scaffolding creates expected structure
  • Validate command outputs
  • Catch regressions early

Scope

Phase 1: Basic CLI Testing

  • Set up CLI testing framework
  • Implement tests for basic file/directory generation
  • Add assertions for common scaffolding patterns
  • Verify command execution results
  • Add tests for error cases and edge scenarios

Phase 2 (Optional): UI Snapshot Testing

  • Implement snapshot testing for generated UI components
  • Set up snapshot comparison in CI

Technical Details

Test Coverage Should Include

  • Directory structure validation
  • File content verification
  • Command execution flows
  • Error handling scenarios

Example Test Cases

- Verify scaffold command creates expected directory structure
- Ensure config files are generated in correct locations
- Validate file permissions and contents
- Test command option combinations
- Verify error messages for invalid inputs

Success Criteria

  • All CLI commands have automated tests
  • CI pipeline includes CLI testing
  • Reduced time spent on manual testing
  • Improved confidence in CLI functionality
  • Documentation for writing new CLI tests
@c12i c12i self-assigned this Oct 23, 2024
@jost-s jost-s unassigned c12i Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready for refinement
Development

No branches or pull requests

1 participant