Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add component test apps #381

Merged
merged 22 commits into from
Sep 27, 2024
Merged

ci: add component test apps #381

merged 22 commits into from
Sep 27, 2024

Conversation

igrr
Copy link
Member

@igrr igrr commented Sep 18, 2024

Done in this PR:

  • Moved all existing unit tests into per-component test apps — Convert the tests from test components to component-specific test apps (IEC-11) #149.
  • Added per-component .build-test-rules.yml files
  • For components which don't have any unit tests and don't have examples, added a build-only test app
  • Added some new tests where it was easy
    • added pytest for coremark example
    • libsodium test now actually compares test output to .exp files
    • added test for libpng
  • Build examples and test_apps based on changes in the PR (only relevant components) — Target testing strategy (IEC-1) #16
  • Junit reports from examples and test_apps are combined and the results are published
  • Removed all target markers (pytest.mark.supported_targets, pytest.mark.esp32) because they didn't do anything
  • Added a pre-commit hook with a few consistency checks
    • every component is added to the upload job
    • every component is mentioned in the issue template — Update PR template: check for adding new component (IEC-13) #157
    • every .build-test-rules.yml file is mentioned in the idf-build-apps config
      • we can add a "glob" feature to idf-build-apps, then this can be simplified
    • we can add more checks (e.g. changelog, examples, idf_component.yml fields, SBOM files, ...)

Next PR:

  • Migrate clang-tidy job from test_app
  • Remove test_app
  • Run target test jobs on more chips
  • Re-add test parametrization by chip targets using pytest.mark.parametrize over target parameter
  • Revisit build-test-rules files, remove unnecessary builds, move some target tests to either linux target or qemu

🚧 — means that this is something to be done in future PRs. I will create some issues about this when/if this PR gets merged.

Component Has example(s) Example has pytest Test app added Test app has tests
bdc_motor     yes no (build only)
catch2 yes yes no, rely on example  
cbor yes yes no, rely on example  
ccomp_timer no 🚧   yes yes
coap yes yes no, rely on example  
coremark yes yes, added no, rely on example  
dhara no 🚧   no, rely on spi_nand_flash example no
eigen yes no 🚧 no, rely on example  
esp_delta_ota yes no 🚧 no, rely on example  
esp_encrypted_img no 🚧   yes yes
esp_jpeg yes no 🚧 no, rely on example  
esp_lcd_qemu_rgb yes no 🚧 no, rely on example  
esp_serial_slave_link no 🚧   yes no (build only) 🚧
expat no 🚧   yes yes
fmt yes yes no, rely on example  
freetype yes yes no, rely on example  
iqmath yes no 🚧 no, rely on example  
jsmn no 🚧   yes no (build only) 🚧
json_generator no 🚧   yes no (build only) 🚧
json_parser no 🚧   yes yes
led_strip yes no 🚧 no, rely on example  
libpng no 🚧   yes yes
libsodium no 🚧   yes yes
network_provisioning yes no 🚧 no, rely on example  
nghttp no 🚧   no, rely on sh2lib example no
onewire_bus no 🚧   yes no (build only) 🚧
pcap no 🚧   yes no (build only) 🚧
pid_ctrl no 🚧   yes no (build only) 🚧
qrcode no 🚧   yes no (build only) 🚧
quirc no 🚧   yes yes
sh2lib yes yes no, rely on example  
spi_nand_flash yes no 🚧 no, rely on example  
supertinycron yes no 🚧 no, rely on example  
thorvg yes no 🚧 no, rely on example  
zlib no 🚧   yes no (build only) 🚧

@igrr igrr self-assigned this Sep 18, 2024

This comment was marked as outdated.

@igrr igrr force-pushed the ci/test_apps branch 6 times, most recently from 5567121 to 7375c24 Compare September 20, 2024 11:42
Copy link

github-actions bot commented Sep 20, 2024

Test Results

 10 files   10 suites   25m 22s ⏱️
 33 tests  33 ✅ 0 💤 0 ❌
165 runs  165 ✅ 0 💤 0 ❌

Results for commit 2e27d50.

♻️ This comment has been updated with latest results.

@igrr igrr force-pushed the ci/test_apps branch 7 times, most recently from 6eeb78c to ce21960 Compare September 20, 2024 15:26
.idf_build_apps.toml Outdated Show resolved Hide resolved
@igrr igrr force-pushed the ci/test_apps branch 7 times, most recently from f9cdcac to 705a018 Compare September 23, 2024 15:47
@igrr igrr force-pushed the ci/test_apps branch 2 times, most recently from 42ca438 to 69adfbe Compare September 27, 2024 10:09
@igrr igrr merged commit 0603d10 into master Sep 27, 2024
43 checks passed
@igrr igrr deleted the ci/test_apps branch September 27, 2024 13:14
@igrr
Copy link
Member Author

igrr commented Sep 27, 2024

Thanks for the reviews folks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants