Integrating M@S Unit Tests and Coverage in Milo PR Builds

[yesil]

Parts of M@S[1] have recently been migrated from an external repository into Milo. This change aims to enhance the developer experience and streamline the PR process, especially given the growing number of contributors and the increasing volume of pending PRs.

Currently, however, M@S unit tests are not included in Milo's PR build process, and their code coverage is not reflected in the Milo codecov report accessible from the PRs.

Although there are ongoing internal discussions about potentially simplifying or even phasing out the M@S build, this may take some time to implement.

In the interim, I would like to discuss how we can integrate a Milo feature located under libs/features/, which has its own build mechanism, into the Milo pipeline.

1. https://github.com/adobecom/milo/tree/stage/libs/features/mas

[mokimo]

As for the unit tests, completely fine to contribute a workflow to run those. With the following caveats:

• Consistency. Introducing unreliable tests are currently not an option as the stage procress depends on green tests. Alternatively, you can run them without "failing" the suite and more informational.
• Time. Usually not an issue for unit tests, however - if the suite would run for say 30 minutes - it would also block the process and that would not be acceptable.

As for the build process, in theory you can also automate that portion with a workflow if you want to.

[3ch023]

Right, not having Merch tests running in the Milo pipeline is our tech dept, thanks for raising it.
One good part is that we don't have any corp dependencies, so it should be possible to run those tests in github actions.

@mokimo I would say merch tests are reliable, they don't rely on outer dependencies. and timewise it should be ok too.
can we add it to run-tests.yaml or is the separate WF preferable?

[mokimo]

I'd create a separate WF such as run-merch-tests.yaml so the output is clearer on the PR in case merch tests failed rather than the general milo ones.

However, if you'd just run them using web-test-runner - you might be able to just tweak web test runner around to also include running the M&S tests, rather than adding another workflow.

Both options are valid, I'd prefer using WTR to include the M&S tests by default - but if this complicates the overall setup, a new WF might be easier and cleaner.

[3ch023]

yeah, probably a separate workflow doesn't hurt..
will be a bit faster to troubleshoot

[yesil]

Thanks @mokimo @3ch023 for the feedback.

WTR already runs all mech blocks that covers behind the scene for MAS.
And MAS unit tests can add another 30s to the execution which would degrade DX.
So it is better to keep them outside for now.

I'm adding the .github/workflows/run-mas-tests.yaml that is for MAS feature.