-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: Add coverage tracking #469
Conversation
8c2e721
to
d8f1300
Compare
Current dependencies on/for this PR: This comment was auto-generated by Graphite. |
d8f1300
to
56594d3
Compare
56594d3
to
6685dcf
Compare
6685dcf
to
1b6e06c
Compare
1b6e06c
to
8afe2d9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks sane, I imagine we're gonna end up with some cross repo duplication here but feels like we can come back to that when stable.
8afe2d9
to
9dd475a
Compare
f865c49
to
4aec3a2
Compare
59cef54
to
c024792
Compare
From local experimentation on master, it seems that this test is just really long-running. It takes 7+ minutes to delete the aws:ecs:Service for some reason, which eats up the bulk of the test runtime. |
c024792
to
6954144
Compare
Sounds like some AWS resources taking long to delete is a known thing. |
Codecov Report
@@ Coverage Diff @@
## main #469 +/- ##
=======================================
Coverage ? 61.90%
=======================================
Files ? 30
Lines ? 7533
Branches ? 0
=======================================
Hits ? 4663
Misses ? 2485
Partials ? 385 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
6954144
to
f7f5e82
Compare
6815bba
to
c000ca1
Compare
c000ca1
to
cfbac1b
Compare
Okay, so this turned into a yak shave. #464 (comment) has an update from 3 weeks ago on the tooling side. On top of that, even with the tooling, there was another problem: we SIGKILLed all plugins so we couldn't get plugin coverage data anyway. However, with the changes I've made to pu/pu recently, namely the following, plugins can now be terminated gracefully.
I've stacked this PR on top of #501. It also needs a version of pu/pu with the above two PRs merged into it, after which binaries are able to produce meaningful coverage data. |
The PR has changed quite significantly since initial review.
47cbb10
to
54079d5
Compare
01c5139
to
48cab9c
Compare
04536a9
to
e5cd816
Compare
e5cd816
to
9dcec3a
Compare
9dcec3a
to
1e45204
Compare
Adds a `make test_cover` target to the Makefile which builds pulumi-language-yaml with coverage instrumentation and runs all tests with coverage tracking. Note that right now, there are no tests that invoke pulumi-language-yaml so integration test data is empty. For CI, adds an coverage option that, when enabled, will run `make test_cover` instead of `make test`, and upload the results to codecov. This option is only enabled for tests invoked for PRs and by the `/run-acceptance-tests` command. Additionally, similarly to pulumi/pulumi#13334, this adds a workflow that runs tests on master with coverage tracking every 12 hours. Resolves #464
Adds an experimental tool that runs `go test -c` and then runs the tests for each package separately. This will provide combined coverage data for unit and integration tests.
Adds a flag to use test2json to run the test binary, enabling compatibility with gotestsum.
1e45204
to
91a783f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Adds a
make test_cover
target to the Makefilewhich builds pulumi-language-yaml with coverage instrumentation
and runs all tests with coverage tracking.
For CI, adds an coverage option that, when enabled,
will run
make test_cover
instead ofmake test
,and upload the results to codecov.
This option is only enabled for tests invoked for PRs
and by the
/run-acceptance-tests
command.Additionally, similarly to pulumi/pulumi#13334,
this adds a workflow that runs tests on master with coverage tracking
every 12 hours.
Resolves #464