Solve test measurements issue with CI #895

LuisAlfredoNu · 2024-09-06T14:29:35Z

Before submitting

Please complete the following checklist when submitting a PR:

All new features must include a unit test.
If you've fixed a bug or added code that should be tested, add a test to the
tests directory!
All new functions and code must be clearly commented and documented.
If you do make documentation changes, make sure that the docs build and
render correctly by running make docs.
Ensure that the test suite passes, by running make test.
Add a new entry to the .github/CHANGELOG.md file, summarizing the
change, and including a link back to the PR.
Ensure that code is properly formatted by running make format.

When all the above are checked, delete everything above the dashed
line and fill in the pull request template.

Context:
Revert changes about testing the measurements with shots. The probability of failing the test is non zero then something the test can fail randomly and reduces the CI quality in all PR.

Description of the Change:

Benefits:
Stop failing the CI regarding the measurement test with shots

Possible Drawbacks:

Related GitHub Issues:

github-actions · 2024-09-06T14:37:24Z

Hello. You may have forgotten to update the changelog!
Please edit .github/CHANGELOG.md with:

A one-to-two sentence description of the change. You may include a small working example for new features.
A link back to this PR.
Your name (or GitHub username) in the contributors section.

…ane-lightning into testMeasurementsIssue

codecov · 2024-09-06T15:47:03Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.17%. Comparing base (7943928) to head (0c9906e).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (7943928) and HEAD (0c9906e). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (7943928) HEAD (0c9906e)

9 8

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #895      +/-   ##
==========================================
- Coverage   92.20%   83.17%   -9.03%     
==========================================
  Files          93       24      -69     
  Lines       11376     2146    -9230     
==========================================
- Hits        10489     1785    -8704     
+ Misses        887      361     -526

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ane-lightning into testMeasurementsIssue

AmintorDusko

Could you please report the change in testing times that this change will bring?

mlxd · 2024-09-09T15:19:48Z

Given the current CI pressure, I am willing to say 1e6 shot workloads aren't something we should be adding to the mix.

LuisAlfredoNu · 2024-09-09T15:27:05Z

@AmintorDusko

Could you please report the change in testing times that this change will bring?

Running locally on a laptop with a CPU: 12th Gen Intel(R) Core(TM) i7-1265U and OMP_NUM_THREADS = 10

Command test:

PL_DEVICE=lightning.qubit python -m pytest tests/lightning_qubit/test_measurements_class.py --maxfail=1

Timing with the current implementation:
2030 passed, 516 skipped in 278.14s (0:04:38)

Timing with master branch
2030 passed, 516 skipped in 253.85s (0:04:13)

Don't change too much

LuisAlfredoNu · 2024-09-09T16:18:53Z

@mlxd I agree with that but the CIs fail randomly and my major concern is to delay the general workflow of the others. That is why I propose this quick fix 😕

I am working on a solution but is not ready yet.

LuisAlfredoNu · 2024-09-09T18:14:26Z

@AmintorDusko

Extracting the run time from the last CI testing in master branch and the current PR

Summary

Test	CI `master`	CI current PR
LQ	946 s	941 s
LK CPU	1739 s	1756 s
LK GPU	621 s	616 s
LGPU	133 s	133 s

Run time from CI in `master`

Python Tests (lightning_qubit, blas-OFF, test-group-1,2,3,4)

=========== 2072 passed, 358 skipped, 3 xpassed in 246.96s (0:04:06) ===========
================ 2070 passed, 363 skipped in 234.28s (0:03:54) =================
=========== 2083 passed, 348 skipped, 2 xfailed in 235.76s (0:03:55) ===========
=========== 2072 passed, 360 skipped, 1 xpassed in 231.76s (0:03:51) ===========

Python Tests (lightning_kokkos, kokkos-4.3.01, model-OPENMP, test-group-1,2,3,4,5,6,7)

= 1084 passed, 351 skipped, 8584 deselected, 2 xfailed, 1 xpassed in 247.77s (0:04:07) =
== 1077 passed, 359 skipped, 8585 deselected, 1 xfailed in 249.85s (0:04:09) ===
== 1079 passed, 356 skipped, 8584 deselected, 3 xpassed in 255.04s (0:04:15) ===
======== 1087 passed, 351 skipped, 8584 deselected in 242.57s (0:04:02) ========
======== 1096 passed, 342 skipped, 8584 deselected in 252.37s (0:04:12) ========
== 1082 passed, 353 skipped, 8584 deselected, 3 xfailed in 243.75s (0:04:03) ===
======== 1091 passed, 346 skipped, 8585 deselected in 251.49s (0:04:11) ========

Python Tests (lightning_kokkos, kokkos-4.3.01, model-CUDA)

= 7352 passed, 2363 skipped, 297 deselected, 6 xfailed, 4 xpassed in 621.49s (0:10:21) =

Python Tests (lightning_gpu, cuda-12)

===== 2887 passed, 1248 skipped, 6 xfailed, 2 xpassed in 133.34s (0:02:13) =====

Run time from CI in current PR

Python Tests (lightning_qubit, blas-OFF, test-group-1,2,3,4)

=========== 2075 passed, 356 skipped, 2 xfailed in 244.50s (0:04:04) ===========
=========== 2078 passed, 355 skipped, 1 xpassed in 227.26s (0:03:47) ===========
================ 2075 passed, 358 skipped in 232.73s (0:03:52) =================
=========== 2070 passed, 360 skipped, 3 xpassed in 238.31s (0:03:58) ===========

Python Tests (lightning_kokkos, kokkos-4.3.01, model-OPENMP, test-group-1,2,3,4,5,6,7)

= 1080 passed, 356 skipped, 8585 deselected, 1 xfailed, 1 xpassed in 256.76s (0:04:16) =
= 1086 passed, 347 skipped, 8586 deselected, 2 xfailed, 2 xpassed in 249.96s (0:04:09) =
======== 1097 passed, 342 skipped, 8584 deselected in 256.74s (0:04:16) ========
======== 1087 passed, 351 skipped, 8585 deselected in 243.43s (0:04:03) ========
======== 1080 passed, 358 skipped, 8585 deselected in 249.57s (0:04:09) ========
= 1079 passed, 356 skipped, 8585 deselected, 2 xfailed, 1 xpassed in 251.63s (0:04:11) =
== 1088 passed, 348 skipped, 8586 deselected, 1 xfailed in 252.41s (0:04:12) ===

Python Tests (lightning_kokkos, kokkos-4.3.01, model-CUDA)

= 7353 passed, 2363 skipped, 297 deselected, 6 xfailed, 4 xpassed in 616.41s (0:10:16) =

Python Tests (lightning_gpu, cuda-12)

===== 2887 passed, 1248 skipped, 6 xfailed, 2 xpassed in 133.13s (0:02:13) =====

LuisAlfredoNu · 2024-09-09T18:23:11Z

@AmintorDusko
The Windows and iOS tests just run the basic penneylane tests suit.

tests/lightning_qubit/test_measurements_class.py

AmintorDusko

As times were not quite affected by these changes and precision improved, I'm good with it.
Thank you @LuisAlfredoNu.

maliasadi

LGTM! Just reverting the meas tests back to their original configs to prevent the random failures on CIs. 👍

LuisAlfredoNu added 2 commits September 6, 2024 10:12

increase the number of shots

438ba76

reduce flaky to 5

da1d850

LuisAlfredoNu marked this pull request as ready for review September 6, 2024 14:29

Auto update version from '0.39.0-dev5' to '0.39.0-dev6'

398149a

LuisAlfredoNu added the ci:build_wheels Activate wheel building. label Sep 6, 2024

LuisAlfredoNu added 2 commits September 6, 2024 11:01

trigger CIs

be2e941

Merge branch 'testMeasurementsIssue' of github.com:PennyLaneAI/pennyl…

7d6f4ce

…ane-lightning into testMeasurementsIssue

LuisAlfredoNu and others added 8 commits September 6, 2024 15:53

update tolerences

7393043

Merge branch 'master' into testMeasurementsIssue

ab516c5

Merge branch 'master' into testMeasurementsIssue

821ca2f

Merge branch 'master' into testMeasurementsIssue

4870f73

change tolerences

462a0c4

Auto update version from '0.39.0-dev6' to '0.39.0-dev7'

1e6e9b3

trigger CIs

ad73faa

Merge branch 'testMeasurementsIssue' of github.com:PennyLaneAI/pennyl…

f7e21e1

…ane-lightning into testMeasurementsIssue

AmintorDusko reviewed Sep 9, 2024

View reviewed changes

tests/lightning_qubit/test_measurements_class.py Show resolved Hide resolved

AmintorDusko approved these changes Sep 9, 2024

View reviewed changes

Merge branch 'master' into testMeasurementsIssue

6ebbc90

maliasadi approved these changes Sep 9, 2024

View reviewed changes

maliasadi removed the ci:build_wheels Activate wheel building. label Sep 9, 2024

trigger ci

0c9906e

LuisAlfredoNu merged commit a36fe22 into master Sep 9, 2024
71 of 72 checks passed

LuisAlfredoNu deleted the testMeasurementsIssue branch September 9, 2024 21:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solve test measurements issue with CI #895

Solve test measurements issue with CI #895

LuisAlfredoNu commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024

codecov bot commented Sep 6, 2024 •

edited

Loading

AmintorDusko left a comment

mlxd commented Sep 9, 2024 •

edited

Loading

LuisAlfredoNu commented Sep 9, 2024

LuisAlfredoNu commented Sep 9, 2024

LuisAlfredoNu commented Sep 9, 2024

LuisAlfredoNu commented Sep 9, 2024

AmintorDusko left a comment

maliasadi left a comment

Solve test measurements issue with CI #895

Solve test measurements issue with CI #895

Conversation

LuisAlfredoNu commented Sep 6, 2024 • edited Loading

Before submitting

github-actions bot commented Sep 6, 2024

codecov bot commented Sep 6, 2024 • edited Loading

Codecov Report

AmintorDusko left a comment

Choose a reason for hiding this comment

mlxd commented Sep 9, 2024 • edited Loading

LuisAlfredoNu commented Sep 9, 2024

LuisAlfredoNu commented Sep 9, 2024

LuisAlfredoNu commented Sep 9, 2024

Summary

Run time from CI in master

Python Tests (lightning_qubit, blas-OFF, test-group-1,2,3,4)

Python Tests (lightning_kokkos, kokkos-4.3.01, model-OPENMP, test-group-1,2,3,4,5,6,7)

Python Tests (lightning_kokkos, kokkos-4.3.01, model-CUDA)

Python Tests (lightning_gpu, cuda-12)

Run time from CI in current PR

Python Tests (lightning_qubit, blas-OFF, test-group-1,2,3,4)

Python Tests (lightning_kokkos, kokkos-4.3.01, model-OPENMP, test-group-1,2,3,4,5,6,7)

Python Tests (lightning_kokkos, kokkos-4.3.01, model-CUDA)

Python Tests (lightning_gpu, cuda-12)

LuisAlfredoNu commented Sep 9, 2024

AmintorDusko left a comment

Choose a reason for hiding this comment

maliasadi left a comment

Choose a reason for hiding this comment

LuisAlfredoNu commented Sep 6, 2024 •

edited

Loading

codecov bot commented Sep 6, 2024 •

edited

Loading

mlxd commented Sep 9, 2024 •

edited

Loading

Run time from CI in `master`