Fix-test-suite-not-run-on-macos #6631

brettle · 2024-08-19T20:01:33Z

Description

Makes the test suite run successfully on macos-13.

Switched from macos-14/macos-latest to macos-13 because the macos-14 runner crashes during the build, presumably due to the fact that is has half as much memory.
Increased various timeouts/delays to accomodate the fact that the macos runner is much slower than the other runners. Let me know if you want those increases to be restricted to just the macos CI environment.
Skipped the billboard test during macos CI because it currently fails there but passes on my Mac laptop. Presumably the issue is some combination of the macos runner being headless (and thus only supporting OpenGL 2.1) and the fact that it is run minimized and without rendering. Fwiw that test also fails for me on my local Fedora 40 installation when using x11, but not when using wayland or when running without --minimize.
Various smaller fixes to allow the tests to run and make it easier to diagnose errors.

6.4.3 is no longer supported, while 6.5.3 is the latest LTS release and is supported through 31 Mar 2026. For this to work, the precompiled binaries at https://cyberbotics.com/files/repository/dependencies/ will need to be updated but I don't think I have the ability to do that.

…nux_runtime_dependencies.sh.

…ite to run with the new linux builds.

…d at least 6.4.3.

…u 20.04. Building libs is still TODO.

…e bundle the libs with webots.

… CI.

…nd webots executable.

…on MacOS.

…s much RAM as the latter.

…os during CI.

The test passes locally, so the failure is presumably due to the CI server being headless.

tests/default/controllers/test_suite_supervisor/test_suite_supervisor.py

Co-authored-by: CoolSpy3 <[email protected]>

brettle · 2024-08-29T19:26:11Z

The latest macos build and test-suite runs took nearly 2.5 hours, much longer than Ubuntu. Do we want to only run it when some additional or other label is present? @omichel?

omichel · 2024-08-30T05:50:07Z

The latest macos build and test-suite runs took nearly 2.5 hours, much longer than Ubuntu. Do we want to only run it when some additional or other label is present? @omichel?

Yes, that seems reasonable. Which label would fire it?

omichel

That looks good to me.
Thanks.

tests/default/controllers/test_suite_supervisor/test_suite_supervisor.py

Co-authored-by: Olivier Michel <[email protected]>

brettle · 2024-08-30T13:23:28Z

The latest macos build and test-suite runs took nearly 2.5 hours, much longer than Ubuntu. Do we want to only run it when some additional or other label is present? @omichel?

Yes, that seems reasonable. Which label would fire it?

I just added a "test suite mac" label and made the job require it. If that works for you, please reapprove.

brettle added 21 commits August 14, 2024 17:14

Add changelog entry.

473ce86

Merge branch 'master' into fix-to-use-supported-qt-version

068d715

Get Qt from system on ubuntu 22.04 and via aqtinstall on Ubuntu 20.04.

8293c4c

Move package installation to linux_compilation_dependencies.sh and li…

bc574a2

…nux_runtime_dependencies.sh.

Fix package name for Qt WebSockets on Ubuntu 22.04.

6de0828

Temporarily move Mac back to Qt 6.4.3 in hopes of getting the test su…

b192745

…ite to run with the new linux builds.

Temporarily allow access for debugging.

ef65c5d

Use aqt for all linux because Ubuntu 22.04 only has Qt 6.2 and we nee…

79b6b62

…d at least 6.4.3.

Add back libxcb-cursor0.

bd9f821

Add openssl as a runtime dep.

19287c6

Copy system openssl3 libs on Ubuntu 22.04 and use built libs on Ubunt…

f7a8341

…u 20.04. Building libs is still TODO.

Build openssl3 from source on Ubuntu 20.04.

67e9f26

Get rid of unneeded and unrecognized pushd in Makefile.linux.

e672407

Use aqt to install Qt 6.5.3 for mac.

87fa748

Allow login to mac if build fails.

8a49ced

Make sure the Contents/Frameworks folder is created.

eac55f7

Remove upterm steps that were added for debugging.

3d313b6

Make openssl just a compilation dependency for Ubuntu 22.04 because w…

37c0f00

…e bundle the libs with webots.

Try to run the test suite during macos CI.

71d6a43

Set WEBOTS_HOME_PATH such that tests can find Makefile.include during…

f1d55d3

… CI.

brettle added the test suite Start the test suite label Aug 19, 2024

brettle added 8 commits August 19, 2024 13:35

Allow login for debugging.

f4bbf59

Change WEBOTS_HOME to Webots.app and add logic to test_suite.py to fi…

a0ab607

…nd webots executable.

Get qt_warning_filters.conf from correct location on macos.

86873a1

Remove accidentally added checkout step.

9417594

Hopefully squash clang-format complaint.

d95696e

Ignore (but warn about) error trying to connect to background Webots …

70f7d7b

…on MacOS.

Fix formatting.

be29ef1

On Mac, timeout test after 60 minutes instead of 10.

9488367

brettle added 12 commits August 26, 2024 13:51

Try using macos-13 runner instead of macos-14. The former has twice a…

554a9cb

…s much RAM as the latter.

Merge branch 'master' into fix-test-suite-not-run-on-macos

60f7682

Install urdfdom during macos CI because it is needed by tests suite.

4a1c4da

Merge branch 'master' into fix-test-suite-not-run-on-macos

5e26dbd

Assert that move succeeds.

c0c2150

Print age of ode_tmp.dif so we can see how long we should wait on mac…

b51cef0

…os during CI.

Allow up to 20 seconds for world reload.

892614a

Try waiting longer before timing out.

9d850f8

Skip billboard.wbt test during macos CI.

ee341c4

The test passes locally, so the failure is presumably due to the CI server being headless.

Restore multithreaded build.

418c422

Remove unneeded comment.

553c537

Remove upterm.

b301158

brettle marked this pull request as ready for review August 29, 2024 15:14

brettle requested a review from a team as a code owner August 29, 2024 15:14

CoolSpy3 linked an issue Aug 29, 2024 that may be closed by this pull request

Test Suite Makes Webots Crashes on macOS #2110

Closed

CoolSpy3 reviewed Aug 29, 2024

View reviewed changes

tests/default/controllers/test_suite_supervisor/test_suite_supervisor.py Outdated Show resolved Hide resolved

Only increase timeout on macos.

1468e14

Co-authored-by: CoolSpy3 <[email protected]>

omichel previously approved these changes Aug 30, 2024

View reviewed changes

tests/default/controllers/test_suite_supervisor/test_suite_supervisor.py Outdated Show resolved Hide resolved

Simplify some syntax.

6fbe675

Co-authored-by: Olivier Michel <[email protected]>

brettle dismissed omichel’s stale review via 6fbe675 August 30, 2024 13:09

Only run the test suite if the "test suite mac" label is present.

60fdb16

omichel self-requested a review August 30, 2024 13:26

CoolSpy3 added the test suite mac Start the test suite on macos. Can take hours. label Aug 30, 2024

brettle mentioned this pull request Aug 31, 2024

Run test suite on windows during CI #6648

Merged

omichel approved these changes Aug 31, 2024

View reviewed changes

brettle merged commit 2a0c4ea into cyberbotics:master Aug 31, 2024
44 checks passed

brettle deleted the fix-test-suite-not-run-on-macos branch August 31, 2024 11:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix-test-suite-not-run-on-macos #6631

Fix-test-suite-not-run-on-macos #6631

brettle commented Aug 19, 2024 •

edited

Loading

brettle commented Aug 29, 2024 •

edited

Loading

omichel commented Aug 30, 2024

omichel left a comment

brettle commented Aug 30, 2024

Fix-test-suite-not-run-on-macos #6631

Fix-test-suite-not-run-on-macos #6631

Conversation

brettle commented Aug 19, 2024 • edited Loading

brettle commented Aug 29, 2024 • edited Loading

omichel commented Aug 30, 2024

omichel left a comment

Choose a reason for hiding this comment

brettle commented Aug 30, 2024

brettle commented Aug 19, 2024 •

edited

Loading

brettle commented Aug 29, 2024 •

edited

Loading