Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Buildfarmer log 2024/04/15 - 2024/04/22 #43

Closed
github-actions bot opened this issue Apr 15, 2024 · 15 comments
Closed

Buildfarmer log 2024/04/15 - 2024/04/22 #43

github-actions bot opened this issue Apr 15, 2024 · 15 comments
Labels
buildfarmer-log Log of issues in the buildfarm

Comments

@github-actions
Copy link
Contributor

Previous log #39
@github-actions github-actions bot added the buildfarmer-log Log of issues in the buildfarm label Apr 15, 2024
@github-actions GitHub Actions
Copy link
Contributor Author

Apr 15

Build regressions 
Rci__nightly-release_ubuntu_jammy_amd64|805|2024-04-14 13:42:00.647000|Number of failed test exceeds the threshold|run tests
gz-math8-debbuilder|257|2024-04-15 09:46:30.867000|Failed to compile and install build|install build dependencies from debian/control
gz-msgs11-debbuilder|257|2024-04-15 10:28:50.252000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|252|2024-04-14 12:36:49.603000|Failed to compile and install build|install build dependencies from debian/control
sdformat15-debbuilder|250|2024-04-14 12:09:49.726000|Failed to compile and install build|install build dependencies from debian/control
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 100.0% in 15 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #2 in nightly_win_rep
- Failing 53.33% in 8 out of 15 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #3 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.benchmark_string_conversions

Issue #4 in nightly_win_rep
- Failing 40.0% in 6 out of 15 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #5 in nightly_win_rep
- Failing 40.0% in 6 out of 15 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #6 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 5 out of 5 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #7 in nightly_linux-rhel_repeated
- Failing 31.25% in 5 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-rhel_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

Issue #8 in gz_fuel_tools-8-win
- Failing 80.0% in 4 out of 5 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #9 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent
	ShaderParamTest.ShaderParam
	ServerRepeat/FollowActorTest.PublishCmd/0
	ModelPhotoShootTest.ModelPhotoShootRandomJoints
	ModelPhotoShootTest.ModelPhotoShootDefaultJoints

Issue #10 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #11 in nightly_linux-aarch64_repeated
- Failing 25.0% in 4 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

Issue #12 in nightly_linux-rhel_repeated
- Failing 25.0% in 4 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)

Issue #13 in nightly_linux-aarch64_repeated
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #14 in nightly_win_rep
- Failing 26.67% in 4 out of 15 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_deb" job)

Issue #15 in nightly_win_rep
- Failing 26.67% in 4 out of 15 builds
	rclcpp.TestExecutors/MultiThreadedExecutor.spinWithTimer

Issue #16 in nightly_win_rep
- Failing 26.67% in 4 out of 15 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #17 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 75.0% in 3 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #18 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 75.0% in 3 out of 4 builds
	JointFeaturesDetachTest/0.JointDetach
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #19 in gz_rendering-7-win
- Failing 100.0% in 3 out of 3 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #20 in nightly_linux-aarch64_repeated
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_record__rmw_cyclonedds_cpp

Issue #21 in nightly_win_rep
- Failing 20.0% in 3 out of 15 builds
	rclcpp.TestTimeSource.callbacks
	projectroot.test.rclcpp.test_time_source
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #22 in nightly_win_rep
- Failing 20.0% in 3 out of 15 builds
	rclpy.test_node.xunit.missing_result
	rclcpp.TestUtilities.wait_for_message_twice_one_sub
	rclcpp.TestUtilities.wait_for_message_indefinitely
	rclcpp.TestUtilities.wait_for_message
	rclcpp.TestTimeSource.reattach
	rclcpp.TestTimeSource.parameter_activation
	rclcpp.TestTimeSource.no_pre_jump_callback
	rclcpp.TestTimeSource.detachUnattached
	rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic
	rclcpp.TestTimeSource.clock
	rclcpp.TestTimeSource.check_use_clock_thread_value
	rclcpp.TestTimeSource.check_clock_thread_status
	rclcpp.TestTimeSource.callback_handler_erasure
	rclcpp.TestTimeSource.ROS_time_valid_wall_time
	rclcpp.TestTimeSource.ROS_time_valid_sim_time
	rclcpp.TestTimeSource.ROS_time_valid_attach_detach
	rclcpp.TestTimeSource.ROS_invalid_sim_time
	rclcpp.TestLoggerService.check_connect_get_logger_service
	rclcpp.TestInterfaceTraits.has_node_base_interface
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	projectroot.test_node
	projectroot.test.rclcpp.test_wait_for_message
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits

Issue #23 in gz_sim-8-win
- Failing 33.33% in 2 out of 6 builds
	INTEGRATION_mesh_inertia_calculation.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)

Issue #24 in gz_transport-ign-transport8-win
- Failing 50.0% in 2 out of 4 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #25 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)

Issue #26 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #27 in nightly_linux_repeated
- Failing 12.5% in 2 out of 16 builds
	ros2action.ros2action.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #28 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 13.33% in 2 out of 15 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #29 in nightly_win_rep
- Failing 13.33% in 2 out of 15 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #30 in nightly_win_rep
- Failing 13.33% in 2 out of 15 builds
	rclcpp.TimeSource.valid_clock_type_for_sim_time
	rclcpp.TimeSource.invalid_sim_time_parameter_override
	rclcpp.TimeSource.invalid_clock_type_for_sim_time
	rclcpp.TestLoggerService.test_set_logging_level_with_partial_invalid_param
	rclcpp.TestLoggerService.test_set_logging_level_with_invalid_param
	rclcpp.TestLoggerService.test_set_and_get_one_logging_level
	rclcpp.TestLoggerService.test_set_and_get_multi_logging_level
	rclcpp.TestLoggerService.check_connect_set_logger_service
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.addNodeTwoExecutors
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/SingleThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/SingleThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/SingleThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/SingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/SingleThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/SingleThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/SingleThreadedExecutor.addNodeTwoExecutors
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/MultiThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/MultiThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/MultiThreadedExecutor.spinAll
	rclcpp.TestExecutors/MultiThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/MultiThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/MultiThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/MultiThreadedExecutor.addNodeTwoExecutors
	rclcpp.PerTimerType/TestTimer.test_timer_without_autostart/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_timer_without_autostart/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_failures_with_exceptions/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_failures_with_exceptions/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.callback_with_period_zero/generic_timer (4-byte object <01-00 00-00>)                     

Issue #31 in Hci__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #32 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Plugin_TEST.test_ran
	UNIT_MainWindow_TEST.test_ran

Issue #33 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	LogSystemTest.LogResources
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #34 in gz_sim-7-win
- Failing 33.33% in 1 out of 3 builds
	UtilTest.ResolveSdfWorldFile
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-focal-amd64" job)
	  (also in: "gz_sim-7-win" job)

Issue #35 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/ServerFixture.RunNonBlockingMultiple/0

Issue #36 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/FuelCachedServer.CachedFuelWorld/0
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)

Issue #37 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	rosbag2_py.test_transport_py.xunit.missing_result
	projectroot.test_transport_py

Issue #38 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2doctor.ros2doctor.test.test_qos_compatibility.test_qos_compatibility

Issue #39 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestAddCallbackGroupsToExecutorStable/SingleThreadedExecutor.subscriber_triggered_to_receive_message
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #40 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 15, 2024 •
edited
Loading

Gz-debbuilders noble | Build regression | Investigating 🕵️

Follow up form #39 (comment)

Debbuilders can't find python3-disutils package, and it seems currently it's not available for noble

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 15, 2024 •
edited
Loading

Ros2 Humble, Iron and Rolling | New warning | Investigating 🕵️

Reference build:

In iceoryx_posh output:

[100%] Built target ext_cpptoml
error: patch failed: CMakeLists.txt:1
error: CMakeLists.txt: patch does not apply
CMake Warning at cmake/cpptoml/CMakeLists.txt:81 (message):
CMake step [patch] for 'cpptoml-build' failed! Error code: 1! Build of 
'cpptoml-build' might fail

Maybe a possible culprit could be: eclipse-iceoryx/iceoryx#2232

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 15, 2024 •
edited
Loading

Gz-sim-main | Test regression | Reported 📚

Reference build: https://build.osrfoundation.org/job/gz_sim-ci-main-jammy-amd64/63/

Test regressions:

Integration NetworkHandshake output 
/home/jenkins/workspace/gz_sim-ci-main-jammy-amd64/gz-sim/test/integration/network_handshake.cc:220
Expected: (100u) <= (zPos.size()), actual: 100 vs 70
Integration ServerRepeat output 
/home/jenkins/workspace/gz_sim-ci-main-jammy-amd64/gz-sim/test/integration/scene_broadcaster_system.cc:588
Expected: (stepMsg.stats().real_time().nsec()) < (_msg.stats().real_time().nsec()), actual: 928317066 vs 29291622

None of the test has changed in the last 2 years:

This issue was reported in: gazebosim/gz-sim#2371

@github-actions GitHub Actions
Copy link
Contributor Author

Apr 16

Build regressions 
nightly_linux-rhel_repeated|1867|2024-04-16 04:00:01.088000|Jenkins channel closing down (Agent disconnected)|test
nightly_win_rep|3326|2024-04-16 04:00:01.942000|Number of failed test exceeds the threshold|test
Rci__nightly-release_ubuntu_jammy_amd64|806|2024-04-15 18:39:25.063000|Number of failed test exceeds the threshold|run tests
gz-math8-debbuilder|259|2024-04-16 09:47:30.529000|Failed to compile and install build|install build dependencies from debian/control
gz-msgs11-debbuilder|259|2024-04-16 10:27:41.315000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|254|2024-04-15 11:58:35.136000|Failed to compile and install build|install build dependencies from debian/control
gz_plugin-ci-ign-plugin1-homebrew-amd64|36|2024-04-16 11:06:23.751000|Unknown Reason|brew information
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_deb
- Failing 73.33% in 11 out of 15 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rel" job)

Issue #2 in nightly_win_deb
- Failing 66.67% in 10 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #3 in nightly_linux-aarch64_repeated
- Failing 37.5% in 6 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #4 in nightly_linux-aarch64_repeated
- Failing 37.5% in 6 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_multiple_nodes_launch_test.launch_testing_examples.check_multiple_nodes_launch_test (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)

Issue #5 in nightly_linux_iron_coverage
- Failing 46.15% in 6 out of 13 builds
	test_tracetools.test_tracetools.test.test_timer.TestTimer.test_all

Issue #6 in nightly_linux_iron_coverage
- Failing 46.15% in 6 out of 13 builds
	projectroot.test_timer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)

Issue #7 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.benchmark_string_conversions

Issue #8 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 5 out of 5 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #9 in gz_fuel_tools-8-win
- Failing 100.0% in 4 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #10 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #11 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #12 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent
	ShaderParamTest.ShaderParam
	ServerRepeat/FollowActorTest.PublishCmd/0
	ModelPhotoShootTest.ModelPhotoShootRandomJoints
	ModelPhotoShootTest.ModelPhotoShootDefaultJoints

Issue #13 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #14 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 20.0% in 3 out of 15 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)

Issue #15 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 100.0% in 3 out of 3 builds
	JointFeaturesDetachTest/0.JointDetach
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #16 in nightly_linux-aarch64_repeated
- Failing 18.75% in 3 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #17 in nightly_linux_repeated
- Failing 18.75% in 3 out of 16 builds
	ros2action.ros2action.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #18 in nightly_linux-aarch64_repeated
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #19 in nightly_win_rel
- Failing 18.75% in 3 out of 16 builds
	ros2cli.ros2cli.test.test_daemon.test_get_name

Issue #20 in gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #21 in gz_sim-ci-gz-sim7-focal-amd64
- Failing 66.67% in 2 out of 3 builds
	UtilTest.ResolveSdfWorldFile
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-focal-amd64" job)
	  (also in: "gz_sim-7-win" job)

Issue #22 in gz_transport-ign-transport8-win
- Failing 50.0% in 2 out of 4 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #23 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	ros2bag.test.test_record.test_record
	  (also in: "nightly_linux_repeated" job)

Issue #24 in nightly_linux_repeated
- Failing 12.5% in 2 out of 16 builds
	ros2param.ros2param.test.test_verb_list.test_verb_list (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #25 in gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)

Issue #26 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Plugin_TEST.test_ran
	UNIT_MainWindow_TEST.test_ran

Issue #27 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	RgbdCameraTest.RgbdCameraBox

Issue #28 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Util_TEST.test_ran
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)

Issue #29 in gz_sim-ign-gazebo6-win
- Failing 33.33% in 1 out of 3 builds
	ServerRepeat/ServerFixture.RunNonBlockingMultiple/0

Issue #30 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/FuelCachedServer.CachedFuelWorld/0
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)

Issue #31 in Hci__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #32 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy
	  (also in: "nightly_win_deb" job)

Issue #33 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #34 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.test_node_name__rmw_cyclonedds_cpp__rmw_fastrtps_cpp.xunit.missing_result
	projectroot.test_node_name__rmw_cyclonedds_cpp__rmw_fastrtps_cpp
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #35 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	test_rmw_implementation.test_wait_set__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.test_unique_identifiers__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.test_subscription__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.test_service__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.test_graph_api__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.test_event__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.test_client__rmw_fastrtps_cpp.gtest.missing_result
	test_rmw_implementation.TestDurationInfinite.create_publisher
	projectroot.test_wait_set__rmw_fastrtps_cpp
	projectroot.test_unique_identifiers__rmw_fastrtps_cpp
	projectroot.test_subscription__rmw_fastrtps_cpp
	projectroot.test_service__rmw_fastrtps_cpp
	projectroot.test_graph_api__rmw_fastrtps_cpp
	projectroot.test_event__rmw_fastrtps_cpp
	projectroot.test_duration_infinite__rmw_fastrtps_cpp
	projectroot.test_client__rmw_fastrtps_cpp

Issue #36 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rcl_action.TestActionClientWait.test_wait_set_add_action_client
	projectroot.test_wait
	  (also in: "nightly_win_rep" job)

Issue #37 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_action_client
	  (also in: "nightly_win_rep" job)

Issue #38 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rcl_action.TestActionServerInitFini.test_action_server_init_fini
	rcl_action.TestActionGraphFixture__rmw_fastrtps_cpp.test_action_get_client_names_and_types_by_node
	rcl_action.TestActionCommunication__rmw_fastrtps_cpp.test_valid_goal_comm
	rcl_action.TestActionClientServerInteraction__rmw_fastrtps_cpp.test_interaction
	rcl_action.TestActionClientBaseFixture.test_action_client_init_fini
	projectroot.test_graph__rmw_fastrtps_cpp
	projectroot.test_action_server
	projectroot.test_action_interaction__rmw_fastrtps_cpp
	projectroot.test_action_communication__rmw_fastrtps_cpp
	launch_pytest.test.launch_pytest.tools.test_process.test_sync_process_tools

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 16, 2024 •
edited
Loading

Rolling CI Release | Build regression by time out | Investigating 🕵️

Reference build: https://build.ros2.org/view/Rci/job/Rci__nightly-release_ubuntu_jammy_amd64/805/

Build timed out after 300 minutes

I've been checking the log time differences between build 804 (passing) and 805 (failing):

Build 804 vs 805
Section Duration 1 (s) Duration 2 (s) Difference (s)
Check free disk space 0.003 0.011 0.008
cpu info 0.022 0.025 0.003
docker version 0.039 0.039 0.0
docker info 0.017 0.021 0.004
Check docker status 0.509 0.614 0.105
Embed wrapper scripts 0.002 0.002 0.0
Clone ros_buildfarm 0.784 0.861 0.077
Write PGP repository keys 0.002 0.002 0.0
Generate Dockerfile - CI tasks 0.427 0.502 0.075
Build Dockerfile - generating CI tasks 4.936 4.126 -0.81
Run Dockerfile - generating CI tasks 4.104 4.807 0.703
Build Dockerfile - create workspace 3.845 3.819 -0.026
Run Dockerfile - create workspace 50.169 47.505 -2.664
Copy dependency list 0.004 0.006 0.002
Build Dockerfile - build and install 0.538 0.616 0.078
Run Dockerfile - build and install 3588.16 4476.131 887.976
Compress install space 33.434 37.739 4.305
Build Dockerfile - build and test 0.3 0.598 0.298
Run Dockerfile - build and test 11971.2 NA NA
Clean up to save disk space on agents 4.391 NA NA
fetch repos files(s) [subsection] 0.008 0.006 -0.002
import repositories [subsection] 30.964 33.599 2.635
vcs export --exact [subsection] 0.899 0.794 -0.105
mark packages with IGNORE files [subsection] 4.786 3.39 -1.396
Enumerating packages needed to build [subsection] 0.891 0.818 -0.073
Resolving packages versions using apt cache [subsection] 1.073 1.273 0.2
build workspace in isolation and install [subsection] 3586.39 4475.144 888.757
build workspace in isolation [subsection] 7145.94 8521.609 1375.673
build tests [subsection] 69.705 80.782 11.077
run tests [subsection] 4754.62 NA NA
Build 804 vs 806
Section Duration 1 (s) Duration 2 (s) Difference (s)
Check free disk space 0.003 0.002 -0.001
cpu info 0.022 0.025 0.003
docker version 0.039 0.067 0.028
docker info 0.017 0.028 0.011
Check docker status 0.509 0.642 0.133
Embed wrapper scripts 0.002 0.005 0.003
Clone ros_buildfarm 0.784 0.888 0.104
Write PGP repository keys 0.002 0.003 0.001
Generate Dockerfile - CI tasks 0.427 0.473 0.046
Build Dockerfile - generating CI tasks 4.936 4.888 -0.048
Run Dockerfile - generating CI tasks 4.104 4.349 0.245
Build Dockerfile - create workspace 3.845 4.616 0.771
Run Dockerfile - create workspace 50.169 50.12 -0.049
Copy dependency list 0.004 0.006 0.002
Build Dockerfile - build and install 0.538 113.401 112.863
Run Dockerfile - build and install 3588.16 4329.414 741.259
Compress install space 33.434 38.053 4.619
Build Dockerfile - build and test 0.3 64.321 64.021
Run Dockerfile - build and test 11971.2 NA NA
Clean up to save disk space on agents 4.391 NA NA
fetch repos files(s) [subsection] 0.008 0.006 -0.002
import repositories [subsection] 30.964 32.303 1.339
vcs export --exact [subsection] 0.899 0.788 -0.111
mark packages with IGNORE files [subsection] 4.786 4.496 -0.29
Enumerating packages needed to build [subsection] 0.891 0.842 -0.049
Resolving packages versions using apt cache [subsection] 1.073 1.154 0.081
build workspace in isolation and install [subsection] 3586.39 4309.564 723.177
build workspace in isolation [subsection] 7145.94 8435.468 1289.532
build tests [subsection] 69.705 80.285 10.58
run tests [subsection] 4754.62 NA NA

@Blast545
Copy link
Contributor

Blast545 commented Apr 16, 2024 •
edited
Loading

INVESTIGATION Tracking down of builds failing for a long time.

Running this build in our database:

SELECT job_name, last_success_time
FROM (
    SELECT bs.job_name, MAX(bs.build_datetime) AS last_success_time
    FROM build_status bs
    INNER JOIN server_status ss ON bs.job_name = ss.job_name
    WHERE bs.status = 'SUCCESS'
    GROUP BY bs.job_name
) AS last_success_times
ORDER BY last_success_time ASC;

Gets you a sorted version of the last time some builds passed

Query result 
Rci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64 | 2022-09-19 10:32:48
Rci__nightly-connext_ubuntu_jammy_amd64 | 2023-01-10 05:38:47.375000
gz_plugin-ci-ign-plugin1-homebrew-amd64 | 2023-12-04 22:06:13.468000
nightly_win_rep | 2024-01-12 04:00:01.346000
gz_sim-ci-ign-gazebo6-homebrew-amd64 | 2024-02-23 03:06:19.670000
nightly_linux-rhel_repeated | 2024-02-23 04:52:05.529000
gz_physics-6-win | 2024-03-06 09:08:07.012000
Rci__nightly-cyclonedds_ubuntu_jammy_amd64 | 2024-03-13 10:03:52.333000
gz_sim-ci-ign-gazebo6-focal-amd64 | 2024-03-17 22:06:19.114000
gz_physics-ci-main-homebrew-amd64 | 2024-03-22 18:47:11.571000
gz_physics-ci-gz-physics6-homebrew-amd64 | 2024-03-23 20:06:37.508000
Rci__benchmark_ubuntu_jammy_amd64 | 2024-03-24 18:36:02.560000
nightly_linux_repeated | 2024-03-26 06:12:58.540000
gz_physics-ci-main-jammy-amd64 | 2024-03-26 07:06:30.570000
sdformat-ci-sdf13-homebrew-amd64 | 2024-03-27 05:06:26.990000
gz_rendering-ci-main-jammy-amd64 | 2024-03-27 19:09:32.086000
gz_sensors-ci-main-jammy-amd64 | 2024-03-27 23:06:33.885000
gz_fuel_tools-ign-fuel-tools7-win | 2024-03-29 03:02:08.742000
gz_physics-ci-ign-physics2-homebrew-amd64 | 2024-03-31 04:06:30.234000
gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64 | 2024-03-31 07:06:18.781000
gz_fuel_tools-ign-fuel-tools4-win | 2024-03-31 07:06:19.006000
gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64 | 2024-03-31 13:06:18.983000
gz_fuel_tools-8-win | 2024-03-31 13:06:19.263000
gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64 | 2024-03-31 13:06:19.378000
gz-sim9-debbuilder | 2024-04-01 12:34:00.642000
gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64 | 2024-04-01 17:06:41.369000
gz_physics-ci-ign-physics5-homebrew-amd64 | 2024-04-01 22:06:20.818000
gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64 | 2024-04-04 00:06:08.807000
Ici__benchmark_ubuntu_jammy_amd64 | 2024-04-05 00:06:58.693000
gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64 | 2024-04-06 04:06:10.061000
gz_sim-ci-gz-sim7-focal-amd64 | 2024-04-06 08:06:09.111000
gz_gui-ci-ign-gui6-focal-amd64 | 2024-04-09 01:06:07.016000
Hci__nightly-cyclonedds_ubuntu_jammy_amd64 | 2024-04-09 06:15:00.259000
Hci__nightly-debug_ubuntu_jammy_amd64 | 2024-04-09 06:15:00.923000
gz_transport-13-win | 2024-04-09 17:37:19.238000
Hci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64 | 2024-04-09 18:48:07.532000
gz_transport-ign-transport8-win | 2024-04-09 19:06:09.464000
Hci__nightly-fastrtps_ubuntu_jammy_amd64 | 2024-04-09 22:34:46.517000
Hci__nightly-release_ubuntu_jammy_amd64 | 2024-04-09 23:14:45.862000
gz_common-ci-main-homebrew-amd64 | 2024-04-10 03:06:10.845000
gz_common-ci-gz-common5-homebrew-amd64 | 2024-04-10 04:22:06.092000
Ici__nightly-cyclonedds_ubuntu_jammy_amd64 | 2024-04-10 06:15:01.284000
Ici__nightly-debug_ubuntu_jammy_amd64 | 2024-04-10 06:15:01.288000
Rci__nightly-fastrtps_ubuntu_jammy_amd64 | 2024-04-10 16:42:28.799000
Rci__nightly-release_ubuntu_jammy_amd64 | 2024-04-10 19:14:28.530000
Ici__nightly-fastrtps_ubuntu_jammy_amd64 | 2024-04-11 00:03:29.096000
Ici__nightly-release_ubuntu_jammy_amd64 | 2024-04-11 00:47:38.035000
Rci__nightly-debug_ubuntu_jammy_amd64 | 2024-04-11 06:15:00.565000
gz_sim-8-win | 2024-04-11 10:54:06.466000
gz_sim-ci-ign-gazebo3-focal-amd64 | 2024-04-11 23:06:13.964000
gz_sensors-ign-sensors3-win | 2024-04-12 12:06:04.208000
gz_sim-ci-gz-sim8-homebrew-amd64 | 2024-04-12 13:50:08.962000
gz_cmake-ci-ign-cmake2-homebrew-amd64 | 2024-04-12 14:06:05.239000
gz_tools-ci-ign-tools1-homebrew-amd64 | 2024-04-12 14:06:05.249000
gz_msgs-main-win | 2024-04-12 14:06:05.256000
gz_plugin-ign-plugin1-win | 2024-04-12 15:06:10.444000
sdformat-ci-sdf9-homebrew-amd64 | 2024-04-12 16:06:03.780000
gz_gui-ign-gui3-win | 2024-04-12 17:06:10.602000
gz_fuel_tools-ci-main-homebrew-amd64 | 2024-04-12 18:06:03.388000
gz_msgs-ign-msgs5-win | 2024-04-12 18:06:03.552000
gz_launch-ci-ign-launch2-focal-amd64 | 2024-04-12 18:06:03.825000
gz_math-ci-main-homebrew-amd64 | 2024-04-12 19:06:10.739000
gz_plugin-ci-main-homebrew-amd64 | 2024-04-12 20:06:04.904000
gz_sensors-ci-main-homebrew-amd64 | 2024-04-12 20:06:04.916000
gz_transport-ci-ign-transport11-homebrew-amd64 | 2024-04-12 21:06:13.244000
gz_msgs-ci-ign-msgs5-focal-amd64 | 2024-04-12 23:06:10.798000
gz_physics-ign-physics2-win | 2024-04-13 00:06:04.694000
gz_cmake-ci-main-jammy-amd64 | 2024-04-13 00:06:04.775000
gz_gui-ci-ign-gui3-focal-amd64 | 2024-04-13 01:08:45.154000
gz_common-ign-common3-win | 2024-04-13 02:06:05.488000
gz_msgs-ci-gz-msgs9-focal-amd64 | 2024-04-13 02:06:05.574000
gz_msgs-ci-main-jammy-amd64 | 2024-04-13 02:06:05.736000
gz_msgs-ign-msgs8-win | 2024-04-13 04:06:04.744000
gz_plugin-2-win | 2024-04-13 04:06:04.750000
gz_cmake-main-win | 2024-04-13 05:06:26.160000
gz_physics-ci-gz-physics6-focal-amd64 | 2024-04-13 05:06:26.334000
gz_physics-ci-ign-physics2-focal-amd64 | 2024-04-13 05:12:10.904000
gz_transport-ign-transport11-win | 2024-04-13 06:06:05.739000
sdformat-sdf9-win | 2024-04-13 06:06:06.284000
gz_gui-main-win | 2024-04-13 07:06:13.539000
gz_sensors-main-win | 2024-04-13 08:06:06.215000
gz_plugin-ci-ign-plugin1-focal-amd64 | 2024-04-13 08:06:06.297000
gz_cmake-ci-main-homebrew-amd64 | 2024-04-13 09:06:18.021000
gz_plugin-ci-gz-plugin2-homebrew-amd64 | 2024-04-13 10:06:08.718000
gz_utils-ci-main-homebrew-amd64 | 2024-04-13 10:06:08.727000
gz_plugin-ci-gz-plugin2-jammy-amd64 | 2024-04-13 10:06:08.735000
gz_gui-ci-main-homebrew-amd64 | 2024-04-13 11:06:14.281000
gz_plugin-ci-gz-plugin2-focal-amd64 | 2024-04-13 17:06:16.431000
gz_common-ci-ign-common3-focal-amd64 | 2024-04-13 19:06:13.876000
gz_fuel_tools-main-win | 2024-04-13 19:06:14.046000
gz_fuel_tools-ci-main-jammy-amd64 | 2024-04-13 19:06:14.131000
gz_msgs-ci-gz-msgs9-homebrew-amd64 | 2024-04-13 20:06:08.169000
gz_sensors-ci-ign-sensors3-homebrew-amd64 | 2024-04-13 21:06:20.681000
gz_utils-ci-ign-utils1-homebrew-amd64 | 2024-04-13 21:06:20.800000
gz_launch-ci-gz-launch6-focal-amd64 | 2024-04-13 21:06:20.838000
gz_sensors-ci-gz-sensors8-homebrew-amd64 | 2024-04-13 22:06:05.597000
gz_gui-ign-gui6-win | 2024-04-13 23:06:21.406000
gz_launch-ci-main-homebrew-amd64 | 2024-04-13 23:06:21.411000
gz_launch-ci-main-jammy-amd64 | 2024-04-13 23:06:21.587000
gz_physics-ign-physics5-win | 2024-04-14 00:06:04.823000
gz_msgs-ci-ign-msgs8-focal-amd64 | 2024-04-14 00:06:04.835000
gz_msgs-ci-gz-msgs10-homebrew-amd64 | 2024-04-14 02:06:05.357000
gz_math-ci-gz-math7-focal-amd64 | 2024-04-14 02:06:05.535000
gz_cmake-ci-gz-cmake3-homebrew-amd64 | 2024-04-14 03:06:20.144000
gz_fuel_tools-9-win | 2024-04-14 03:06:20.228000
gz_sim-main-win | 2024-04-14 03:06:22.042000
gz_math-ci-gz-math7-jammy-amd64 | 2024-04-14 03:06:22.132000
nightly_linux-aarch64_repeated | 2024-04-14 04:00:01.136000
gz_sensors-8-win | 2024-04-14 04:06:05.341000
gz_utils-2-win | 2024-04-14 05:06:16.905000
gz_cmake-ci-gz-cmake3-focal-amd64 | 2024-04-14 05:06:16.990000
gz_sim-ci-main-homebrew-amd64 | 2024-04-14 06:06:06.746000
gz_msgs-10-win | 2024-04-14 06:06:07.026000
gz_gui-ci-main-jammy-amd64 | 2024-04-14 06:10:05.272000
gz_launch-ci-gz-launch6-homebrew-amd64 | 2024-04-14 07:06:14.163000
gz_msgs-9-win | 2024-04-14 07:06:14.299000
gz_tools-ign-tools1-win | 2024-04-14 07:06:14.306000
gz_physics-ci-ign-physics5-focal-amd64 | 2024-04-14 07:08:05.912000
gz_transport-ci-gz-transport13-homebrew-amd64 | 2024-04-14 08:06:10.405000
gz_utils-ign-utils1-win | 2024-04-14 08:06:10.413000
gz_utils-main-win | 2024-04-14 08:06:10.432000
gz_rendering-ci-gz-rendering7-focal-amd64 | 2024-04-14 08:06:10.444000
gz_rendering-ci-ign-rendering3-focal-amd64 | 2024-04-14 08:08:23.277000
sdformat-sdf12-win | 2024-04-14 09:06:16.590000
gz_sensors-ci-gz-sensors7-focal-amd64 | 2024-04-14 10:19:33.030000
gz_sensors-ci-gz-sensors8-jammy-amd64 | 2024-04-14 10:27:21.657000
gz_sensors-ci-ign-sensors3-focal-amd64 | 2024-04-14 11:06:13.260000
gz_fuel_tools-ci-gz-fuel-tools9-homebrew-amd64 | 2024-04-14 12:06:04.337000
sdformat-ci-sdf12-homebrew-amd64 | 2024-04-14 12:06:04.344000
gz_math-ign-math6-win | 2024-04-14 13:06:17.882000
gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64 | 2024-04-14 14:06:06.959000
gz_msgs-ci-gz-msgs10-jammy-amd64 | 2024-04-14 14:06:06.963000
gz_transport-ci-gz-transport13-jammy-amd64 | 2024-04-14 15:06:24.047000
gz_tools-ci-gz-tools2-homebrew-amd64 | 2024-04-14 17:06:22.822000
gz_math-7-win | 2024-04-14 17:06:22.851000
gz_tools-ci-gz-tools2-focal-amd64 | 2024-04-14 17:06:23.007000
gz_math-ci-gz-math7-homebrew-amd64 | 2024-04-14 18:06:00.858000
gz_math-main-win | 2024-04-14 18:06:01.034000
gz_cmake-ci-gz-cmake3-jammy-amd64 | 2024-04-14 18:06:01.039000
gz_common-5-win | 2024-04-14 19:06:23.142000
gz_cmake-3-win | 2024-04-14 20:06:06.222000
gz_math-ci-ign-math6-focal-amd64 | 2024-04-14 22:06:08.776000
gz_math-ci-main-jammy-amd64 | 2024-04-14 23:06:18.477000
gz_sensors-7-win | 2024-04-15 00:06:07.797000
gz_tools-ci-gz-tools2-jammy-amd64 | 2024-04-15 01:06:21.520000
gz_tools-ci-ign-tools1-focal-amd64 | 2024-04-15 01:06:21.639000
gz_tools-2-win | 2024-04-15 02:06:06.797000
gz_transport-ci-gz-transport12-focal-amd64 | 2024-04-15 02:06:07.153000
gz_gui-ci-gz-gui7-focal-amd64 | 2024-04-15 03:06:14.215000
gz_plugin-ci-main-jammy-amd64 | 2024-04-15 03:06:14.256000
gz_transport-ci-ign-transport11-focal-amd64 | 2024-04-15 03:06:14.417000
nightly_win_deb | 2024-04-15 04:00:01.709000
nightly_win_rel | 2024-04-15 04:00:01.715000
gz_transport-ci-main-homebrew-amd64 | 2024-04-15 04:06:01.061000
gz_common-ci-gz-common5-jammy-amd64 | 2024-04-15 04:06:01.162000
gz_transport-ci-ign-transport8-focal-amd64 | 2024-04-15 04:06:01.279000
gz_common-ign-common4-win | 2024-04-15 05:06:18.326000
gz_common-ci-gz-common5-focal-amd64 | 2024-04-15 05:06:19.959000
nightly_linux_iron_coverage | 2024-04-15 05:53:42.317000
gz_rendering-ci-gz-rendering8-homebrew-amd64 | 2024-04-15 06:06:10.292000
gz_physics-main-win | 2024-04-15 06:06:10.992000
gz_rendering-ci-ign-rendering6-focal-amd64 | 2024-04-15 06:06:11.130000
gz_rendering-ci-gz-rendering8-jammy-amd64 | 2024-04-15 06:06:11.134000
gz_utils-ci-gz-utils2-focal-amd64 | 2024-04-15 07:06:06.588000
gz_utils-ci-ign-utils1-focal-amd64 | 2024-04-15 07:06:06.598000
gz_utils-ci-main-jammy-amd64 | 2024-04-15 07:06:06.617000
sdformat-ci-main-homebrew-amd64 | 2024-04-15 08:06:18.709000
sdformat-ci-sdf14-homebrew-amd64 | 2024-04-15 09:06:08.159000
gz_plugin-main-win | 2024-04-15 09:06:08.166000
sdformat-ci-sdf12-focal-amd64 | 2024-04-15 09:06:10.111000
sdformat-ci-sdf13-focal-amd64 | 2024-04-15 09:06:10.115000
sdformat-ci-sdf14-jammy-amd64 | 2024-04-15 10:06:21.246000
sdformat-ci-sdf9-focal-amd64 | 2024-04-15 10:06:21.260000
sdformat-sdf14-win | 2024-04-15 11:06:18.692000
sdformat15-debbuilder | 2024-04-15 11:06:44.409000
gz-fuel-tools10-debbuilder | 2024-04-15 11:43:08.782000
gz-transport14-debbuilder | 2024-04-15 11:49:29.683000
gz_transport-ci-main-jammy-amd64 | 2024-04-15 12:06:21.766000
gz-gui9-debbuilder | 2024-04-15 12:14:43.700000
gz-sensors9-debbuilder | 2024-04-15 12:33:17.407000
gz_transport-main-win | 2024-04-15 13:06:13.354000
gz-physics8-debbuilder | 2024-04-15 13:17:10.444000
gz_math-ci-ign-math6-homebrew-amd64 | 2024-04-15 14:06:10.662000
gz_sensors-ci-ign-sensors6-homebrew-amd64 | 2024-04-15 14:06:10.676000
sdformat-sdf13-win | 2024-04-15 14:06:10.730000
gz_physics-ci-gz-physics7-homebrew-amd64 | 2024-04-15 15:06:06.914000
gz-launch8-debbuilder | 2024-04-15 15:34:11.492000
gz_launch-ci-ign-launch5-focal-amd64 | 2024-04-15 16:06:12.737000
gz_utils-ci-gz-utils2-jammy-amd64 | 2024-04-15 18:06:16.982000
sdformat-ci-main-jammy-amd64 | 2024-04-15 18:06:16.994000
gz_gui-7-win | 2024-04-15 20:06:17.078000
gz_sim-7-win | 2024-04-15 20:06:18.440000
gz_sensors-ci-ign-sensors6-focal-amd64 | 2024-04-15 20:06:18.446000
gz_gui-ci-gz-gui8-jammy-amd64 | 2024-04-15 20:06:18.455000
gz_sim-ci-gz-sim7-homebrew-amd64 | 2024-04-15 21:06:03.894000
gz_common-main-win | 2024-04-15 21:06:03.901000
gz_utils-ci-gz-utils2-homebrew-amd64 | 2024-04-15 22:06:24.193000
gz_sensors-ign-sensors6-win | 2024-04-15 22:06:24.245000
sdformat-main-win | 2024-04-15 22:06:24.379000
gz_rendering-ci-gz-rendering7-homebrew-amd64 | 2024-04-15 23:06:06.123000
gz_transport-12-win | 2024-04-15 23:06:06.216000
gz_msgs-ci-main-homebrew-amd64 | 2024-04-16 00:06:17.745000
gz_cmake-ci-ign-cmake2-focal-amd64 | 2024-04-16 01:06:08.583000
Hci__benchmark_ubuntu_jammy_amd64 | 2024-04-16 01:52:25.147000
gz_gui-ci-gz-gui8-homebrew-amd64 | 2024-04-16 02:06:01.460000
gz_gui-ci-gz-gui7-homebrew-amd64 | 2024-04-16 03:06:24.887000
gz_cmake-ign-cmake2-win | 2024-04-16 03:06:24.953000
gz_gui-8-win | 2024-04-16 03:06:24.958000
gz_common-ci-ign-common4-focal-amd64 | 2024-04-16 03:06:25.309000
nightly_linux-aarch64_debug | 2024-04-16 04:00:00.940000
nightly_linux-aarch64_release | 2024-04-16 04:00:00.945000
nightly_linux-rhel_release | 2024-04-16 04:00:01.037000
nightly_linux-rhel_debug | 2024-04-16 04:00:01.092000
gz_launch-ci-ign-launch5-homebrew-amd64 | 2024-04-16 04:06:05.056000
gz_sim-ci-gz-sim8-jammy-amd64 | 2024-04-16 04:06:05.071000
gz_launch-ci-gz-launch7-jammy-amd64 | 2024-04-16 04:06:05.086000
packaging_linux-aarch64 | 2024-04-16 04:47:59.642000
gz_physics-ci-gz-physics7-jammy-amd64 | 2024-04-16 04:49:08.956000
gz_sensors-ci-gz-sensors7-homebrew-amd64 | 2024-04-16 05:06:18.857000
gz_transport-ci-gz-transport12-homebrew-amd64 | 2024-04-16 05:06:19.050000
nightly_linux_coverage | 2024-04-16 05:18:25.955000
gz_physics-7-win | 2024-04-16 05:23:10.236000
nightly_linux_debug | 2024-04-16 05:35:55.771000
nightly_linux_humble_coverage | 2024-04-16 05:53:43.829000
gz_gui-ci-ign-gui3-homebrew-amd64 | 2024-04-16 06:06:11.315000
gz_launch-ci-gz-launch7-homebrew-amd64 | 2024-04-16 06:06:11.332000
gz_transport-ci-ign-transport8-homebrew-amd64 | 2024-04-16 06:06:11.458000
gz_common-ci-main-jammy-amd64 | 2024-04-16 06:06:11.626000
nightly_linux_release | 2024-04-16 06:50:03.383000
packaging_linux | 2024-04-16 07:27:29.735000
packaging_linux-rhel | 2024-04-16 08:00:27.518000
packaging_windows | 2024-04-16 08:02:27.324000
packaging_windows_debug | 2024-04-16 08:30:04.382000
gz_rendering-ci-main-homebrew-amd64 | 2024-04-16 08:50:08.615000
gz_gui-ci-ign-gui6-homebrew-amd64 | 2024-04-16 09:06:11.356000
gz-cmake4-debbuilder | 2024-04-16 09:13:38.909000
gz-tools2-debbuilder | 2024-04-16 09:22:43.888000
gz-utils3-debbuilder | 2024-04-16 09:29:16.291000
gz-math8-debbuilder | 2024-04-16 09:34:07.711000
gz-plugin3-debbuilder | 2024-04-16 09:55:59.746000
gz_msgs-ci-ign-msgs5-homebrew-amd64 | 2024-04-16 10:06:04.932000
gz_msgs-ci-ign-msgs8-homebrew-amd64 | 2024-04-16 10:06:04.941000
gz-common6-debbuilder | 2024-04-16 10:10:14.903000
gz-msgs11-debbuilder | 2024-04-16 10:20:37.752000
gz-rendering9-debbuilder | 2024-04-16 10:48:32.374000

And this query returns the list of job_names that don't have any SUCCESS build:

SELECT bs.job_name
FROM build_status bs
INNER JOIN server_status ss ON bs.job_name = ss.job_name
WHERE bs.job_name NOT IN (
    SELECT job_name
    FROM build_status
    WHERE status = 'SUCCESS'
)
GROUP BY bs.job_name;
Query result 
Hci__nightly-connext_ubuntu_jammy_amd64
Hci__nightly-performance_ubuntu_jammy_amd64
Ici__nightly-connext_ubuntu_jammy_amd64
Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64
Ici__nightly-performance_ubuntu_jammy_amd64
Rci__nightly-performance_ubuntu_jammy_amd64
gz_common-ci-ign-common3-homebrew-amd64
gz_common-ci-ign-common4-homebrew-amd64
gz_launch-6-win
gz_launch-7-win
gz_launch-ci-ign-launch2-homebrew-amd64
gz_launch-ign-launch5-win
gz_launch-main-win
gz_rendering-7-win
gz_rendering-8-win
gz_rendering-ci-ign-rendering3-homebrew-amd64
gz_rendering-ci-ign-rendering6-homebrew-amd64
gz_rendering-ign-rendering3-win
gz_rendering-ign-rendering6-win
gz_rendering-main-win
gz_sim-ci-ign-gazebo3-homebrew-amd64
gz_sim-ci-main-jammy-amd64
gz_sim-ign-gazebo6-win

As part of our efforts to increase greenness metrics of the buildfarms, I would like us to take a closer look to get those to green,

@github-actions GitHub Actions
Copy link
Contributor Author

Apr 17

Build regressions 
nightly_linux_coverage|2343|2024-04-17 04:44:20.034000|Number of failed test exceeds the threshold|coverage analysis
Rci__nightly-performance_ubuntu_jammy_amd64|674|2024-04-16 18:54:47.096000|Packages performance_test Failed|build workspace in isolation and install
gz-math8-debbuilder|261|2024-04-17 10:36:10.301000|Failed to compile and install build|install build dependencies from debian/control
gz-sim9-debbuilder|253|2024-04-16 14:43:02.619000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|256|2024-04-16 11:58:49.888000|Failed to compile and install build|install build dependencies from debian/control
sdformat15-debbuilder|254|2024-04-16 11:34:18.263000|Failed to compile and install build|install build dependencies from debian/control
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_deb
- Failing 66.67% in 10 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #2 in nightly_linux_iron_coverage
- Failing 46.15% in 6 out of 13 builds
	test_tracetools.test_tracetools.test.test_timer.TestTimer.test_all

Issue #3 in nightly_linux_iron_coverage
- Failing 46.15% in 6 out of 13 builds
	projectroot.test_timer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_iron_coverage" job)

Issue #4 in nightly_win_rep
- Failing 42.86% in 6 out of 14 builds
	projectroot.test_service
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #5 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 100.0% in 4 out of 4 builds
	projectroot.benchmark_string_conversions

Issue #6 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 26.67% in 4 out of 15 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)

Issue #7 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 80.0% in 4 out of 5 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #8 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #9 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesDetachTest/0.JointDetach
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #10 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #11 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #12 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent
	ShaderParamTest.ShaderParam
	ServerRepeat/FollowActorTest.PublishCmd/0
	ModelPhotoShootTest.ModelPhotoShootRandomJoints
	ModelPhotoShootTest.ModelPhotoShootDefaultJoints

Issue #13 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	rclpy.test_subscription.xunit.missing_result
	  (also in: "nightly_win_rel" job)

Issue #14 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	projectroot.test_subscription
	  (also in: "nightly_win_rel" job)
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #15 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	projectroot.test_record_all__rmw_cyclonedds_cpp

Issue #16 in gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #17 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 3 out of 3 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #18 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #19 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	rclpy.test_waitable.xunit.missing_result
	rclpy.test_wait_for_message.xunit.missing_result
	rclpy.test_timer.xunit.missing_result
	rclpy.test_time_source.xunit.missing_result
	rclpy.test_service_introspection.xunit.missing_result
	rclpy.test_service.xunit.missing_result
	rclpy.test_rosout_subscription.xunit.missing_result
	rclpy.test_rate.xunit.missing_result
	rclpy.test_qos_overriding_options.xunit.missing_result
	projectroot.test_waitable
	projectroot.test_wait_for_message
	projectroot.test_time_source
	projectroot.test_service_introspection
	projectroot.test_rosout_subscription
	projectroot.test_rate
	projectroot.test_qos_overriding_options
	  (also in: "nightly_win_rel" job)

Issue #20 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	rclpy.test_node.xunit.missing_result
	projectroot.test_node

Issue #21 in gz_sim-ci-gz-sim7-focal-amd64
- Failing 66.67% in 2 out of 3 builds
	UtilTest.ResolveSdfWorldFile
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-focal-amd64" job)
	  (also in: "gz_sim-7-win" job)

Issue #22 in nightly_linux_repeated
- Failing 12.5% in 2 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_multiple_nodes_launch_test.launch_testing_examples.check_multiple_nodes_launch_test (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)

Issue #23 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]

Issue #24 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	projectroot.test_publisher (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)

Issue #25 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	ros2cli.ros2cli.test.test_strategy.test_with_no_daemon_running[3-3]
	rclpy.test_qos_event.xunit.missing_result
	rclpy.test_publisher.xunit.missing_result
	rclpy.test_parameter_service.xunit.missing_result
	rclpy.test_parameter_event_handler.xunit.missing_result
	rclpy.test_parameter_client.xunit.missing_result
	projectroot.test_qos_event
	projectroot.test_parameter_service
	projectroot.test_parameter_event_handler
	projectroot.test_parameter_client

Issue #26 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #27 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #28 in Hci__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #29 in gz_fuel_tools-8-win
- Failing 20.0% in 1 out of 5 builds
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-8-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)

Issue #30 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_ImageDisplay_TEST.test_ran

Issue #31 in gz_sim-8-win
- Failing 14.29% in 1 out of 7 builds
	ServerRepeat/ServerFixture.RunNonBlockingMultiple/0
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #32 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	RgbdCameraTest.RgbdCameraBox

Issue #33 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/FuelCachedServer.CachedFuelWorld/0
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #34 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Util_TEST.test_ran
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)

Issue #35 in gz_transport-ign-transport8-win
- Failing 33.33% in 1 out of 3 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #36 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #37 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_server
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #38 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #39 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #40 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_client
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #41 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	rcl.TestGraphFixture.test_graph_guard_condition_trigger_check
	projectroot.test.test_graph__rmw_cyclonedds_cpp

Issue #42 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #43 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_service
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #44 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_publisher (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)

Issue #45 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #46 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_guard_condition (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)

Issue #47 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #48 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #49 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	rosidl_typesupport_c.test_service_type_support.gtest.missing_result
	rosidl_typesupport_c.test_message_type_support.gtest.missing_result
	rcutils.test_time.gtest.missing_result
	rcutils.test_strerror.gtest.missing_result
	rcutils.test_logging_custom_env2.gtest.missing_result
	rcutils.test_logging_custom_env.gtest.missing_result
	rcutils.test_filesystem.gtest.missing_result
	rclcpp_lifecycle.test_transition_wrapper.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_node.gtest.missing_result
	rclcpp_action.test_server_goal_handle.gtest.missing_result
	rclcpp_action.test_server.gtest.missing_result
	rclcpp_action.test_client.gtest.missing_result
	rclcpp.test_utilities.gtest.missing_result
	rclcpp.test_timer.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_subscription.gtest.missing_result
	rclcpp.test_storage_policy_common.gtest.missing_result
	rclcpp.test_static_single_threaded_executor.gtest.missing_result
	rclcpp.test_service.gtest.missing_result
	rclcpp.test_qos_event__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_qos_event__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_publisher.gtest.missing_result
	rclcpp.test_node_options.gtest.missing_result
	rclcpp.test_node_interfaces__node_waitables.gtest.missing_result
	rclcpp.test_node_interfaces__node_topics.gtest.missing_result
	rclcpp.test_node_interfaces__node_timers.gtest.missing_result
	rclcpp.test_node_interfaces__node_services.gtest.missing_result
	rclcpp.test_node_interfaces__node_parameters.gtest.missing_result
	rclcpp.test_node_interfaces__node_graph.gtest.missing_result
	rclcpp.test_node_interfaces__node_base.gtest.missing_result
	rclcpp.test_node.gtest.missing_result
	rclcpp.test_loaned_message.gtest.missing_result
	rclcpp.test_init_options.gtest.missing_result
	rclcpp.test_guard_condition.gtest.missing_result
	rclcpp.test_graph_listener.gtest.missing_result
	rclcpp.test_generic_client.gtest.missing_result
	rclcpp.test_expand_topic_or_service_name.gtest.missing_result
	rclcpp.test_exceptions.gtest.missing_result
	rclcpp.test_client_common.gtest.missing_result
	rclcpp.test_client.gtest.missing_result
	rcl_yaml_param_parser.test_parser.gtest.missing_result
	rcl_yaml_param_parser.test_parse.gtest.missing_result
	rcl.test_wait__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_wait__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_validate_enclave_name.gtest.missing_result
	rcl.test_subscription__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_subscription__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_security.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_node__rmw_fastrtps_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_node__rmw_cyclonedds_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_logging.gtest.missing_result
	rcl.test_init__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_init__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_expand_topic_name.gtest.missing_result
	rcl.test_domain_id.gtest.missing_result
	rcl.test_context__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_context__rmw_cyclonedds_cpp.gtest.missing_result
	projectroot.test_transition_wrapper
	projectroot.test_time
	projectroot.test_strerror
	projectroot.test_service_type_support
	projectroot.test_server_goal_handle
	projectroot.test_parser
	projectroot.test_parse
	projectroot.test_message_type_support
	projectroot.test_logging_custom_env2
	projectroot.test_logging_custom_env
	projectroot.test_filesystem
	projectroot.test.test_wait__rmw_fastrtps_cpp
	projectroot.test.test_wait__rmw_cyclonedds_cpp
	projectroot.test.test_validate_enclave_name
	projectroot.test.test_subscription__rmw_fastrtps_cpp
	projectroot.test.test_subscription__rmw_cyclonedds_cpp
	projectroot.test.test_service_event_publisher__rmw_fastrtps_cpp
	projectroot.test.test_service_event_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_service__rmw_fastrtps_cpp
	projectroot.test.test_service__rmw_cyclonedds_cpp
	projectroot.test.test_security
	projectroot.test.test_rmw_impl_id_check_func__rmw_fastrtps_cpp
	projectroot.test.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp
	projectroot.test.test_publisher__rmw_fastrtps_cpp
	projectroot.test.test_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_node__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_node__rmw_cyclonedds_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_logging
	projectroot.test.test_init__rmw_fastrtps_cpp
	projectroot.test.test_init__rmw_cyclonedds_cpp
	projectroot.test.test_guard_condition__rmw_fastrtps_cpp
	projectroot.test.test_guard_condition__rmw_cyclonedds_cpp
	projectroot.test.test_expand_topic_name
	projectroot.test.test_domain_id
	projectroot.test.test_context__rmw_fastrtps_cpp
	projectroot.test.test_context__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_utilities
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_qos_event__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_qos_event__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_node_options
	projectroot.test.rclcpp.test_node_interfaces__node_waitables
	projectroot.test.rclcpp.test_node_interfaces__node_topics
	projectroot.test.rclcpp.test_node_interfaces__node_timers
	projectroot.test.rclcpp.test_node_interfaces__node_services
	projectroot.test.rclcpp.test_node_interfaces__node_parameters
	projectroot.test.rclcpp.test_node_interfaces__node_graph
	projectroot.test.rclcpp.test_node_interfaces__node_base
	projectroot.test.rclcpp.test_loaned_message
	projectroot.test.rclcpp.test_init_options
	projectroot.test.rclcpp.test_generic_client
	projectroot.test.rclcpp.test_expand_topic_or_service_name
	projectroot.test.rclcpp.test_exceptions
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #50 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_client_common
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #51 in nightly_linux-aarch64_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_subscription
	projectroot.test.rclcpp.test_client
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #52 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #53 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #54 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #55 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_tf2_geometry_msgs
	projectroot.display_factory_test

Issue #56 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.test_local_parameters.local_synchronous_repeated

Issue #57 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #58 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_communication.TestRequesterReplierAfterShutdown.test_processes_finished_gracefully[Empty]
	  (also in: "nightly_win_deb" job)

Issue #59 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[3-3]
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #60 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)

Issue #61 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_communication.TestRequesterReplier.test_requester_finishes_in_a_finite_amount_of_time[Empty]
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	rclpy.test_messages.xunit.missing_result
	rclpy.test_logging_service.xunit.missing_result
	projectroot.test_messages
	projectroot.test_logging_service
	projectroot.test_client_scope_cpp__rmw_fastrtps_cpp
	projectroot.gtest_timeout_subscriber__rmw_fastrtps_cpp
	launch_testing_ros.test.examples.talker_listener_launch_test.talker_listener_launch_test
	launch_testing_ros.test.examples.check_msgs_launch_test.check_msgs_launch_test

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 17, 2024 •
edited
Loading

Cleanup report: Jobs failing for a long time

Note: I skipepd Rci jobs because we migrated to noble today, so we expect some change in terms of greeness (and also we lose track of test regressions in the internal database

Gz-plugin1: (since 2023-12-04)

CMake warning "Policy CMP0153 is not set" needs to be fixed - 📚 Reported in https://github.com/gazebosim/gz-plugin/issues/144

Reference build: https://build.osrfoundation.org/job/gz_plugin-ci-ign-plugin1-homebrew-amd64/37/

Output:

CMake Warning (dev) at /usr/local/share/cmake/ignition-cmake2/cmake2/IgnSetCompilerFlags.cmake:77 (exec_program):
  Policy CMP0153 is not set: The exec_program command should not be called.
  Run "cmake --help-policy CMP0153" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  Use execute_process() instead.
Call Stack (most recent call first):
  /usr/local/share/cmake/ignition-cmake2/cmake2/IgnSetCompilerFlags.cmake:33 (ign_setup_unix)
  /usr/local/share/cmake/ignition-cmake2/cmake2/IgnConfigureBuild.cmake:88 (ign_set_compiler_flags)
  CMakeLists.txt:58 (ign_configure_build)
This warning is for project developers.  Use -Wno-dev to suppress it.

Gz-sim-gazebo6-homebrew: (since 2024-02-23)

failing consistently because of testFixture_TEST regression, and some flakies

Also, there are some flaky tests that sometimes affect this job.

This issue is still affecting consistently to gz-sim6-homebrew (reference build)

Log output (almost the same as before):

Note the module error message:

273:     from ignition.math import Vector3d
273: E   ModuleNotFoundError: No module named 'ignition.math'

test 273
        Start 273: testFixture_TEST.py

273: Test command: /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12 "-m" "pytest" "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/gz-sim/python/test/testFixture_TEST.py" "--junitxml" "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/build/test_results/UNIT_testFixture_TEST.xml"
273: Working Directory: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/build/python
273: Environment variables: 
273:  PYTHONPATH=/usr/local/Cellar/gz-sim6/HEAD/lib/python/
273:  LD_LIBRARY_PATH=/usr/local/Cellar/gz-sim6/HEAD/lib:
273: Test timeout computed to be: 1500
273: ============================= test session starts ==============================
273: platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
273: rootdir: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64
273: collected 0 items / 1 error
273: 
273: ==================================== ERRORS ====================================
273: ___________ ERROR collecting gz-sim/python/test/testFixture_TEST.py ____________
273: ImportError while importing test module '/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/gz-sim/python/test/testFixture_TEST.py'.
273: Hint: make sure your test modules/packages have valid Python names.
273: Traceback:
273: /usr/local/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
273:     return _bootstrap._gcd_import(name[level:], package, level)
273: ../../gz-sim/python/test/testFixture_TEST.py:20: in <module>
273:     from ignition.math import Vector3d
273: E   ModuleNotFoundError: No module named 'ignition.math'
273: - generated xml file: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/build/test_results/UNIT_testFixture_TEST.xml -
273: =========================== short test summary info ============================
273: ERROR ../../gz-sim/python/test/testFixture_TEST.py
273: !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
273: =============================== 1 error in 0.60s ===============================
273/273 Test #273: testFixture_TEST.py .....................................***Failed    0.90 sec

This issue was solved in other distributions. We're missing explicitly this fortress

Gz-physics6-win:

(since 2024-03-06) Failing because of truncation warning

Reference build:

Output:

C:\J\workspace\gz_physics-6-win\ws\gz-physics\bullet-featherstone\src\Base.cc(51,46): warning C4305: '=': truncation from 'double' to 'btScalar' [C:\J\workspace\gz_physics-6-win\ws\build\gz-physics6\bullet-featherstone\gz-physics6-bullet-featherstone-plugin.vcxproj]

It was a fix here: gazebosim/gz-physics#602 (comment), but the warning is still popping up.

Gz-sim-gazebo6-focal:

(since 2024-03-17) pending investigation 🕵️

Has been accumulating consistent test regressions

Opening external investigation

Gz-physics-main Homebrew

: (since 2024-03-22) Test regression - 📚 Reported in https://github.com/gazebosim/gz-physics/issues/627

Reference builds:

Failing test regression: JointFeaturesDetachTest/0.JointDetach

/Users/jenkins/jenkins-agent/workspace/gz_physics-ci-main-homebrew-amd64/gz-physics/test/common_test/joint_features.cc:969
The difference between 0.0 and upperLinkLinearVelocity.X() is 0.00035244933516466117, which exceeds 1e-6, where
0.0 evaluates to 0,
upperLinkLinearVelocity.X() evaluates to -0.00035244933516466117, and
1e-6 evaluates to 9.9999999999999995e-07.

Might be related to gazebosim/gz-physics#620

Gz-physics6 Homebrew

: (since 2024-03-23) Test regression - 📚 Reported in https://github.com/gazebosim/gz-physics/issues/626

Reference builds:

Failing test regression: DetachableJointTest/0.CorrectAttachmentPoints

/Users/jenkins/jenkins-agent/workspace/gz_physics-ci-gz-physics6-homebrew-amd64/gz-physics/test/common_test/detachable_joint.cc:190
The difference between 0.15 and frameDataC1L1.pose.translation().z() is 0.10148802840906343, which exceeds 1e-2, where
0.15 evaluates to 0.14999999999999999,
frameDataC1L1.pose.translation().z() evaluates to 0.048511971590936565, and
1e-2 evaluates to 0.01.

Test hasn't been modified for 6 months. It was added in: gazebosim/gz-physics#530

@github-actions GitHub Actions
Copy link
Contributor Author

Apr 18

Build regressions 
nightly_linux_coverage|2344|2024-04-18 05:15:11.146000|Number of failed test exceeds the threshold|coverage analysis
nightly_linux_iron_coverage|400|2024-04-18 06:06:48.111000|Jenkins channel closing down (Agent disconnected)|test
gz-math8-debbuilder|263|2024-04-18 10:01:30.322000|Failed to compile and install build|install build dependencies from debian/control
gz-msgs11-debbuilder|263|2024-04-18 10:48:39.377000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|258|2024-04-17 12:56:29.483000|Failed to compile and install build|install build dependencies from debian/control
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 14 out of 14 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #2 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 14 out of 14 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 14 out of 14 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #4 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 14 out of 14 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #5 in nightly_win_deb
- Failing 66.67% in 10 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #6 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 28.57% in 4 out of 14 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #7 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 80.0% in 4 out of 5 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #8 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #9 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesDetachTest/0.JointDetach
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #10 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #11 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #12 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 100.0% in 4 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent
	ShaderParamTest.ShaderParam
	ServerRepeat/FollowActorTest.PublishCmd/0
	ModelPhotoShootTest.ModelPhotoShootRandomJoints
	ModelPhotoShootTest.ModelPhotoShootDefaultJoints

Issue #13 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 4 out of 4 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #14 in nightly_linux_repeated
- Failing 25.0% in 4 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #15 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	projectroot.test_timer
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_iron_coverage" job)

Issue #16 in gz_sim-ci-gz-sim7-focal-amd64
- Failing 66.67% in 2 out of 3 builds
	UtilTest.ResolveSdfWorldFile
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-focal-amd64" job)
	  (also in: "gz_sim-7-win" job)

Issue #17 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_server
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #18 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #19 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_client
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #20 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #21 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_guard_condition (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)

Issue #22 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	rosidl_typesupport_c.test_service_type_support.gtest.missing_result
	rosidl_typesupport_c.test_message_type_support.gtest.missing_result
	rcutils.test_time.gtest.missing_result
	rcutils.test_strerror.gtest.missing_result
	rcutils.test_logging_custom_env2.gtest.missing_result
	rcutils.test_logging_custom_env.gtest.missing_result
	rcutils.test_filesystem.gtest.missing_result
	rclcpp_lifecycle.test_transition_wrapper.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_node.gtest.missing_result
	rclcpp_action.test_server_goal_handle.gtest.missing_result
	rclcpp_action.test_server.gtest.missing_result
	rclcpp_action.test_client.gtest.missing_result
	rclcpp.test_utilities.gtest.missing_result
	rclcpp.test_timer.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_subscription.gtest.missing_result
	rclcpp.test_storage_policy_common.gtest.missing_result
	rclcpp.test_static_single_threaded_executor.gtest.missing_result
	rclcpp.test_service.gtest.missing_result
	rclcpp.test_qos_event__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_qos_event__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_publisher.gtest.missing_result
	rclcpp.test_node_options.gtest.missing_result
	rclcpp.test_node_interfaces__node_waitables.gtest.missing_result
	rclcpp.test_node_interfaces__node_topics.gtest.missing_result
	rclcpp.test_node_interfaces__node_timers.gtest.missing_result
	rclcpp.test_node_interfaces__node_services.gtest.missing_result
	rclcpp.test_node_interfaces__node_parameters.gtest.missing_result
	rclcpp.test_node_interfaces__node_graph.gtest.missing_result
	rclcpp.test_node_interfaces__node_base.gtest.missing_result
	rclcpp.test_node.gtest.missing_result
	rclcpp.test_loaned_message.gtest.missing_result
	rclcpp.test_init_options.gtest.missing_result
	rclcpp.test_guard_condition.gtest.missing_result
	rclcpp.test_graph_listener.gtest.missing_result
	rclcpp.test_generic_client.gtest.missing_result
	rclcpp.test_expand_topic_or_service_name.gtest.missing_result
	rclcpp.test_exceptions.gtest.missing_result
	rclcpp.test_client_common.gtest.missing_result
	rclcpp.test_client.gtest.missing_result
	rcl_yaml_param_parser.test_parser.gtest.missing_result
	rcl_yaml_param_parser.test_parse.gtest.missing_result
	rcl.test_wait__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_wait__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_validate_enclave_name.gtest.missing_result
	rcl.test_subscription__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_subscription__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_security.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_node__rmw_fastrtps_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_node__rmw_cyclonedds_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_logging.gtest.missing_result
	rcl.test_init__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_init__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_expand_topic_name.gtest.missing_result
	rcl.test_domain_id.gtest.missing_result
	rcl.test_context__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_context__rmw_cyclonedds_cpp.gtest.missing_result
	projectroot.test_transition_wrapper
	projectroot.test_time
	projectroot.test_strerror
	projectroot.test_service_type_support
	projectroot.test_server_goal_handle
	projectroot.test_parser
	projectroot.test_parse
	projectroot.test_message_type_support
	projectroot.test_logging_custom_env2
	projectroot.test_logging_custom_env
	projectroot.test_filesystem
	projectroot.test.test_wait__rmw_fastrtps_cpp
	projectroot.test.test_wait__rmw_cyclonedds_cpp
	projectroot.test.test_validate_enclave_name
	projectroot.test.test_subscription__rmw_fastrtps_cpp
	projectroot.test.test_subscription__rmw_cyclonedds_cpp
	projectroot.test.test_service_event_publisher__rmw_fastrtps_cpp
	projectroot.test.test_service_event_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_service__rmw_fastrtps_cpp
	projectroot.test.test_service__rmw_cyclonedds_cpp
	projectroot.test.test_security
	projectroot.test.test_rmw_impl_id_check_func__rmw_fastrtps_cpp
	projectroot.test.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp
	projectroot.test.test_publisher__rmw_fastrtps_cpp
	projectroot.test.test_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_node__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_node__rmw_cyclonedds_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_logging
	projectroot.test.test_init__rmw_fastrtps_cpp
	projectroot.test.test_init__rmw_cyclonedds_cpp
	projectroot.test.test_guard_condition__rmw_fastrtps_cpp
	projectroot.test.test_guard_condition__rmw_cyclonedds_cpp
	projectroot.test.test_expand_topic_name
	projectroot.test.test_domain_id
	projectroot.test.test_context__rmw_fastrtps_cpp
	projectroot.test.test_context__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_utilities
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_qos_event__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_node_options
	projectroot.test.rclcpp.test_node_interfaces__node_waitables
	projectroot.test.rclcpp.test_node_interfaces__node_topics
	projectroot.test.rclcpp.test_node_interfaces__node_timers
	projectroot.test.rclcpp.test_node_interfaces__node_services
	projectroot.test.rclcpp.test_node_interfaces__node_parameters
	projectroot.test.rclcpp.test_node_interfaces__node_graph
	projectroot.test.rclcpp.test_node_interfaces__node_base
	projectroot.test.rclcpp.test_loaned_message
	projectroot.test.rclcpp.test_init_options
	projectroot.test.rclcpp.test_generic_client
	projectroot.test.rclcpp.test_expand_topic_or_service_name
	projectroot.test.rclcpp.test_exceptions
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #23 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_client_common
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #24 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_client
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #25 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	rclcpp.test_executor.gtest.missing_result
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #26 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_subscription
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #27 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_service
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #28 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_qos_event__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #29 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_publisher (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)

Issue #30 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #31 in nightly_win_rel
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #32 in nightly_linux-aarch64_debug
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #33 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	ros2cli.ros2cli.test.test_strategy.test_with_no_daemon_running[2-3]

Issue #34 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp

Issue #35 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	projectroot.test_action_client
	  (also in: "nightly_win_rel" job)

Issue #36 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_linux_repeated" job)

Issue #37 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	demo_nodes_cpp.TestExecutablesTutorialAfterShutdown.test_last_process_exit_code
	  (also in: "nightly_win_deb" job)

Issue #38 in gz_fuel_tools-8-win
- Failing 20.0% in 1 out of 5 builds
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-8-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)

Issue #39 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_ImageDisplay_TEST.test_ran

Issue #40 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	RgbdCameraTest.RgbdCameraBox

Issue #41 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/FuelCachedServer.CachedFuelWorld/0
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #42 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Util_TEST.test_ran
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)

Issue #43 in gz_sim-ci-main-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	<init>.ServerRepeat/SceneBroadcasterTest
	<init>.NetworkHandshake

Issue #44 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestNNodes.test_10_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #45 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2bag.test.test_record_qos_profiles.test_record_qos_profiles

Issue #46 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #47 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #48 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)

Issue #49 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestService.server_qos
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #50 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestSubscription.on_new_message_callback
	rclcpp.TestService.on_new_request_callback
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #51 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestQosEvent.test_on_new_event_callback
	rclcpp.TestQosEvent.test_invalid_on_new_event_callback
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #52 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestWaitForAllAckedWithParm/TestPublisherWaitForAllAcked.check_wait_for_all_acked_with_QosPolicy/2 ((88-byte object <01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 01-00 00-00 02-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 
	rclcpp.TestWaitForAllAckedWithParm/TestPublisherWaitForAllAcked.check_wait_for_all_acked_with_QosPolicy/1 ((88-byte object <01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 02-00 00-00 02-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 
	rclcpp.TestWaitForAllAckedWithParm/TestPublisherWaitForAllAcked.check_wait_for_all_acked_with_QosPolicy/0 ((88-byte object <01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 01-00 00-00 02-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 
	rclcpp.TestPublisherThrows/TestPublisherInvalidIntraprocessQos.test_publisher_throws/keep_all_qos (keep_all_qos)                     
	rclcpp.TestPublisherSub.construction_and_destruction
	rclcpp.TestPublisher.serialized_message_publish
	rclcpp.TestPublisher.run_event_handlers
	rclcpp.TestPublisher.rcl_publisher_init_error
	rclcpp.TestPublisher.rcl_publisher_get_subscription_count_publisher_invalid
	rclcpp.TestPublisher.rcl_publisher_get_rmw_handle_error
	rclcpp.TestPublisher.rcl_publisher_get_options_error
	rclcpp.TestPublisher.rcl_publisher_get_gid_for_publisher_error
	rclcpp.TestPublisher.rcl_publisher_get_actual_qos_error
	rclcpp.TestPublisher.rcl_publisher_fini_error
	rclcpp.TestPublisher.publishers_equal_rmw_compare_gids_error
	rclcpp.TestPublisher.lowest_available_ipm_capacity
	rclcpp.TestPublisher.intra_process_transient_local
	rclcpp.TestPublisher.intra_process_publish_failures
	rclcpp.TestPublisher.inter_process_publish_failures
	rclcpp.TestPublisher.get_network_flow_endpoints_errors
	rclcpp.TestPublisher.do_loaned_message_publish_error
	rclcpp.TestPublisher.default_incompatible_qos_callback
	rclcpp.TestPublisher.check_wait_for_all_acked_return
	  (also in: "nightly_win_rep" job)

Issue #53 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.PerTimerType/TestTimer.test_simple_cancel/wall_timer (4-byte object <00-00 00-00>)                     
	  (also in: "nightly_win_rep" job)

Issue #54 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestTimeSource.callbacks
	projectroot.test.rclcpp.test_time_source
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #55 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_time (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #56 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_subscription_topic_statistics
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #57 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestServiceIntrospection.service_introspection_enable_disable_events
	rclcpp.TestServiceIntrospection.service_introspection_enable_disable_event_content
	projectroot.test.rclcpp.test_service_introspection
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #58 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_publisher_with_type_adapter
	  (also in: "nightly_win_deb" job)

Issue #59 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestUtilities.wait_for_message_twice_one_sub
	rclcpp.TestUtilities.wait_for_message_indefinitely
	rclcpp.TestUtilities.wait_for_message
	rclcpp.TestTimeSource.reattach
	rclcpp.TestTimeSource.parameter_activation
	rclcpp.TestTimeSource.no_pre_jump_callback
	rclcpp.TestTimeSource.detachUnattached
	rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic
	rclcpp.TestTimeSource.clock
	rclcpp.TestTimeSource.check_use_clock_thread_value
	rclcpp.TestTimeSource.check_clock_thread_status
	rclcpp.TestTimeSource.callback_handler_erasure
	rclcpp.TestTimeSource.ROS_time_valid_wall_time
	rclcpp.TestTimeSource.ROS_time_valid_sim_time
	rclcpp.TestTimeSource.ROS_time_valid_attach_detach
	rclcpp.TestTimeSource.ROS_invalid_sim_time
	rclcpp.TestLoggerService.check_connect_get_logger_service
	rclcpp.TestInterfaceTraits.has_node_base_interface
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	projectroot.test.rclcpp.test_wait_for_message
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits
	  (also in: "nightly_win_rep" job)

Issue #60 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.test_intra_process_within_one_node.type_adapted_messages_unique_pub_received_by_intra_process_subscription
	rclcpp.test_intra_process_within_one_node.type_adapted_messages_unique_ptr_ros_message_pub_received_by_intra_process_subscription
	rclcpp.test_intra_process_within_one_node.type_adapted_messages_ros_message_ref_pub_received_by_intra_process_subscription
	rclcpp.test_intra_process_within_one_node.type_adapted_messages_const_ref_pub_received_by_intra_process_subscription
	rclcpp.TestTypeSupport.test_service_ts_set_params_srv
	rclcpp.TestTypeSupport.test_service_ts_set_params_atomically_srv
	rclcpp.TestTypeSupport.test_service_ts_list_params_srv
	rclcpp.TestTypeSupport.test_service_ts_get_params_srv_type
	rclcpp.TestTypeSupport.test_service_ts_get_params_srv
	rclcpp.TestTypeSupport.test_service_ts_get_parameters_types_srv
	rclcpp.TestTypeSupport.test_service_ts_describe_params_srv
	rclcpp.TestTypeSupport.basic_getters
	rclcpp.TestSubscriptionPublisherCount/one_context_test.increasing_and_decreasing_counts
	rclcpp.TestSubscription.various_creation_signatures
	rclcpp.TestSubscription.take_serialized
	rclcpp.TestSubscription.take
	rclcpp.TestSubscription.rcl_take_type_erased_error
	rclcpp.TestSubscription.rcl_take_serialized_message_error
	rclcpp.TestSubscription.rcl_subscription_init_error
	rclcpp.TestSubscription.rcl_subscription_get_publisher_count_error
	rclcpp.TestSubscription.rcl_subscription_get_actual_qos_error
	rclcpp.TestSubscription.rcl_subscription_fini_error
	rclcpp.TestSubscription.on_new_intra_process_message_callback
	rclcpp.TestSubscription.handle_loaned_message
	rclcpp.TestSubscription.get_network_flow_endpoints_errors
	rclcpp.TestSubscription.construction_and_destruction
	rclcpp.TestSubscription.check_type_adapted_messages_are_received_by_intra_process_subscription
	rclcpp.TestSubscription.check_type_adapted_messages_are_received_by_inter_process_subscription
	rclcpp.TestSubscription.callback_bind
	rclcpp.TestServiceIntrospection.service_introspection_nominal
	rclcpp.TestService.take_request
	rclcpp.TestService.server_qos_depth
	rclcpp.TestService.send_response
	rclcpp.TestService.rcl_service_response_publisher_get_actual_qos_error
	rclcpp.TestService.rcl_service_request_subscription_get_actual_qos_error
	rclcpp.TestService.construction_and_destruction_rcl_errors
	rclcpp.TestService.construction_and_destruction
	rclcpp.TestService.basic_public_getters
	rclcpp.TestSerializedMessageAllocator.borrow_from_subscription
	rclcpp.TestQosParameters.declare_with_id
	rclcpp.TestQosParameters.declare_with_callback
	rclcpp.TestQosParameters.declare_qos_subscription_parameters
	rclcpp.TestQosParameters.declare_no_parameters_interface
	rclcpp.TestQosParameters.declare
	rclcpp.TestQosEvent.test_subscription_constructor
	rclcpp.TestQosEvent.test_sub_matched_event_by_set_event_callback
	rclcpp.TestQosEvent.test_sub_matched_event_by_option_event_callback
	rclcpp.TestQosEvent.test_publisher_constructor
	rclcpp.TestQosEvent.test_pub_matched_event_by_set_event_callback
	rclcpp.TestQosEvent.test_pub_matched_event_by_option_event_callback
	rclcpp.TestQosEvent.test_default_incompatible_qos_callbacks
	rclcpp.TestQosEvent.execute
	rclcpp.TestQosEvent.construct_destruct_rcl_error
	rclcpp.TestQosEvent.add_to_wait_set
	rclcpp.TestPublisherSubscriptionCount/two_subscriptions_intraprocess_comm.increasing_and_decreasing_counts
	rclcpp.TestPublisher.various_creation_signatures
	rclcpp.TestPublisher.test_large_message_unique
	rclcpp.TestPublisher.test_large_message_constref
	rclcpp.TestPublisher.conversion_exception_is_passed_up
	rclcpp.TestPublisher.construction_and_destruction
	rclcpp.TestPublisher.check_type_adapted_message_is_sent_and_received
	rclcpp.TestPublisher.basic_getters
	rclcpp.TestParameterService.set_parameters_atomically
	rclcpp.TestParameterService.set_parameters
	rclcpp.TestParameterService.list_parameters
	rclcpp.TestParameterService.get_parameters
	rclcpp.TestParameterService.get_parameter_types
	rclcpp.TestParameterService.describe_parameters
	rclcpp.TestParameterClient.sync_parameter_load_parameters
	rclcpp.TestParameterClient.sync_parameter_is_ready
	rclcpp.TestParameterClient.sync_parameter_get_parameters_allow_undeclared
	rclcpp.TestParameterClient.sync_parameter_get_parameters
	rclcpp.TestParameterClient.sync_parameter_get_parameter_types_allow_undeclared
	rclcpp.TestParameterClient.sync_parameter_get_parameter_types
	rclcpp.TestParameterClient.sync_parameter_get_parameter
	rclcpp.TestParameterClient.sync_parameter_event_subscription
	rclcpp.TestParameterClient.sync_parameter_describe_parameters_allow_undeclared
	rclcpp.TestParameterClient.sync_parameter_describe_parameters
	rclcpp.TestParameterClient.sync_parameter_delete_parameters
	rclcpp.TestParameterClient.sync_construction_and_destruction
	rclcpp.TestParameterClient.async_parameter_set_parameters_atomically
	rclcpp.TestParameterClient.async_parameter_load_parameters_from_map
	rclcpp.TestParameterClient.async_parameter_load_parameters_complicated_regex
	rclcpp.TestParameterClient.async_parameter_load_parameters
	rclcpp.TestParameterClient.async_parameter_load_no_valid_parameter
	rclcpp.TestParameterClient.async_parameter_load_from_map_no_valid_parameter
	rclcpp.TestParameterClient.async_parameter_list_parameters
	rclcpp.TestParameterClient.async_parameter_get_parameters_allow_undeclared
	rclcpp.TestParameterClient.async_parameter_get_parameters
	rclcpp.TestParameterClient.async_parameter_get_parameter_types_allow_undeclared
	rclcpp.TestParameterClient.async_parameter_get_parameter_types
	rclcpp.TestParameterClient.async_parameter_event_subscription
	rclcpp.TestParameterClient.async_parameter_describe_parameters_allow_undeclared
	rclcpp.TestParameterClient.async_parameter_describe_parameters
	rclcpp.TestParameterClient.async_parameter_delete_parameters
	rclcpp.TestParameterClient.async_construction_and_destruction
	rclcpp.TestNode.SameParameterDifferentNode
	rclcpp.TestNode.RegisterParameterCallback
	rclcpp.TestNode.MultipleParameterCallbacks
	rclcpp.TestNode.LastInFirstCallForParameterEventCallbacks
	rclcpp.TestNode.LastInFirstCallForParameterCallbacks
	rclcpp.TestNode.GetParametersFromEvent
	rclcpp.TestNode.GetParameterFromEvent
	rclcpp.TestNode.EventCallback
	rclcpp.TestLogger.get_node_logger
	rclcpp.TestFindWeakNodes.allocator_strategy_with_weak_nodes
	rclcpp.TestFindWeakNodes.allocator_strategy_no_weak_nodes
	rclcpp.TestExternallyDefinedServices.extern_defined_uninitialized
	rclcpp.TestExternallyDefinedServices.extern_defined_initialized
	rclcpp.TestExternallyDefinedServices.extern_defined_destructor
	rclcpp.TestExternallyDefinedServices.default_behavior
	rclcpp.TestClockSleep.bad_clock_type
	rclcpp.RclcppGenericNodeFixture.publisher_and_subscriber_work
	rclcpp.RclcppGenericNodeFixture.publish_loaned_msg_work
	rclcpp.RclcppGenericNodeFixture.generic_subscription_uses_qos
	rclcpp.RclcppGenericNodeFixture.generic_subscription_different_callbacks
	rclcpp.RclcppGenericNodeFixture.generic_publisher_uses_qos
	projectroot.test.rclcpp.test_type_support
	projectroot.test.rclcpp.test_subscription_with_type_adapter
	projectroot.test.rclcpp.test_subscription_publisher_with_same_type_adapter
	projectroot.test.rclcpp.test_subscription_publisher_count_api
	projectroot.test.rclcpp.test_serialized_message_allocator
	projectroot.test.rclcpp.test_qos_parameters
	projectroot.test.rclcpp.test_publisher_subscription_count_api (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	projectroot.test.rclcpp.test_parameter_service
	projectroot.test.rclcpp.test_parameter_event_handler
	projectroot.test.rclcpp.test_parameter_client
	projectroot.test.rclcpp.test_logger
	projectroot.test.rclcpp.test_generic_pubsub__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_find_weak_nodes
	projectroot.test.rclcpp.test_externally_defined_services

Issue #61 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestIntraProcessWaitable.test_that_waitable_stays_ready_after_second_wait
	rclcpp.TestCallbackGroup.valid_callback_group
	rclcpp.TestCallbackGroup.invalid_callback_group
	projectroot.test.rclcpp.test_intra_process_waitable
	projectroot.test.rclcpp.test_executors_callback_group_behavior
	  (also in: "nightly_win_rep" job)

Issue #62 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestWaitSet.add_remove_wait
	rclcpp.TestWaitSet.add_guard_condition_to_two_different_wait_set
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer2CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer1CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT1Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBackTimerCancelThenResetBehavior
	rclcpp.TestThreadSafeStorage.wait_waitable
	rclcpp.TestThreadSafeStorage.wait_timer
	rclcpp.TestThreadSafeStorage.wait_subscription
	rclcpp.TestThreadSafeStorage.wait_client_service
	rclcpp.TestThreadSafeStorage.iterables_construct_destruct
	rclcpp.TestThreadSafeStorage.default_construct_destruct
	rclcpp.TestThreadSafeStorage.add_remove_out_of_scope
	rclcpp.TestThreadSafeStorage.add_remove_nullptr
	rclcpp.TestThreadSafeStorage.add_remove_dynamically
	rclcpp.TestSubscriptionTopicStatisticsFixture.test_invalid_publish_period
	rclcpp.TestSubscriptionOptions.topic_statistics_options_node_default_mode
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_resize_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_fini_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_clear_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_timer_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_subscription_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_service_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_guard_condition_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_client_error
	rclcpp.TestStoragePolicyCommon.add_waitable_error
	rclcpp.TestStaticStorage.wait_waitable
	rclcpp.TestStaticStorage.wait_timer
	rclcpp.TestStaticStorage.wait_subscription
	rclcpp.TestStaticStorage.wait_guard_condition
	rclcpp.TestStaticStorage.wait_client_service
	rclcpp.TestStaticStorage.iterables_construct_destruct
	rclcpp.TestStaticStorage.fixed_storage_needs_pruning
	rclcpp.TestStaticSingleThreadedExecutor.add_callback_group_trigger_guard_failed
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_parent_log
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_node_rosout_disabled
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild_hierarchy
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_child_log
	rclcpp.TestMultiThreadedExecutor.timer_over_take
	rclcpp.TestIntraprocessExecutors/SingleThreadedExecutor.testIntraprocessRetrigger
	rclcpp.TestGraphListener.construction_and_destruction
	rclcpp.TestExecutorNotifyWaitable.add_remove_guard_conditions
	rclcpp.TestExecutorEntitiesCollector.add_remove_node
	rclcpp.TestEventsExecutor.run_pub_sub
	rclcpp.TestDynamicStorage.wait_waitable
	rclcpp.TestDynamicStorage.wait_timer
	rclcpp.TestDynamicStorage.wait_subscription
	rclcpp.TestDynamicStorage.wait_client_service
	rclcpp.TestDynamicStorage.iterables_construct_destruct
	rclcpp.TestDynamicStorage.default_construct_destruct
	rclcpp.TestDynamicStorage.add_remove_out_of_scope
	rclcpp.TestDynamicStorage.add_remove_nullptr
	rclcpp.TestDynamicStorage.add_remove_dynamically
	rclcpp.TestContentFilterSubscription.set_content_filter_error
	rclcpp.TestContentFilterSubscription.set_content_filter
	rclcpp.TestContentFilterSubscription.is_cft_enabled
	rclcpp.TestContentFilterSubscription.get_content_filter_error
	rclcpp.TestContentFilterSubscription.get_content_filter
	rclcpp.TestContentFilterSubscription.content_filter_reset
	rclcpp.TestContentFilterSubscription.content_filter_get_later
	rclcpp.TestContentFilterSubscription.content_filter_get_begin
	projectroot.test.rclcpp.test_wait_set
	projectroot.test.rclcpp.test_thread_safe_synchronization
	projectroot.test.rclcpp.test_subscription_options
	projectroot.test.rclcpp.test_static_storage
	projectroot.test.rclcpp.test_rosout_subscription
	projectroot.test.rclcpp.test_multi_threaded_executor
	projectroot.test.rclcpp.test_executors_intraprocess
	projectroot.test.rclcpp.test_executor_notify_waitable
	projectroot.test.rclcpp.test_events_executor
	projectroot.test.rclcpp.test_entities_collector
	projectroot.test.rclcpp.test_dynamic_storage
	  (also in: "nightly_win_rep" job)

Issue #63 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	rcl_action.TestActionServerInitFini.test_action_server_init_fini
	rcl_action.TestActionGraphFixture__rmw_fastrtps_cpp.test_action_get_client_names_and_types_by_node
	rcl_action.TestActionClientWait.test_wait_set_add_action_client
	projectroot.test_wait
	projectroot.test_graph__rmw_fastrtps_cpp
	projectroot.test_action_server
	projectroot.test_action_interaction__rmw_fastrtps_cpp
	projectroot.test_action_communication__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)

Issue #64 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_parameter_server_cpp
	rclpy.test_action_server.xunit.missing_result
	rclpy.rclpy.test.test_timer.test_zero_callback[0.1]
	rcl_action.test_action_interaction__rmw_fastrtps_cpp.gtest.missing_result
	rcl_action.TestActionClientFixture.test_action_server_is_available_maybe_fail
	projectroot.test_parameter_server_cpp__rmw_fastrtps_cpp
	TestActionCommunication__rmw_fastrtps_cpp.test_valid_goal_comm

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 18, 2024 •
edited
Loading

Nightly Linux Coverage | Build regression | Reported 📚

Reference build: https://ci.ros2.org/job/nightly_linux_coverage/2344/

It started failing when the job configuration started using Noble as Ubuntu distro.

The coverage analysis section was the last log output before the agent got killed:

Processing build/test_tf2/CMakeFiles/test_message_filter.dir/test/test_message_filter.cpp.gcda
geninfo: WARNING: using JSON module "JSON::PP" - which is much slower than some alternatives.  Consider installing one of JSON::XS or Cpanel::JSON::XS
geninfo: WARNING: /usr/include/c++/13/bits/atomic_base.h:501: unexecuted block on non-branch line with non-zero hit count.  Use "geninfo --rc geninfo_unexecuted_blocks=1 to set count to zero.
geninfo: ERROR: mismatched end line for _ZN31MessageFilter_noTransforms_Test8TestBodyEv at /home/jenkins-agent/workspace/nightly_linux_coverage/ws/src/ros2/geometry2/test_tf2/test/test_message_filter.cpp:79: 79 -> 100
    (use "geninfo --ignore-errors mismatch ..." to bypass this error)
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/run_ros2_batch.py", line 32, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 154, in main
    return run(args, build_function, blacklisted_package_names=blacklisted_package_names)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 689, in run
    rc = build_function(args, job, colcon_script)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 386, in build_and_test
    process_coverage(args, job)
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 280, in process_coverage
    subprocess.run(cmd, check=True)
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['lcov', '--capture', '--directory', 'build', '--output', 'build/coverage.info']' returned non-zero exit status 1.
Build step 'Execute shell' marked build as failure

Also, we had 3 test regressions:

/usr/bin/colcon test-result --test-result-base "build"
build/ros2topic/pytest.xml: 9 tests, 0 errors, 1 failure, 0 skipped^M
build/rosidl_typesupport_introspection_tests/Testing/20240418-0558/Test.xml: 1 test, 0 errors, 1 failure, 0 skipped
build/rosidl_typesupport_introspection_tests/test_results/rosidl_typesupport_introspection_tests/test_multi_nested_message_introspection.gtest.xml: 6 tests, 0 errors, 1 failure, 0 skipped

Reported in: ros2/ci#770

@github-actions GitHub Actions
Copy link
Contributor Author

Apr 19

Build regressions 
nightly_linux_coverage|2345|2024-04-19 04:27:45.274000|Number of failed test exceeds the threshold|coverage analysis
nightly_linux_release|3048|2024-04-19 08:18:13.591000|Failed to fetch some archives|Build Dockerfile
nightly_linux_release|3047|2024-04-19 05:49:03.858000|Jenkins channel closing down (Agent disconnected)|test
nightly_linux-rhel_release|1867|2024-04-19 04:00:01.125000|Jenkins channel closing down (Agent disconnected)|build
Rci__nightly-performance_ubuntu_noble_amd64|1|2024-04-18 19:17:10.004000|Packages performance_test Failed|build workspace in isolation and install
gz-cmake4-debbuilder|267|2024-04-19 09:23:05.920000|Failed to fetch some archives|see Dockerfile
gz-cmake4-debbuilder|266|2024-04-19 09:17:37.553000|Unknown Reason|see Dockerfile
gz-tools2-debbuilder|1616|2024-04-19 09:54:15.822000|Unknown Reason|see Dockerfile
gz-tools2-debbuilder|1615|2024-04-19 09:41:35.434000|Failed to fetch some archives|see Dockerfile
gz-tools2-debbuilder|1614|2024-04-19 09:28:32.203000|Unknown Reason|see Dockerfile
gz-transport14-debbuilder|260|2024-04-18 12:36:05.081000|Failed to compile and install build|install build dependencies from debian/control
gz_utils-ci-main-jammy-amd64|37|2024-04-19 09:06:13.271000|Failed to fetch some archives|see Dockerfile
sdformat15-debbuilder|258|2024-04-18 12:09:44.947000|Failed to compile and install build|install build dependencies from debian/control
gz_rendering-ci-ign-rendering6-focal-amd64|46|2024-04-19 08:06:28.749000|Failed to fetch some archives|see Dockerfile
gz_utils-ci-ign-utils1-focal-amd64|43|2024-04-19 09:06:13.223000|Failed to fetch some archives|see Dockerfile
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_deb
- Failing 66.67% in 10 out of 15 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rel" job)

Issue #2 in nightly_linux-rhel_repeated
- Failing 43.75% in 7 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #3 in nightly_win_rep
- Failing 50.0% in 7 out of 14 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #4 in nightly_win_rep
- Failing 50.0% in 7 out of 14 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #5 in nightly_linux_debug
- Failing 31.25% in 5 out of 16 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #6 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 80.0% in 4 out of 5 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #7 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesDetachTest/0.JointDetach (known issue in: https://github.com/gazebosim/gz-physics/issues/627 - OPEN)
	  (also in: "gz_physics-ci-main-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-physics/issues/627 - OPEN)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-physics/issues/627 - OPEN)

Issue #8 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #9 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 4 out of 4 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #10 in nightly_win_rel
- Failing 25.0% in 4 out of 16 builds
	ros2cli.ros2cli.test.test_daemon.test_get_name

Issue #11 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #12 in gz_rendering-7-win
- Failing 100.0% in 3 out of 3 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #13 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_server
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #14 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #15 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #16 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_client
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #17 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_subscription
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #18 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_service
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #19 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_qos_event__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #20 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_publisher (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)

Issue #21 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #22 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_guard_condition (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)

Issue #23 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	rosidl_typesupport_c.test_service_type_support.gtest.missing_result
	rosidl_typesupport_c.test_message_type_support.gtest.missing_result
	rcutils.test_time.gtest.missing_result
	rcutils.test_strerror.gtest.missing_result
	rcutils.test_logging_custom_env2.gtest.missing_result
	rcutils.test_logging_custom_env.gtest.missing_result
	rcutils.test_filesystem.gtest.missing_result
	rclcpp_lifecycle.test_transition_wrapper.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_node.gtest.missing_result
	rclcpp_action.test_server_goal_handle.gtest.missing_result
	rclcpp_action.test_server.gtest.missing_result
	rclcpp_action.test_client.gtest.missing_result
	rclcpp.test_utilities.gtest.missing_result
	rclcpp.test_timer.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_subscription.gtest.missing_result
	rclcpp.test_storage_policy_common.gtest.missing_result
	rclcpp.test_static_single_threaded_executor.gtest.missing_result
	rclcpp.test_service.gtest.missing_result
	rclcpp.test_qos_event__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_qos_event__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_publisher.gtest.missing_result
	rclcpp.test_node_options.gtest.missing_result
	rclcpp.test_node_interfaces__node_waitables.gtest.missing_result
	rclcpp.test_node_interfaces__node_topics.gtest.missing_result
	rclcpp.test_node_interfaces__node_timers.gtest.missing_result
	rclcpp.test_node_interfaces__node_services.gtest.missing_result
	rclcpp.test_node_interfaces__node_parameters.gtest.missing_result
	rclcpp.test_node_interfaces__node_graph.gtest.missing_result
	rclcpp.test_node_interfaces__node_base.gtest.missing_result
	rclcpp.test_node.gtest.missing_result
	rclcpp.test_loaned_message.gtest.missing_result
	rclcpp.test_init_options.gtest.missing_result
	rclcpp.test_guard_condition.gtest.missing_result
	rclcpp.test_graph_listener.gtest.missing_result
	rclcpp.test_generic_client.gtest.missing_result
	rclcpp.test_expand_topic_or_service_name.gtest.missing_result
	rclcpp.test_exceptions.gtest.missing_result
	rclcpp.test_client_common.gtest.missing_result
	rclcpp.test_client.gtest.missing_result
	rcl_yaml_param_parser.test_parser.gtest.missing_result
	rcl_yaml_param_parser.test_parse.gtest.missing_result
	rcl.test_wait__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_wait__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_validate_enclave_name.gtest.missing_result
	rcl.test_subscription__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_subscription__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_security.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_node__rmw_fastrtps_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_node__rmw_cyclonedds_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_logging.gtest.missing_result
	rcl.test_init__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_init__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_expand_topic_name.gtest.missing_result
	rcl.test_domain_id.gtest.missing_result
	rcl.test_context__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_context__rmw_cyclonedds_cpp.gtest.missing_result
	projectroot.test_transition_wrapper
	projectroot.test_time
	projectroot.test_strerror
	projectroot.test_service_type_support
	projectroot.test_server_goal_handle
	projectroot.test_parser
	projectroot.test_parse
	projectroot.test_message_type_support
	projectroot.test_logging_custom_env2
	projectroot.test_logging_custom_env
	projectroot.test_filesystem
	projectroot.test.test_wait__rmw_fastrtps_cpp
	projectroot.test.test_wait__rmw_cyclonedds_cpp
	projectroot.test.test_validate_enclave_name
	projectroot.test.test_subscription__rmw_fastrtps_cpp
	projectroot.test.test_subscription__rmw_cyclonedds_cpp
	projectroot.test.test_service_event_publisher__rmw_fastrtps_cpp
	projectroot.test.test_service_event_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_service__rmw_fastrtps_cpp
	projectroot.test.test_service__rmw_cyclonedds_cpp
	projectroot.test.test_security
	projectroot.test.test_rmw_impl_id_check_func__rmw_fastrtps_cpp
	projectroot.test.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp
	projectroot.test.test_publisher__rmw_fastrtps_cpp
	projectroot.test.test_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_node__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_node__rmw_cyclonedds_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_logging
	projectroot.test.test_init__rmw_fastrtps_cpp
	projectroot.test.test_init__rmw_cyclonedds_cpp
	projectroot.test.test_guard_condition__rmw_fastrtps_cpp
	projectroot.test.test_guard_condition__rmw_cyclonedds_cpp
	projectroot.test.test_expand_topic_name
	projectroot.test.test_domain_id
	projectroot.test.test_context__rmw_fastrtps_cpp
	projectroot.test.test_context__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_utilities
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_qos_event__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_node_options
	projectroot.test.rclcpp.test_node_interfaces__node_waitables
	projectroot.test.rclcpp.test_node_interfaces__node_topics
	projectroot.test.rclcpp.test_node_interfaces__node_timers
	projectroot.test.rclcpp.test_node_interfaces__node_services
	projectroot.test.rclcpp.test_node_interfaces__node_parameters
	projectroot.test.rclcpp.test_node_interfaces__node_graph
	projectroot.test.rclcpp.test_node_interfaces__node_base
	projectroot.test.rclcpp.test_loaned_message
	projectroot.test.rclcpp.test_init_options
	projectroot.test.rclcpp.test_generic_client
	projectroot.test.rclcpp.test_expand_topic_or_service_name
	projectroot.test.rclcpp.test_exceptions
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #24 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_client_common
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #25 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_client
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #26 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp

Issue #27 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #28 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	projectroot.test_action_client
	  (also in: "nightly_win_rel" job)

Issue #29 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #30 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #31 in nightly_linux-aarch64_debug
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #32 in Rci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 2 out of 2 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #33 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[2-3]
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #34 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	test_rclcpp.test_node_name__rmw_cyclonedds_cpp__rmw_fastrtps_cpp.xunit.missing_result
	projectroot.test_node_name__rmw_cyclonedds_cpp__rmw_fastrtps_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #35 in nightly_linux_repeated
- Failing 12.5% in 2 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #36 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	test_rclcpp.TestNNodes.test_10_nodes
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #37 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rcl_action.TestActionServerInitFini.test_action_server_init_fini
	rcl_action.TestActionGraphFixture__rmw_fastrtps_cpp.test_action_get_client_names_and_types_by_node
	rcl_action.TestActionClientWait.test_wait_set_add_action_client
	projectroot.test_wait
	projectroot.test_graph__rmw_fastrtps_cpp
	projectroot.test_action_server
	projectroot.test_action_interaction__rmw_fastrtps_cpp
	projectroot.test_action_communication__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)

Issue #38 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TestTimeSource.callbacks
	projectroot.test.rclcpp.test_time_source
	  (also in: "nightly_win_rel" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #39 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	projectroot.test.rclcpp.test_subscription_topic_statistics
	  (also in: "nightly_win_rel" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #40 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TestUtilities.wait_for_message_twice_one_sub
	rclcpp.TestUtilities.wait_for_message_indefinitely
	rclcpp.TestUtilities.wait_for_message
	rclcpp.TestTimeSource.reattach
	rclcpp.TestTimeSource.parameter_activation
	rclcpp.TestTimeSource.no_pre_jump_callback
	rclcpp.TestTimeSource.detachUnattached
	rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic
	rclcpp.TestTimeSource.clock
	rclcpp.TestTimeSource.check_use_clock_thread_value
	rclcpp.TestTimeSource.check_clock_thread_status
	rclcpp.TestTimeSource.callback_handler_erasure
	rclcpp.TestTimeSource.ROS_time_valid_wall_time
	rclcpp.TestTimeSource.ROS_time_valid_sim_time
	rclcpp.TestTimeSource.ROS_time_valid_attach_detach
	rclcpp.TestTimeSource.ROS_invalid_sim_time
	rclcpp.TestLoggerService.check_connect_get_logger_service
	rclcpp.TestInterfaceTraits.has_node_base_interface
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	projectroot.test.rclcpp.test_wait_for_message
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits
	  (also in: "nightly_win_rel" job)

Issue #41 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TestWaitSet.add_remove_wait
	rclcpp.TestWaitSet.add_guard_condition_to_two_different_wait_set
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer2CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer1CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT1Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBackTimerCancelThenResetBehavior
	rclcpp.TestThreadSafeStorage.wait_waitable
	rclcpp.TestThreadSafeStorage.wait_timer
	rclcpp.TestThreadSafeStorage.wait_subscription
	rclcpp.TestThreadSafeStorage.wait_client_service
	rclcpp.TestThreadSafeStorage.iterables_construct_destruct
	rclcpp.TestThreadSafeStorage.default_construct_destruct
	rclcpp.TestThreadSafeStorage.add_remove_out_of_scope
	rclcpp.TestThreadSafeStorage.add_remove_nullptr
	rclcpp.TestThreadSafeStorage.add_remove_dynamically
	rclcpp.TestSubscriptionTopicStatisticsFixture.test_invalid_publish_period
	rclcpp.TestSubscriptionOptions.topic_statistics_options_node_default_mode
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_resize_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_fini_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_clear_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_timer_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_subscription_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_service_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_guard_condition_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_client_error
	rclcpp.TestStoragePolicyCommon.add_waitable_error
	rclcpp.TestStaticStorage.wait_waitable
	rclcpp.TestStaticStorage.wait_timer
	rclcpp.TestStaticStorage.wait_subscription
	rclcpp.TestStaticStorage.wait_guard_condition
	rclcpp.TestStaticStorage.wait_client_service
	rclcpp.TestStaticStorage.iterables_construct_destruct
	rclcpp.TestStaticStorage.fixed_storage_needs_pruning
	rclcpp.TestStaticSingleThreadedExecutor.add_callback_group_trigger_guard_failed
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_parent_log
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_node_rosout_disabled
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild_hierarchy
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_child_log
	rclcpp.TestMultiThreadedExecutor.timer_over_take
	rclcpp.TestIntraprocessExecutors/SingleThreadedExecutor.testIntraprocessRetrigger
	rclcpp.TestIntraProcessWaitable.test_that_waitable_stays_ready_after_second_wait
	rclcpp.TestGraphListener.construction_and_destruction
	rclcpp.TestExecutorNotifyWaitable.add_remove_guard_conditions
	rclcpp.TestExecutorEntitiesCollector.add_remove_node
	rclcpp.TestEventsExecutor.run_pub_sub
	rclcpp.TestDynamicStorage.wait_waitable
	rclcpp.TestDynamicStorage.wait_timer
	rclcpp.TestDynamicStorage.wait_subscription
	rclcpp.TestDynamicStorage.wait_client_service
	rclcpp.TestDynamicStorage.iterables_construct_destruct
	rclcpp.TestDynamicStorage.default_construct_destruct
	rclcpp.TestDynamicStorage.add_remove_out_of_scope
	rclcpp.TestDynamicStorage.add_remove_nullptr
	rclcpp.TestDynamicStorage.add_remove_dynamically
	rclcpp.TestContentFilterSubscription.set_content_filter_error
	rclcpp.TestContentFilterSubscription.set_content_filter
	rclcpp.TestContentFilterSubscription.is_cft_enabled
	rclcpp.TestContentFilterSubscription.get_content_filter_error
	rclcpp.TestContentFilterSubscription.get_content_filter
	rclcpp.TestContentFilterSubscription.content_filter_reset
	rclcpp.TestContentFilterSubscription.content_filter_get_later
	rclcpp.TestContentFilterSubscription.content_filter_get_begin
	rclcpp.TestCallbackGroup.valid_callback_group
	rclcpp.TestCallbackGroup.invalid_callback_group
	projectroot.test.rclcpp.test_wait_set
	projectroot.test.rclcpp.test_thread_safe_synchronization
	projectroot.test.rclcpp.test_subscription_options
	projectroot.test.rclcpp.test_static_storage
	projectroot.test.rclcpp.test_rosout_subscription
	projectroot.test.rclcpp.test_multi_threaded_executor
	projectroot.test.rclcpp.test_intra_process_waitable
	projectroot.test.rclcpp.test_executors_intraprocess
	projectroot.test.rclcpp.test_executors_callback_group_behavior
	projectroot.test.rclcpp.test_executor_notify_waitable
	projectroot.test.rclcpp.test_events_executor
	projectroot.test.rclcpp.test_entities_collector
	projectroot.test.rclcpp.test_dynamic_storage
	  (also in: "nightly_win_rel" job)

Issue #42 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 1 out of 1 builds
	projectroot.benchmark_string_conversions
	  (also in: "Rci__benchmark_ubuntu_jammy_amd64" job)

Issue #43 in gz_common-5-win
- Failing 20.0% in 1 out of 5 builds
	PluginSpecialization.AccessTime

Issue #44 in gz_fuel_tools-8-win
- Failing 20.0% in 1 out of 5 builds
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-8-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)

Issue #45 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_ImageDisplay_TEST.test_ran

Issue #46 in gz_math-ign-math6-win
- Failing 25.0% in 1 out of 4 builds
	Stopwatch.StartStopReset

Issue #47 in gz_rendering-ci-gz-rendering8-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_WireBox_TEST_ogre_gl3plus.test_ran
	UNIT_WireBox_TEST_ogre2_metal.test_ran
	UNIT_Visual_TEST_ogre_gl3plus.test_ran
	UNIT_Visual_TEST_ogre2_metal.test_ran
	UNIT_Utils_TEST_ogre_gl3plus.test_ran
	UNIT_Utils_TEST_ogre2_metal.test_ran
	UNIT_TransformController_TEST_ogre_gl3plus.test_ran
	UNIT_TransformController_TEST_ogre2_metal.test_ran
	UNIT_ThermalCamera_TEST_ogre_gl3plus.test_ran
	UNIT_ThermalCamera_TEST_ogre2_metal.test_ran
	UNIT_Text_TEST_ogre_gl3plus.test_ran
	UNIT_Text_TEST_ogre2_metal.test_ran
	UNIT_ShaderType_TEST.test_ran
	UNIT_ShaderParams_TEST.test_ran
	UNIT_ShaderParam_TEST.test_ran
	UNIT_SegmentationCamera_TEST_ogre_gl3plus.test_ran
	UNIT_SegmentationCamera_TEST_ogre2_metal.test_ran
	UNIT_Scene_TEST_ogre_gl3plus.test_ran
	UNIT_Scene_TEST_ogre2_metal.test_ran
	UNIT_RenderingIface_TEST_ogre_gl3plus.test_ran
	UNIT_RenderingIface_TEST_ogre2_metal.test_ran
	UNIT_RenderTarget_TEST_ogre_gl3plus.test_ran
	UNIT_RenderTarget_TEST_ogre2_metal.test_ran
	UNIT_RenderPassSystem_TEST_ogre_gl3plus.test_ran
	UNIT_RenderPassSystem_TEST_ogre2_metal.test_ran
	UNIT_RenderEngine_TEST_ogre_gl3plus.test_ran
	UNIT_RenderEngine_TEST_ogre2_metal.test_ran
	UNIT_RenderEngineManager_TEST_ogre_gl3plus.test_ran
	UNIT_RenderEngineManager_TEST_ogre2_metal.test_ran
	UNIT_RayQuery_TEST_ogre_gl3plus.test_ran
	UNIT_RayQuery_TEST_ogre2_metal.test_ran
	UNIT_Projector_TEST_ogre_gl3plus.test_ran
	UNIT_Projector_TEST_ogre2_metal.test_ran
	UNIT_PixelFormat_TEST.test_ran
	UNIT_ParticleEmitter_TEST_ogre_gl3plus.test_ran
	UNIT_ParticleEmitter_TEST_ogre2_metal.test_ran
	UNIT_OrthoViewController_TEST_ogre_gl3plus.test_ran
	UNIT_OrthoViewController_TEST_ogre2_metal.test_ran
	UNIT_OrbitViewController_TEST_ogre_gl3plus.test_ran
	UNIT_OrbitViewController_TEST_ogre2_metal.test_ran
	UNIT_Node_TEST_ogre_gl3plus.test_ran
	UNIT_Node_TEST_ogre2_metal.test_ran
	UNIT_MoveToHelper_TEST_ogre_gl3plus.test_ran
	UNIT_MoveToHelper_TEST_ogre2_metal.test_ran
	UNIT_Mesh_TEST_ogre_gl3plus.test_ran
	UNIT_Mesh_TEST_ogre2_metal.test_ran
	UNIT_MeshDescriptor_TEST_ogre_gl3plus.test_ran
	UNIT_MeshDescriptor_TEST_ogre2_metal.test_ran
	UNIT_Material_TEST_ogre_gl3plus.test_ran
	UNIT_Material_TEST_ogre2_metal.test_ran
	UNIT_Marker_TEST_ogre_gl3plus.test_ran
	UNIT_Marker_TEST_ogre2_metal.test_ran
	UNIT_Light_TEST_ogre_gl3plus.test_ran
	UNIT_Light_TEST_ogre2_metal.test_ran
	UNIT_LightVisual_TEST_ogre_gl3plus.test_ran
	UNIT_LightVisual_TEST_ogre2_metal.test_ran
	UNIT_LidarVisual_TEST_ogre_gl3plus.test_ran
	UNIT_LidarVisual_TEST_ogre2_metal.test_ran
	UNIT_LensFlarePass_TEST_ogre_gl3plus.test_ran
	UNIT_LensFlarePass_TEST_ogre2_metal.test_ran
	UNIT_InertiaVisual_TEST_ogre_gl3plus.test_ran
	UNIT_InertiaVisual_TEST_ogre2_metal.test_ran
	UNIT_Heightmap_TEST_ogre_gl3plus.test_ran
	UNIT_Heightmap_TEST_ogre2_metal.test_ran
	UNIT_Grid_TEST_ogre_gl3plus.test_ran
	UNIT_Grid_TEST_ogre2_metal.test_ran
	UNIT_GraphicsAPI_TEST.test_ran
	UNIT_GlobalIllumination_TEST_ogre_gl3plus.test_ran
	UNIT_GlobalIllumination_TEST_ogre2_metal.test_ran
	UNIT_GizmoVisual_TEST_ogre_gl3plus.test_ran
	UNIT_GizmoVisual_TEST_ogre2_metal.test_ran
	UNIT_GaussianNoisePass_TEST_ogre_gl3plus.test_ran
	UNIT_GaussianNoisePass_TEST_ogre2_metal.test_ran
	UNIT_Capsule_TEST_ogre_gl3plus.test_ran
	UNIT_Capsule_TEST_ogre2_metal.test_ran
	UNIT_Camera_TEST_ogre_gl3plus.test_ran
	UNIT_Camera_TEST_ogre2_metal.test_ran
	UNIT_COMVisual_TEST_ogre_gl3plus.test_ran
	UNIT_COMVisual_TEST_ogre2_metal.test_ran
	UNIT_BoundingBox_TEST_ogre_gl3plus.test_ran
	UNIT_BoundingBox_TEST_ogre2_metal.test_ran
	UNIT_BoundingBoxCamera_TEST_ogre_gl3plus.test_ran
	UNIT_BoundingBoxCamera_TEST_ogre2_metal.test_ran
	UNIT_AxisVisual_TEST_ogre_gl3plus.test_ran
	UNIT_AxisVisual_TEST_ogre2_metal.test_ran
	UNIT_ArrowVisual_TEST_ogre_gl3plus.test_ran
	UNIT_ArrowVisual_TEST_ogre2_metal.test_ran
	REGRESSION_reload_engine_ogre_gl3plus.test_ran
	REGRESSION_reload_engine_ogre2_metal.test_ran
	PERFORMANCE_scene_factory_ogre_gl3plus.test_ran
	PERFORMANCE_scene_factory_ogre2_metal.test_ran
	INTEGRATION_wide_angle_camera_ogre_gl3plus.test_ran
	INTEGRATION_wide_angle_camera_ogre2_metal.test_ran
	INTEGRATION_waves_ogre_gl3plus.test_ran
	INTEGRATION_waves_ogre2_metal.test_ran
	INTEGRATION_thermal_camera_ogre_gl3plus.test_ran
	INTEGRATION_thermal_camera_ogre2_metal.test_ran
	INTEGRATION_sky_ogre_gl3plus.test_ran
	INTEGRATION_sky_ogre2_metal.test_ran
	INTEGRATION_shadows_ogre_gl3plus.test_ran
	INTEGRATION_shadows_ogre2_metal.test_ran
	INTEGRATION_segmentation_camera_ogre_gl3plus.test_ran
	INTEGRATION_segmentation_camera_ogre2_metal.test_ran
	INTEGRATION_scene_ogre_gl3plus.test_ran
	INTEGRATION_scene_ogre2_metal.test_ran
	INTEGRATION_render_pass_ogre_gl3plus.test_ran
	INTEGRATION_render_pass_ogre2_metal.test_ran
	INTEGRATION_projector_ogre_gl3plus.test_ran
	INTEGRATION_projector_ogre2_metal.test_ran
	INTEGRATION_mesh_ogre_gl3plus.test_ran
	INTEGRATION_mesh_ogre2_metal.test_ran
	INTEGRATION_load_unload_ogre_gl3plus.test_ran
	INTEGRATION_load_unload_ogre2_metal.test_ran
	INTEGRATION_lidar_visual_ogre_gl3plus.test_ran
	INTEGRATION_lidar_visual_ogre2_metal.test_ran
	INTEGRATION_heightmap_ogre_gl3plus.test_ran
	INTEGRATION_heightmap_ogre2_metal.test_ran
	INTEGRATION_gpu_rays_ogre_gl3plus.test_ran
	INTEGRATION_gpu_rays_ogre2_metal.test_ran
	INTEGRATION_depth_camera_ogre_gl3plus.test_ran
	INTEGRATION_depth_camera_ogre2_metal.test_ran
	INTEGRATION_camera_ogre_gl3plus.test_ran
	INTEGRATION_camera_ogre2_metal.test_ran
	INTEGRATION_boundingbox_camera_ogre_gl3plus.test_ran
	INTEGRATION_boundingbox_camera_ogre2_metal.test_ran

Issue #48 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	RgbdCameraTest.RgbdCameraBox

Issue #49 in gz_sim-ci-main-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	<init>.ServerRepeat/SceneBroadcasterTest
	<init>.NetworkHandshake

Issue #50 in gz_transport-ci-main-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	gzTest.TopicEchoNum

Issue #51 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #52 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #53 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	rclcpp.PerTimerType/TestTimer.test_simple_cancel/wall_timer (4-byte object <00-00 00-00>)                     
	  (also in: "nightly_win_rel" job)

Issue #54 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	rcl_action.TestActionCommunication__rmw_fastrtps_cpp.test_valid_goal_comm
	rcl_action.TestActionClientServerInteraction__rmw_fastrtps_cpp.test_interaction
	  (also in: "nightly_win_rel" job)

Issue #55 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[Defaults]
	rclcpp.TestClockSleep.sleep_until_basic_system
	rcl_action.TestActionClientFixture.test_action_client_is_valid
	projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp

Issue #56 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	projectroot.test.rclcpp.test_time (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #57 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

@Crola1702
Copy link
Contributor

Crola1702 commented Apr 19, 2024 •
edited
Loading

Nightly Linux Release | Build regression | Keeping track 👀

Reference build: https://ci.ros2.org/view/nightly/job/nightly_linux_release/3048/

Output:

00:13:29.434 Err:23 http://archive.ubuntu.com/ubuntu noble/main amd64 libicu74 amd64 74.2-1ubuntu3
00:13:29.434   Undetermined Error [IP: 185.125.190.36 80]
00:13:30.420 Err:24 http://archive.ubuntu.com/ubuntu noble/main amd64 libxml2 amd64 2.9.14+dfsg-1.3ubuntu3
00:13:30.420   Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.83), connection timed out [IP: 185.125.190.36 80]
...
00:13:30.444 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/w/wheel/python3-wheel_0.42.0-2_all.deb  Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.36 80]
00:13:30.444 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/p/python-pip/python3-pip_24.0%2bdfsg-1ubuntu1_all.deb  Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.36 80]
00:13:30.444 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
00:13:30.444 The command '/bin/sh -c apt-get update && apt-get install --no-install-recommends -y build-essential ccache cmake pkg-config python3-empy python3-pip python3-setuptools python3-vcstool' returned a non-zero code: 100

Seems like a network issue

@github-actions GitHub Actions
Copy link
Contributor Author

Apr 20

Build regressions 
nightly_linux_coverage|2346|2024-04-20 05:11:50.459000|Number of failed test exceeds the threshold|coverage analysis
nightly_win_rep|3330|2024-04-20 04:00:01.581000|Number of failed test exceeds the threshold|test
Ici__nightly-performance_ubuntu_jammy_amd64|95|2024-04-20 01:42:35.340000|Packages performance_test Failed|build workspace in isolation and install
gz-msgs11-debbuilder|267|2024-04-20 10:48:20.743000|Failed to compile and install build|install build dependencies from debian/control
gz-msgs11-debbuilder|265|2024-04-19 13:40:38.871000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|262|2024-04-19 16:19:08.352000|Failed to compile and install build|install build dependencies from debian/control
gz_transport-main-win|38|2024-04-20 08:17:06.410000|Packages gz-tools2  Failed|colcon compilation without test for dependencies of gz-transport14
gz_transport-main-win|37|2024-04-19 15:06:16.065000|Packages gz-tools2  Failed|colcon compilation without test for dependencies of gz-transport14
sdformat-main-win|43|2024-04-20 08:06:22.199000|Packages gz-tools2  Failed|colcon compilation without test for dependencies of sdformat15
gz_gui-ci-gz-gui8-homebrew-amd64|40|2024-04-20 03:06:13.597000|Failed to compile and install gz-gui8|install gz-gui8 dependencies
gz_launch-ci-gz-launch7-homebrew-amd64|41|2024-04-20 07:06:05.228000|Failed to compile and install gz-launch7|install gz-launch7 dependencies
sdformat-ci-sdf14-jammy-amd64|65|2024-04-20 08:57:07.485000|Jenkins channel closing down (Agent disconnected)|running tests
gz_gui-ci-gz-gui7-homebrew-amd64|37|2024-04-20 04:06:11.150000|Failed to compile and install gz-gui7|install gz-gui7 dependencies
gz_sensors-ci-gz-sensors7-homebrew-amd64|37|2024-04-20 06:06:12.762000|Failed to compile and install gz-sensors7|install gz-sensors7 dependencies
gz_sim-7-win|49|2024-04-20 03:06:14.622000|Jenkins channel closing down (Agent disconnected)|colcon compilation without test for dependencies of gz-sim7
gz_sim-ci-gz-sim7-homebrew-amd64|45|2024-04-19 22:06:06.221000|Failed to compile and install gz-sim7|install gz-sim7 dependencies
gz_transport-12-win|40|2024-04-20 06:06:13.067000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_deb
- Failing 66.67% in 10 out of 15 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes

Issue #2 in nightly_linux-aarch64_repeated
- Failing 37.5% in 6 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #3 in Rci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 4 out of 4 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #4 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #5 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #6 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 4 out of 4 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #7 in nightly_linux-aarch64_repeated
- Failing 25.0% in 4 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

Issue #8 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #9 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_server
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #10 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #11 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #12 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_client
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #13 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #14 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_subscription
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #15 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_service
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #16 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_qos_event__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #17 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_publisher (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)

Issue #18 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #19 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_guard_condition (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)

Issue #20 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #21 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #22 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	rosidl_typesupport_c.test_service_type_support.gtest.missing_result
	rosidl_typesupport_c.test_message_type_support.gtest.missing_result
	rcutils.test_time.gtest.missing_result
	rcutils.test_strerror.gtest.missing_result
	rcutils.test_logging_custom_env2.gtest.missing_result
	rcutils.test_logging_custom_env.gtest.missing_result
	rcutils.test_filesystem.gtest.missing_result
	rclcpp_lifecycle.test_transition_wrapper.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_node.gtest.missing_result
	rclcpp_action.test_server_goal_handle.gtest.missing_result
	rclcpp_action.test_server.gtest.missing_result
	rclcpp_action.test_client.gtest.missing_result
	rclcpp.test_utilities.gtest.missing_result
	rclcpp.test_timer.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_subscription.gtest.missing_result
	rclcpp.test_storage_policy_common.gtest.missing_result
	rclcpp.test_static_single_threaded_executor.gtest.missing_result
	rclcpp.test_service.gtest.missing_result
	rclcpp.test_qos_event__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_qos_event__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_publisher.gtest.missing_result
	rclcpp.test_node_options.gtest.missing_result
	rclcpp.test_node_interfaces__node_waitables.gtest.missing_result
	rclcpp.test_node_interfaces__node_topics.gtest.missing_result
	rclcpp.test_node_interfaces__node_timers.gtest.missing_result
	rclcpp.test_node_interfaces__node_services.gtest.missing_result
	rclcpp.test_node_interfaces__node_parameters.gtest.missing_result
	rclcpp.test_node_interfaces__node_graph.gtest.missing_result
	rclcpp.test_node_interfaces__node_base.gtest.missing_result
	rclcpp.test_node.gtest.missing_result
	rclcpp.test_loaned_message.gtest.missing_result
	rclcpp.test_init_options.gtest.missing_result
	rclcpp.test_guard_condition.gtest.missing_result
	rclcpp.test_graph_listener.gtest.missing_result
	rclcpp.test_generic_client.gtest.missing_result
	rclcpp.test_expand_topic_or_service_name.gtest.missing_result
	rclcpp.test_exceptions.gtest.missing_result
	rclcpp.test_client_common.gtest.missing_result
	rclcpp.test_client.gtest.missing_result
	rcl_yaml_param_parser.test_parser.gtest.missing_result
	rcl_yaml_param_parser.test_parse.gtest.missing_result
	rcl.test_wait__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_wait__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_validate_enclave_name.gtest.missing_result
	rcl.test_subscription__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_subscription__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_security.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_node__rmw_fastrtps_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_node__rmw_cyclonedds_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_logging.gtest.missing_result
	rcl.test_init__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_init__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_expand_topic_name.gtest.missing_result
	rcl.test_domain_id.gtest.missing_result
	rcl.test_context__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_context__rmw_cyclonedds_cpp.gtest.missing_result
	projectroot.test_transition_wrapper
	projectroot.test_time
	projectroot.test_strerror
	projectroot.test_service_type_support
	projectroot.test_server_goal_handle
	projectroot.test_parser
	projectroot.test_parse
	projectroot.test_message_type_support
	projectroot.test_logging_custom_env2
	projectroot.test_logging_custom_env
	projectroot.test_filesystem
	projectroot.test.test_wait__rmw_fastrtps_cpp
	projectroot.test.test_wait__rmw_cyclonedds_cpp
	projectroot.test.test_validate_enclave_name
	projectroot.test.test_subscription__rmw_fastrtps_cpp
	projectroot.test.test_subscription__rmw_cyclonedds_cpp
	projectroot.test.test_service_event_publisher__rmw_fastrtps_cpp
	projectroot.test.test_service_event_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_service__rmw_fastrtps_cpp
	projectroot.test.test_service__rmw_cyclonedds_cpp
	projectroot.test.test_security
	projectroot.test.test_rmw_impl_id_check_func__rmw_fastrtps_cpp
	projectroot.test.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp
	projectroot.test.test_publisher__rmw_fastrtps_cpp
	projectroot.test.test_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_node__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_node__rmw_cyclonedds_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_logging
	projectroot.test.test_init__rmw_fastrtps_cpp
	projectroot.test.test_init__rmw_cyclonedds_cpp
	projectroot.test.test_guard_condition__rmw_fastrtps_cpp
	projectroot.test.test_guard_condition__rmw_cyclonedds_cpp
	projectroot.test.test_expand_topic_name
	projectroot.test.test_domain_id
	projectroot.test.test_context__rmw_fastrtps_cpp
	projectroot.test.test_context__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_utilities
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_qos_event__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_node_options
	projectroot.test.rclcpp.test_node_interfaces__node_waitables
	projectroot.test.rclcpp.test_node_interfaces__node_topics
	projectroot.test.rclcpp.test_node_interfaces__node_timers
	projectroot.test.rclcpp.test_node_interfaces__node_services
	projectroot.test.rclcpp.test_node_interfaces__node_parameters
	projectroot.test.rclcpp.test_node_interfaces__node_graph
	projectroot.test.rclcpp.test_node_interfaces__node_base
	projectroot.test.rclcpp.test_loaned_message
	projectroot.test.rclcpp.test_init_options
	projectroot.test.rclcpp.test_generic_client
	projectroot.test.rclcpp.test_expand_topic_or_service_name
	projectroot.test.rclcpp.test_exceptions
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #23 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_client_common
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #24 in nightly_linux-aarch64_debug
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_client
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #25 in nightly_linux-aarch64_repeated
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #26 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #27 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 100.0% in 3 out of 3 builds
	JointFeaturesDetachTest/0.JointDetach (known issue in: https://github.com/gazebosim/gz-physics/issues/627 - OPEN)
	  (also in: "gz_physics-ci-main-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-physics/issues/627 - OPEN)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-physics/issues/627 - OPEN)

Issue #28 in nightly_linux-aarch64_repeated
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_record__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #29 in nightly_linux_repeated
- Failing 18.75% in 3 out of 16 builds
	rosbag2_transport.RecordIntegrationTestFixture.can_record_again_after_stop
	projectroot.test_record__rmw_fastrtps_cpp
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #30 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #31 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #32 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)

Issue #33 in nightly_linux_repeated
- Failing 12.5% in 2 out of 16 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #34 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 1 out of 1 builds
	projectroot.benchmark_string_conversions
	  (also in: "Rci__benchmark_ubuntu_jammy_amd64" job)

Issue #35 in gz_common-5-win
- Failing 20.0% in 1 out of 5 builds
	PluginSpecialization.AccessTime

Issue #36 in gz_fuel_tools-8-win
- Failing 25.0% in 1 out of 4 builds
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-8-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)

Issue #37 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_ImageDisplay_TEST.test_ran

Issue #38 in gz_math-ign-math6-win
- Failing 25.0% in 1 out of 4 builds
	Stopwatch.StartStopReset

Issue #39 in gz_rendering-ci-gz-rendering8-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_WireBox_TEST_ogre_gl3plus.test_ran
	UNIT_WireBox_TEST_ogre2_metal.test_ran
	UNIT_Visual_TEST_ogre_gl3plus.test_ran
	UNIT_Visual_TEST_ogre2_metal.test_ran
	UNIT_Utils_TEST_ogre_gl3plus.test_ran
	UNIT_Utils_TEST_ogre2_metal.test_ran
	UNIT_TransformController_TEST_ogre_gl3plus.test_ran
	UNIT_TransformController_TEST_ogre2_metal.test_ran
	UNIT_ThermalCamera_TEST_ogre_gl3plus.test_ran
	UNIT_ThermalCamera_TEST_ogre2_metal.test_ran
	UNIT_Text_TEST_ogre_gl3plus.test_ran
	UNIT_Text_TEST_ogre2_metal.test_ran
	UNIT_ShaderType_TEST.test_ran
	UNIT_ShaderParams_TEST.test_ran
	UNIT_ShaderParam_TEST.test_ran
	UNIT_SegmentationCamera_TEST_ogre_gl3plus.test_ran
	UNIT_SegmentationCamera_TEST_ogre2_metal.test_ran
	UNIT_Scene_TEST_ogre_gl3plus.test_ran
	UNIT_Scene_TEST_ogre2_metal.test_ran
	UNIT_RenderingIface_TEST_ogre_gl3plus.test_ran
	UNIT_RenderingIface_TEST_ogre2_metal.test_ran
	UNIT_RenderTarget_TEST_ogre_gl3plus.test_ran
	UNIT_RenderTarget_TEST_ogre2_metal.test_ran
	UNIT_RenderPassSystem_TEST_ogre_gl3plus.test_ran
	UNIT_RenderPassSystem_TEST_ogre2_metal.test_ran
	UNIT_RenderEngine_TEST_ogre_gl3plus.test_ran
	UNIT_RenderEngine_TEST_ogre2_metal.test_ran
	UNIT_RenderEngineManager_TEST_ogre_gl3plus.test_ran
	UNIT_RenderEngineManager_TEST_ogre2_metal.test_ran
	UNIT_RayQuery_TEST_ogre_gl3plus.test_ran
	UNIT_RayQuery_TEST_ogre2_metal.test_ran
	UNIT_Projector_TEST_ogre_gl3plus.test_ran
	UNIT_Projector_TEST_ogre2_metal.test_ran
	UNIT_PixelFormat_TEST.test_ran
	UNIT_ParticleEmitter_TEST_ogre_gl3plus.test_ran
	UNIT_ParticleEmitter_TEST_ogre2_metal.test_ran
	UNIT_OrthoViewController_TEST_ogre_gl3plus.test_ran
	UNIT_OrthoViewController_TEST_ogre2_metal.test_ran
	UNIT_OrbitViewController_TEST_ogre_gl3plus.test_ran
	UNIT_OrbitViewController_TEST_ogre2_metal.test_ran
	UNIT_Node_TEST_ogre_gl3plus.test_ran
	UNIT_Node_TEST_ogre2_metal.test_ran
	UNIT_MoveToHelper_TEST_ogre_gl3plus.test_ran
	UNIT_MoveToHelper_TEST_ogre2_metal.test_ran
	UNIT_Mesh_TEST_ogre_gl3plus.test_ran
	UNIT_Mesh_TEST_ogre2_metal.test_ran
	UNIT_MeshDescriptor_TEST_ogre_gl3plus.test_ran
	UNIT_MeshDescriptor_TEST_ogre2_metal.test_ran
	UNIT_Material_TEST_ogre_gl3plus.test_ran
	UNIT_Material_TEST_ogre2_metal.test_ran
	UNIT_Marker_TEST_ogre_gl3plus.test_ran
	UNIT_Marker_TEST_ogre2_metal.test_ran
	UNIT_Light_TEST_ogre_gl3plus.test_ran
	UNIT_Light_TEST_ogre2_metal.test_ran
	UNIT_LightVisual_TEST_ogre_gl3plus.test_ran
	UNIT_LightVisual_TEST_ogre2_metal.test_ran
	UNIT_LidarVisual_TEST_ogre_gl3plus.test_ran
	UNIT_LidarVisual_TEST_ogre2_metal.test_ran
	UNIT_LensFlarePass_TEST_ogre_gl3plus.test_ran
	UNIT_LensFlarePass_TEST_ogre2_metal.test_ran
	UNIT_InertiaVisual_TEST_ogre_gl3plus.test_ran
	UNIT_InertiaVisual_TEST_ogre2_metal.test_ran
	UNIT_Heightmap_TEST_ogre_gl3plus.test_ran
	UNIT_Heightmap_TEST_ogre2_metal.test_ran
	UNIT_Grid_TEST_ogre_gl3plus.test_ran
	UNIT_Grid_TEST_ogre2_metal.test_ran
	UNIT_GraphicsAPI_TEST.test_ran
	UNIT_GlobalIllumination_TEST_ogre_gl3plus.test_ran
	UNIT_GlobalIllumination_TEST_ogre2_metal.test_ran
	UNIT_GizmoVisual_TEST_ogre_gl3plus.test_ran
	UNIT_GizmoVisual_TEST_ogre2_metal.test_ran
	UNIT_GaussianNoisePass_TEST_ogre_gl3plus.test_ran
	UNIT_GaussianNoisePass_TEST_ogre2_metal.test_ran
	UNIT_Capsule_TEST_ogre_gl3plus.test_ran
	UNIT_Capsule_TEST_ogre2_metal.test_ran
	UNIT_Camera_TEST_ogre_gl3plus.test_ran
	UNIT_Camera_TEST_ogre2_metal.test_ran
	UNIT_COMVisual_TEST_ogre_gl3plus.test_ran
	UNIT_COMVisual_TEST_ogre2_metal.test_ran
	UNIT_BoundingBox_TEST_ogre_gl3plus.test_ran
	UNIT_BoundingBox_TEST_ogre2_metal.test_ran
	UNIT_BoundingBoxCamera_TEST_ogre_gl3plus.test_ran
	UNIT_BoundingBoxCamera_TEST_ogre2_metal.test_ran
	UNIT_AxisVisual_TEST_ogre_gl3plus.test_ran
	UNIT_AxisVisual_TEST_ogre2_metal.test_ran
	UNIT_ArrowVisual_TEST_ogre_gl3plus.test_ran
	UNIT_ArrowVisual_TEST_ogre2_metal.test_ran
	REGRESSION_reload_engine_ogre_gl3plus.test_ran
	REGRESSION_reload_engine_ogre2_metal.test_ran
	PERFORMANCE_scene_factory_ogre_gl3plus.test_ran
	PERFORMANCE_scene_factory_ogre2_metal.test_ran
	INTEGRATION_wide_angle_camera_ogre_gl3plus.test_ran
	INTEGRATION_wide_angle_camera_ogre2_metal.test_ran
	INTEGRATION_waves_ogre_gl3plus.test_ran
	INTEGRATION_waves_ogre2_metal.test_ran
	INTEGRATION_thermal_camera_ogre_gl3plus.test_ran
	INTEGRATION_thermal_camera_ogre2_metal.test_ran
	INTEGRATION_sky_ogre_gl3plus.test_ran
	INTEGRATION_sky_ogre2_metal.test_ran
	INTEGRATION_shadows_ogre_gl3plus.test_ran
	INTEGRATION_shadows_ogre2_metal.test_ran
	INTEGRATION_segmentation_camera_ogre_gl3plus.test_ran
	INTEGRATION_segmentation_camera_ogre2_metal.test_ran
	INTEGRATION_scene_ogre_gl3plus.test_ran
	INTEGRATION_scene_ogre2_metal.test_ran
	INTEGRATION_render_pass_ogre_gl3plus.test_ran
	INTEGRATION_render_pass_ogre2_metal.test_ran
	INTEGRATION_projector_ogre_gl3plus.test_ran
	INTEGRATION_projector_ogre2_metal.test_ran
	INTEGRATION_mesh_ogre_gl3plus.test_ran
	INTEGRATION_mesh_ogre2_metal.test_ran
	INTEGRATION_load_unload_ogre_gl3plus.test_ran
	INTEGRATION_load_unload_ogre2_metal.test_ran
	INTEGRATION_lidar_visual_ogre_gl3plus.test_ran
	INTEGRATION_lidar_visual_ogre2_metal.test_ran
	INTEGRATION_heightmap_ogre_gl3plus.test_ran
	INTEGRATION_heightmap_ogre2_metal.test_ran
	INTEGRATION_gpu_rays_ogre_gl3plus.test_ran
	INTEGRATION_gpu_rays_ogre2_metal.test_ran
	INTEGRATION_depth_camera_ogre_gl3plus.test_ran
	INTEGRATION_depth_camera_ogre2_metal.test_ran
	INTEGRATION_camera_ogre_gl3plus.test_ran
	INTEGRATION_camera_ogre2_metal.test_ran
	INTEGRATION_boundingbox_camera_ogre_gl3plus.test_ran
	INTEGRATION_boundingbox_camera_ogre2_metal.test_ran

Issue #40 in gz_sim-ci-main-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	<init>.ServerRepeat/SceneBroadcasterTest
	<init>.NetworkHandshake

Issue #41 in nightly_linux-aarch64_release
- Failing 6.25% in 1 out of 16 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_humble_coverage" job)
	  (also in: "nightly_linux-aarch64_release" job)

Issue #42 in nightly_linux-aarch64_release
- Failing 6.25% in 1 out of 16 builds
	test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_1
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_0
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_3
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_2
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_1
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_0
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_1
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_0
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_3
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_2
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_1
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_0
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_humble_coverage" job)
	  (also in: "nightly_linux-aarch64_release" job)

Issue #43 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	launch_pytest.test.launch_pytest.tools.test_process.test_sync_process_tools[1-3]

Issue #44 in nightly_linux-rhel_debug
- Failing 6.25% in 1 out of 16 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)

Issue #45 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	  (also in: "nightly_win_rep" job)

Issue #46 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]
	test_cli_remapping.TestCLIRemapping.test_service_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[C:/ci/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

@github-actions GitHub Actions
Copy link
Contributor Author

Apr 21

Build regressions 
nightly_linux_coverage|2347|2024-04-21 04:45:30.044000|Number of failed test exceeds the threshold|coverage analysis
packaging_linux-rhel|1903|2024-04-21 06:56:26.778000|Jenkins channel closing down (Agent disconnected)|build underlay packages
nightly_win_deb|3070|2024-04-21 04:00:01.305000|Packages mcap_vendor  Failed|build
Rci__nightly-performance_ubuntu_noble_amd64|2|2024-04-20 14:31:15.898000|Packages performance_test Failed|build workspace in isolation and install
gz-msgs11-debbuilder|269|2024-04-21 10:50:56.137000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|264|2024-04-20 12:43:52.861000|Failed to compile and install build|install build dependencies from debian/control
gz_gui-main-win|42|2024-04-21 09:06:13.675000|Packages gz-tools2  Failed|colcon compilation without test for dependencies of gz-gui9
gz_msgs-main-win|40|2024-04-20 16:06:07.090000|Packages gz-tools2  Failed|colcon compilation without test for dependencies of gz-msgs11
gz_sensors-main-win|38|2024-04-21 10:06:20.814000|Packages gz-tools2  Failed|colcon compilation without test for dependencies of gz-sensors9
gz_sim-ci-gz-sim8-homebrew-amd64|56|2024-04-20 13:50:09.992000|Failed to compile and install gz-sim8|install gz-sim8 dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 53.85% in 7 out of 13 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #2 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #3 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_server
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #4 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #5 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #6 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_client
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #7 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED)

Issue #8 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_subscription
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #9 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_service
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #10 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_qos_event__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #11 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_publisher (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2230 - CLOSED)

Issue #12 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #13 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_guard_condition (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2435 - OPEN)

Issue #14 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #15 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #16 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	rosidl_typesupport_c.test_service_type_support.gtest.missing_result
	rosidl_typesupport_c.test_message_type_support.gtest.missing_result
	rcutils.test_time.gtest.missing_result
	rcutils.test_strerror.gtest.missing_result
	rcutils.test_logging_custom_env2.gtest.missing_result
	rcutils.test_logging_custom_env.gtest.missing_result
	rcutils.test_filesystem.gtest.missing_result
	rclcpp_lifecycle.test_transition_wrapper.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result
	rclcpp_lifecycle.test_lifecycle_node.gtest.missing_result
	rclcpp_action.test_server_goal_handle.gtest.missing_result
	rclcpp_action.test_server.gtest.missing_result
	rclcpp_action.test_client.gtest.missing_result
	rclcpp.test_utilities.gtest.missing_result
	rclcpp.test_timer.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_subscription.gtest.missing_result
	rclcpp.test_storage_policy_common.gtest.missing_result
	rclcpp.test_static_single_threaded_executor.gtest.missing_result
	rclcpp.test_service.gtest.missing_result
	rclcpp.test_qos_event__rmw_fastrtps_cpp.gtest.missing_result
	rclcpp.test_qos_event__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_publisher.gtest.missing_result
	rclcpp.test_node_options.gtest.missing_result
	rclcpp.test_node_interfaces__node_waitables.gtest.missing_result
	rclcpp.test_node_interfaces__node_topics.gtest.missing_result
	rclcpp.test_node_interfaces__node_timers.gtest.missing_result
	rclcpp.test_node_interfaces__node_services.gtest.missing_result
	rclcpp.test_node_interfaces__node_parameters.gtest.missing_result
	rclcpp.test_node_interfaces__node_graph.gtest.missing_result
	rclcpp.test_node_interfaces__node_base.gtest.missing_result
	rclcpp.test_node.gtest.missing_result
	rclcpp.test_loaned_message.gtest.missing_result
	rclcpp.test_init_options.gtest.missing_result
	rclcpp.test_guard_condition.gtest.missing_result
	rclcpp.test_graph_listener.gtest.missing_result
	rclcpp.test_generic_client.gtest.missing_result
	rclcpp.test_expand_topic_or_service_name.gtest.missing_result
	rclcpp.test_exceptions.gtest.missing_result
	rclcpp.test_client_common.gtest.missing_result
	rclcpp.test_client.gtest.missing_result
	rcl_yaml_param_parser.test_parser.gtest.missing_result
	rcl_yaml_param_parser.test_parse.gtest.missing_result
	rcl.test_wait__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_wait__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_validate_enclave_name.gtest.missing_result
	rcl.test_subscription__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_subscription__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service_event_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_service__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_service__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_security.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_publisher__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_publisher__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_node__rmw_fastrtps_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_node__rmw_cyclonedds_cpp.gtest.missing_result (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	rcl.test_logging.gtest.missing_result
	rcl.test_init__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_init__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_guard_condition__rmw_cyclonedds_cpp.gtest.missing_result
	rcl.test_expand_topic_name.gtest.missing_result
	rcl.test_domain_id.gtest.missing_result
	rcl.test_context__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_context__rmw_cyclonedds_cpp.gtest.missing_result
	projectroot.test_transition_wrapper
	projectroot.test_time
	projectroot.test_strerror
	projectroot.test_service_type_support
	projectroot.test_server_goal_handle
	projectroot.test_parser
	projectroot.test_parse
	projectroot.test_message_type_support
	projectroot.test_logging_custom_env2
	projectroot.test_logging_custom_env
	projectroot.test_filesystem
	projectroot.test.test_wait__rmw_fastrtps_cpp
	projectroot.test.test_wait__rmw_cyclonedds_cpp
	projectroot.test.test_validate_enclave_name
	projectroot.test.test_subscription__rmw_fastrtps_cpp
	projectroot.test.test_subscription__rmw_cyclonedds_cpp
	projectroot.test.test_service_event_publisher__rmw_fastrtps_cpp
	projectroot.test.test_service_event_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_service__rmw_fastrtps_cpp
	projectroot.test.test_service__rmw_cyclonedds_cpp
	projectroot.test.test_security
	projectroot.test.test_rmw_impl_id_check_func__rmw_fastrtps_cpp
	projectroot.test.test_rmw_impl_id_check_func__rmw_cyclonedds_cpp
	projectroot.test.test_publisher__rmw_fastrtps_cpp
	projectroot.test.test_publisher__rmw_cyclonedds_cpp
	projectroot.test.test_node__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_node__rmw_cyclonedds_cpp (known issue in: https://github.com/ros2/rcl/issues/1123 - CLOSED)
	projectroot.test.test_logging
	projectroot.test.test_init__rmw_fastrtps_cpp
	projectroot.test.test_init__rmw_cyclonedds_cpp
	projectroot.test.test_guard_condition__rmw_fastrtps_cpp
	projectroot.test.test_guard_condition__rmw_cyclonedds_cpp
	projectroot.test.test_expand_topic_name
	projectroot.test.test_domain_id
	projectroot.test.test_context__rmw_fastrtps_cpp
	projectroot.test.test_context__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_utilities
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_qos_event__rmw_cyclonedds_cpp
	projectroot.test.rclcpp.test_node_options
	projectroot.test.rclcpp.test_node_interfaces__node_waitables
	projectroot.test.rclcpp.test_node_interfaces__node_topics
	projectroot.test.rclcpp.test_node_interfaces__node_timers
	projectroot.test.rclcpp.test_node_interfaces__node_services
	projectroot.test.rclcpp.test_node_interfaces__node_parameters
	projectroot.test.rclcpp.test_node_interfaces__node_graph
	projectroot.test.rclcpp.test_node_interfaces__node_base
	projectroot.test.rclcpp.test_loaned_message
	projectroot.test.rclcpp.test_init_options
	projectroot.test.rclcpp.test_generic_client
	projectroot.test.rclcpp.test_expand_topic_or_service_name
	projectroot.test.rclcpp.test_exceptions
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #17 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_client_common
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #18 in nightly_linux-aarch64_debug
- Failing 31.25% in 5 out of 16 builds
	projectroot.test.rclcpp.test_client
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #19 in nightly_win_rel
- Failing 31.25% in 5 out of 16 builds
	ros2cli.ros2cli.test.test_daemon.test_get_name

Issue #20 in Rci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 4 out of 4 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #21 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 80.0% in 4 out of 5 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #22 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #23 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #24 in gz_sim-ci-main-jammy-amd64
- Failing 100.0% in 4 out of 4 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #25 in nightly_win_rep
- Failing 30.77% in 4 out of 13 builds
	projectroot.test_action_client
	  (also in: "nightly_win_rel" job)

Issue #26 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp

Issue #27 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)

Issue #28 in nightly_linux_repeated
- Failing 18.75% in 3 out of 16 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #29 in nightly_win_rep
- Failing 23.08% in 3 out of 13 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #30 in nightly_win_rep
- Failing 23.08% in 3 out of 13 builds
	test_rclcpp.test_spin.spin_until_future_complete_interrupted

Issue #31 in nightly_win_rep
- Failing 23.08% in 3 out of 13 builds
	rclpy.test_time_source.xunit.missing_result
	projectroot.test_time_source

Issue #32 in nightly_win_rep
- Failing 23.08% in 3 out of 13 builds
	rcl_action.TestActionServerInitFini.test_action_server_init_fini
	rcl_action.TestActionGraphFixture__rmw_fastrtps_cpp.test_action_get_client_names_and_types_by_node
	rcl_action.TestActionClientWait.test_wait_set_add_action_client
	projectroot.test_wait
	projectroot.test_graph__rmw_fastrtps_cpp
	projectroot.test_action_server
	projectroot.test_action_interaction__rmw_fastrtps_cpp
	projectroot.test_action_communication__rmw_fastrtps_cpp
	  (also in: "nightly_win_rel" job)

Issue #33 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 2 out of 2 builds
	projectroot.benchmark_string_conversions
	  (also in: "Rci__benchmark_ubuntu_jammy_amd64" job)

Issue #34 in gz_fuel_tools-8-win
- Failing 40.0% in 2 out of 5 builds
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	  (also in: "gz_fuel_tools-8-win" job) (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)

Issue #35 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #36 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #37 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_humble_coverage" job)
	  (also in: "nightly_linux-aarch64_release" job)

Issue #38 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	rcl_action.TestActionCommunication__rmw_fastrtps_cpp.test_valid_goal_comm
	rcl_action.TestActionClientServerInteraction__rmw_fastrtps_cpp.test_interaction
	  (also in: "nightly_win_rel" job)

Issue #39 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	rcl_action.TestActionClientFixture.test_action_server_is_available_maybe_fail

Issue #40 in nightly_linux-rhel_debug
- Failing 12.5% in 2 out of 16 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)

Issue #41 in gz_common-5-win
- Failing 25.0% in 1 out of 4 builds
	PluginSpecialization.AccessTime

Issue #42 in gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Zip_TEST.test_ran
	UNIT_WorldIter_TEST.test_ran
	UNIT_WorldIdentifier_TEST.test_ran
	UNIT_Result_TEST.test_ran
	UNIT_RestClient_TEST.test_ran
	UNIT_Model_TEST.test_ran
	UNIT_ModelIter_TEST.test_ran
	UNIT_ModelIdentifier_TEST.test_ran
	UNIT_LocalCache_TEST.test_ran
	UNIT_JSONParser_TEST.test_ran
	UNIT_Interface_TEST.test_ran
	UNIT_Helpers_TEST.test_ran
	UNIT_FuelClient_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	UNIT_CollectionIdentifier_TEST.test_ran
	UNIT_ClientConfig_TEST.test_ran
	INTEGRATION_zip.test_ran

Issue #43 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Teleop_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/503 - CLOSED)
	UNIT_ShutdownButton_TEST.test_ran
	UNIT_Screenshot_TEST.test_ran
	UNIT_Publisher_TEST.test_ran
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran
	INTEGRATION_camera_tracking.test_ran

Issue #44 in gz_math-ign-math6-win
- Failing 25.0% in 1 out of 4 builds
	Stopwatch.StartStopReset

Issue #45 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_tpelib_World_TEST.test_ran
	UNIT_tpelib_Utils_TEST.test_ran
	UNIT_tpelib_Shape_TEST.test_ran
	UNIT_tpelib_Model_TEST.test_ran
	UNIT_tpelib_Link_TEST.test_ran
	UNIT_tpelib_Engine_TEST.test_ran
	UNIT_tpelib_Collision_TEST.test_ran
	UNIT_tpelib_CollisionDetector_TEST.test_ran
	UNIT_tpelib_AABBTree_TEST.test_ran
	UNIT_tpe_SimulationFeatures_TEST.test_ran
	UNIT_tpe_SDFFeatures_TEST.test_ran
	UNIT_tpe_EntityManagement_TEST.test_ran
	UNIT_tpe_Base_TEST.test_ran
	UNIT_WorldFeatures_TEST.test_ran
	UNIT_SpecifyData_TEST.test_ran
	UNIT_SDFFeatures_TEST.test_ran
	UNIT_FindFeatures_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	UNIT_FilterTuple_TEST.test_ran
	UNIT_Feature_TEST.test_ran
	UNIT_FeatureList_TEST.test_ran
	UNIT_EntityManagement_TEST.test_ran
	UNIT_CompositeData_TEST.test_ran
	UNIT_Cloneable_TEST.test_ran
	UNIT_Base_TEST.test_ran
	UNIT_AddedMassFeatures_TEST.test_ran
	PERFORMANCE_symbol_names.test_ran
	PERFORMANCE_ExpectData.test_ran
	INTEGRATION_deprecated_TEST.test_ran
	INTEGRATION_RequestFeatures.test_ran
	INTEGRATION_JointTypes3f.test_ran
	INTEGRATION_JointTypes3d.test_ran
	INTEGRATION_JointTypes2f.test_ran
	INTEGRATION_JointTypes2d.test_ran
	INTEGRATION_FrameSemantics3f.test_ran
	INTEGRATION_FrameSemantics3d.test_ran
	INTEGRATION_FrameSemantics2f.test_ran
	INTEGRATION_FrameSemantics2d.test_ran
	INTEGRATION_FeatureSystem.test_ran
	INTEGRATION_ExamplesBuild_TEST.test_ran
	INTEGRATION_DoublePendulum.test_ran
	INTEGRATION_CanReadWrite.test_ran
	COMMON_TEST_world_features_tpe.test_ran
	COMMON_TEST_world_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_world_features_bullet.test_ran
	COMMON_TEST_world_features_bullet-featherstone.test_ran
	COMMON_TEST_simulation_features_tpe.test_ran
	COMMON_TEST_simulation_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_simulation_features_bullet.test_ran
	COMMON_TEST_simulation_features_bullet-featherstone.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	COMMON_TEST_shape_features_tpe.test_ran
	COMMON_TEST_shape_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_shape_features_bullet.test_ran
	COMMON_TEST_shape_features_bullet-featherstone.test_ran
	COMMON_TEST_link_features_tpe.test_ran
	COMMON_TEST_link_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_link_features_bullet.test_ran
	COMMON_TEST_link_features_bullet-featherstone.test_ran
	COMMON_TEST_kinematic_features_tpe.test_ran
	COMMON_TEST_kinematic_features_dartsim.test_ran
	COMMON_TEST_kinematic_features_bullet.test_ran
	COMMON_TEST_kinematic_features_bullet-featherstone.test_ran
	COMMON_TEST_joint_transmitted_wrench_features_tpe.test_ran
	COMMON_TEST_joint_transmitted_wrench_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_joint_transmitted_wrench_features_bullet.test_ran
	COMMON_TEST_joint_transmitted_wrench_features_bullet-featherstone.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_joint_features_tpe.test_ran
	COMMON_TEST_joint_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_joint_features_bullet.test_ran
	COMMON_TEST_joint_features_bullet-featherstone.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_free_joint_features_tpe.test_ran
	COMMON_TEST_free_joint_features_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_free_joint_features_bullet.test_ran
	COMMON_TEST_free_joint_features_bullet-featherstone.test_ran
	COMMON_TEST_detachable_joint_tpe.test_ran
	COMMON_TEST_detachable_joint_dartsim.test_ran
	COMMON_TEST_detachable_joint_bullet.test_ran
	COMMON_TEST_detachable_joint_bullet-featherstone.test_ran
	COMMON_TEST_construct_empty_world_tpe.test_ran
	COMMON_TEST_construct_empty_world_dartsim.test_ran (known issue in: https://github.com/gazebosim/gz-physics/issues/442 - CLOSED)
	COMMON_TEST_construct_empty_world_bullet.test_ran
	COMMON_TEST_construct_empty_world_bullet-featherstone.test_ran
	COMMON_TEST_collisions_tpe.test_ran
	COMMON_TEST_collisions_dartsim.test_ran
	COMMON_TEST_collisions_bullet.test_ran
	COMMON_TEST_collisions_bullet-featherstone.test_ran
	COMMON_TEST_basic_test_tpe.test_ran
	COMMON_TEST_basic_test_dartsim.test_ran
	COMMON_TEST_basic_test_bullet.test_ran
	COMMON_TEST_basic_test_bullet-featherstone.test_ran
	COMMON_TEST_addexternalforcetorque_tpe.test_ran
	COMMON_TEST_addexternalforcetorque_dartsim.test_ran
	COMMON_TEST_addexternalforcetorque_bullet.test_ran
	COMMON_TEST_addexternalforcetorque_bullet-featherstone.test_ran
	COMMON_TEST_added_mass_tpe.test_ran
	COMMON_TEST_added_mass_dartsim.test_ran
	COMMON_TEST_added_mass_bullet.test_ran
	COMMON_TEST_added_mass_bullet-featherstone.test_ran

Issue #46 in gz_rendering-ci-gz-rendering8-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_WireBox_TEST_ogre_gl3plus.test_ran
	UNIT_WireBox_TEST_ogre2_metal.test_ran
	UNIT_Visual_TEST_ogre_gl3plus.test_ran
	UNIT_Visual_TEST_ogre2_metal.test_ran
	UNIT_Utils_TEST_ogre_gl3plus.test_ran
	UNIT_Utils_TEST_ogre2_metal.test_ran
	UNIT_TransformController_TEST_ogre_gl3plus.test_ran
	UNIT_TransformController_TEST_ogre2_metal.test_ran
	UNIT_ThermalCamera_TEST_ogre_gl3plus.test_ran
	UNIT_ThermalCamera_TEST_ogre2_metal.test_ran
	UNIT_Text_TEST_ogre_gl3plus.test_ran
	UNIT_Text_TEST_ogre2_metal.test_ran
	UNIT_ShaderType_TEST.test_ran
	UNIT_ShaderParams_TEST.test_ran
	UNIT_ShaderParam_TEST.test_ran
	UNIT_SegmentationCamera_TEST_ogre_gl3plus.test_ran
	UNIT_SegmentationCamera_TEST_ogre2_metal.test_ran
	UNIT_Scene_TEST_ogre_gl3plus.test_ran
	UNIT_Scene_TEST_ogre2_metal.test_ran
	UNIT_RenderingIface_TEST_ogre_gl3plus.test_ran
	UNIT_RenderingIface_TEST_ogre2_metal.test_ran
	UNIT_RenderTarget_TEST_ogre_gl3plus.test_ran
	UNIT_RenderTarget_TEST_ogre2_metal.test_ran
	UNIT_RenderPassSystem_TEST_ogre_gl3plus.test_ran
	UNIT_RenderPassSystem_TEST_ogre2_metal.test_ran
	UNIT_RenderEngine_TEST_ogre_gl3plus.test_ran
	UNIT_RenderEngine_TEST_ogre2_metal.test_ran
	UNIT_RenderEngineManager_TEST_ogre_gl3plus.test_ran
	UNIT_RenderEngineManager_TEST_ogre2_metal.test_ran
	UNIT_RayQuery_TEST_ogre_gl3plus.test_ran
	UNIT_RayQuery_TEST_ogre2_metal.test_ran
	UNIT_Projector_TEST_ogre_gl3plus.test_ran
	UNIT_Projector_TEST_ogre2_metal.test_ran
	UNIT_PixelFormat_TEST.test_ran
	UNIT_ParticleEmitter_TEST_ogre_gl3plus.test_ran
	UNIT_ParticleEmitter_TEST_ogre2_metal.test_ran
	UNIT_OrthoViewController_TEST_ogre_gl3plus.test_ran
	UNIT_OrthoViewController_TEST_ogre2_metal.test_ran
	UNIT_OrbitViewController_TEST_ogre_gl3plus.test_ran
	UNIT_OrbitViewController_TEST_ogre2_metal.test_ran
	UNIT_Node_TEST_ogre_gl3plus.test_ran
	UNIT_Node_TEST_ogre2_metal.test_ran
	UNIT_MoveToHelper_TEST_ogre_gl3plus.test_ran
	UNIT_MoveToHelper_TEST_ogre2_metal.test_ran
	UNIT_Mesh_TEST_ogre_gl3plus.test_ran
	UNIT_Mesh_TEST_ogre2_metal.test_ran
	UNIT_MeshDescriptor_TEST_ogre_gl3plus.test_ran
	UNIT_MeshDescriptor_TEST_ogre2_metal.test_ran
	UNIT_Material_TEST_ogre_gl3plus.test_ran
	UNIT_Material_TEST_ogre2_metal.test_ran
	UNIT_Marker_TEST_ogre_gl3plus.test_ran
	UNIT_Marker_TEST_ogre2_metal.test_ran
	UNIT_Light_TEST_ogre_gl3plus.test_ran
	UNIT_Light_TEST_ogre2_metal.test_ran
	UNIT_LightVisual_TEST_ogre_gl3plus.test_ran
	UNIT_LightVisual_TEST_ogre2_metal.test_ran
	UNIT_LidarVisual_TEST_ogre_gl3plus.test_ran
	UNIT_LidarVisual_TEST_ogre2_metal.test_ran
	UNIT_LensFlarePass_TEST_ogre_gl3plus.test_ran
	UNIT_LensFlarePass_TEST_ogre2_metal.test_ran
	UNIT_InertiaVisual_TEST_ogre_gl3plus.test_ran
	UNIT_InertiaVisual_TEST_ogre2_metal.test_ran
	UNIT_Heightmap_TEST_ogre_gl3plus.test_ran
	UNIT_Heightmap_TEST_ogre2_metal.test_ran
	UNIT_Grid_TEST_ogre_gl3plus.test_ran
	UNIT_Grid_TEST_ogre2_metal.test_ran
	UNIT_GraphicsAPI_TEST.test_ran
	UNIT_GlobalIllumination_TEST_ogre_gl3plus.test_ran
	UNIT_GlobalIllumination_TEST_ogre2_metal.test_ran
	UNIT_GizmoVisual_TEST_ogre_gl3plus.test_ran
	UNIT_GizmoVisual_TEST_ogre2_metal.test_ran
	UNIT_GaussianNoisePass_TEST_ogre_gl3plus.test_ran
	UNIT_GaussianNoisePass_TEST_ogre2_metal.test_ran
	UNIT_Capsule_TEST_ogre_gl3plus.test_ran
	UNIT_Capsule_TEST_ogre2_metal.test_ran
	UNIT_Camera_TEST_ogre_gl3plus.test_ran
	UNIT_Camera_TEST_ogre2_metal.test_ran
	UNIT_COMVisual_TEST_ogre_gl3plus.test_ran
	UNIT_COMVisual_TEST_ogre2_metal.test_ran
	UNIT_BoundingBox_TEST_ogre_gl3plus.test_ran
	UNIT_BoundingBox_TEST_ogre2_metal.test_ran
	UNIT_BoundingBoxCamera_TEST_ogre_gl3plus.test_ran
	UNIT_BoundingBoxCamera_TEST_ogre2_metal.test_ran
	UNIT_AxisVisual_TEST_ogre_gl3plus.test_ran
	UNIT_AxisVisual_TEST_ogre2_metal.test_ran
	UNIT_ArrowVisual_TEST_ogre_gl3plus.test_ran
	UNIT_ArrowVisual_TEST_ogre2_metal.test_ran
	REGRESSION_reload_engine_ogre_gl3plus.test_ran
	REGRESSION_reload_engine_ogre2_metal.test_ran
	PERFORMANCE_scene_factory_ogre_gl3plus.test_ran
	PERFORMANCE_scene_factory_ogre2_metal.test_ran
	INTEGRATION_wide_angle_camera_ogre_gl3plus.test_ran
	INTEGRATION_wide_angle_camera_ogre2_metal.test_ran
	INTEGRATION_waves_ogre_gl3plus.test_ran
	INTEGRATION_waves_ogre2_metal.test_ran
	INTEGRATION_thermal_camera_ogre_gl3plus.test_ran
	INTEGRATION_thermal_camera_ogre2_metal.test_ran
	INTEGRATION_sky_ogre_gl3plus.test_ran
	INTEGRATION_sky_ogre2_metal.test_ran
	INTEGRATION_shadows_ogre_gl3plus.test_ran
	INTEGRATION_shadows_ogre2_metal.test_ran
	INTEGRATION_segmentation_camera_ogre_gl3plus.test_ran
	INTEGRATION_segmentation_camera_ogre2_metal.test_ran
	INTEGRATION_scene_ogre_gl3plus.test_ran
	INTEGRATION_scene_ogre2_metal.test_ran
	INTEGRATION_render_pass_ogre_gl3plus.test_ran
	INTEGRATION_render_pass_ogre2_metal.test_ran
	INTEGRATION_projector_ogre_gl3plus.test_ran
	INTEGRATION_projector_ogre2_metal.test_ran
	INTEGRATION_mesh_ogre_gl3plus.test_ran
	INTEGRATION_mesh_ogre2_metal.test_ran
	INTEGRATION_load_unload_ogre_gl3plus.test_ran
	INTEGRATION_load_unload_ogre2_metal.test_ran
	INTEGRATION_lidar_visual_ogre_gl3plus.test_ran
	INTEGRATION_lidar_visual_ogre2_metal.test_ran
	INTEGRATION_heightmap_ogre_gl3plus.test_ran
	INTEGRATION_heightmap_ogre2_metal.test_ran
	INTEGRATION_gpu_rays_ogre_gl3plus.test_ran
	INTEGRATION_gpu_rays_ogre2_metal.test_ran
	INTEGRATION_depth_camera_ogre_gl3plus.test_ran
	INTEGRATION_depth_camera_ogre2_metal.test_ran
	INTEGRATION_camera_ogre_gl3plus.test_ran
	INTEGRATION_camera_ogre2_metal.test_ran
	INTEGRATION_boundingbox_camera_ogre_gl3plus.test_ran
	INTEGRATION_boundingbox_camera_ogre2_metal.test_ran

Issue #47 in gz_sim-ci-main-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	<init>.ServerRepeat/SceneBroadcasterTest
	<init>.NetworkHandshake

Issue #48 in nightly_win_rep
- Failing 7.69% in 1 out of 13 builds
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_not_connecting_0
	projectroot.gtest_timeout_subscriber__rmw_cyclonedds_cpp
	projectroot.gtest_spin__rmw_fastrtps_cpp
	TestExecutors/StaticSingleThreadedExecutor.testSpinUntilSharedFutureComplete
	TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureComplete
	TestExecutors/StaticSingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	TestExecutors/StaticSingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	TestExecutors/StaticSingleThreadedExecutor.testRaceConditionAddNode
	TestExecutors/StaticSingleThreadedExecutor.spin_some_max_duration
	TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	TestExecutors/StaticSingleThreadedExecutor.spinWithTimer
	TestExecutors/StaticSingleThreadedExecutor.spinWhileAlreadySpinning
	TestExecutors/StaticSingleThreadedExecutor.spinAll
	TestExecutors/StaticSingleThreadedExecutor.emptyExecutor
	TestExecutors/StaticSingleThreadedExecutor.detachOnDestruction
	TestExecutors/StaticSingleThreadedExecutor.addTemporaryNode
	TestExecutors/StaticSingleThreadedExecutor.addNodeTwoExecutors
	TestExecutors/MultiThreadedExecutor.testSpinUntilSharedFutureComplete
	TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	TestExecutors/MultiThreadedExecutor.testSpinUntilFutureComplete
	TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	TestExecutors/MultiThreadedExecutor.testRaceConditionAddNode
	TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	TestExecutors/MultiThreadedExecutor.spinWithTimer
	TestExecutors/MultiThreadedExecutor.spinWhileAlreadySpinning
	TestExecutors/MultiThreadedExecutor.spinAll
	TestExecutors/MultiThreadedExecutor.emptyExecutor
	TestExecutors/MultiThreadedExecutor.addTemporaryNode
	TestExecutors/MultiThreadedExecutor.addNodeTwoExecutors
	TestExecutors/EventsExecutor.testSpinUntilSharedFutureComplete
	TestExecutors/EventsExecutor.testSpinUntilFutureCompleteWithTimeout
	TestExecutors/EventsExecutor.testSpinUntilFutureCompleteNoTimeout
	TestExecutors/EventsExecutor.testSpinUntilFutureCompleteInterrupted
	TestExecutors/EventsExecutor.testSpinUntilFutureComplete
	TestExecutors/EventsExecutor.testSpinNodeUntilFutureCompleteNodePtr
	TestExecutors/EventsExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	TestExecutors/EventsExecutor.testRaceConditionAddNode
	TestExecutors/EventsExecutor.spin_some_max_duration
	TestExecutors/EventsExecutor.spin_some
	TestExecutors/EventsExecutor.spinWithTimer
	TestExecutors/EventsExecutor.spinWhileAlreadySpinning
	TestExecutors/EventsExecutor.spinAll
	TestExecutors/EventsExecutor.emptyExecutor
	TestExecutors/EventsExecutor.detachOnDestruction
	TestExecutors/EventsExecutor.addTemporaryNode
	TestExecutors/EventsExecutor.addNodeTwoExecutors
	TestExecutors.testSpinWithNonDefaultContext
	TestExecutors.testSpinUntilFutureCompleteNodePtr
	TestExecutors.testSpinUntilFutureCompleteNodeBasePtr

@github-actions github-actions bot mentioned this issue Apr 22, 2024
Closed
@Crola1702 Crola1702 mentioned this issue Jul 31, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
buildfarmer-log Log of issues in the buildfarm
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Blast545 @Crola1702