2024-09-16 ODD working group weekly meeting (206)

[Igata-ctrl]

Attendee

• Tran Huu Nhat Huy (Student/Tokyo Institute of Technology)
• Zain Khawaja (AWF)
• Ahmed Ebrahim (Leo Drive)
• David Walmroth (PIX Moving)
• @mitsudome-r Ryohsuke Mitsudome (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On August 12, 2024, 357/423 regular and prototype scenarios succeeded.

  • 139/142 public road bus scenarios were successful.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego is ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • UC-PB-008-0001_20_20-Shalun : Rare random system error. Succeeded in on-demand execution today.
    • UC-LD-001-0010-1-Gebze : Ve=2.7 case was failing. Ego size modified to allow more lateral space within the lane. -> Succeeded.

  • 184/198 pull out scenarios were successful.

    • UC-v2-F-15-00002_002_case03 : Failed for (t=0, D=4) case. Ego unnecessarily stopped after successfully pulling out onto the main lane. -> Succeeded after explicitly setting Ego max speed 4.22.
    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00003_001_case04 : Failed for (D1=10, D2=8, t=-0.9)-> Succeeded after explicitly setting Ego max speed 4.22.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed in 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : 2/36 cases failed due to unnecessary stop. After explicitly assigning Ve=4.2 to ego max speed, they succeeded.
    • UC-v2-F-15-00103_001_case(01|02)_cmn_general : Obstacle stop prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe is not capable of it.
    • UC-v2-F-15-01001_001_case01_cmn_general : This scenario has been succeeding for recent two months, but it failed this week by the PR merged last week. -> TIER IV CI/CD team kindly corrected the issue and succeeded. @Igata-ctrl is inviting them so that they can correct other scenarios if necessary.

  • 34/83 Dense Urban scenarios were successful. The success case decreased because the parameters

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case(1|2), UC-NTR-001-0003_case1, UC-NTR-001-0005 : NPC's "take over finished" condition was not met though the rude NPS behaves as intended. -> Succeeded except for case1, Vn=5.55. For this failing case, planning module improvement is awaited.
    • UC-NTR-001-0003_case1 : Succeeded for Vn=11.11 case. Ego starts before NPC finishes the takeover as opposed to the scenario intention. Is this dependent on LHT/RHT?
    • UC-NTR-001-0003_case2 : Succeeded after loading the latest version map (fix local diff problem). -> 1/3 failed this week as Ego started before Npc takeover was finished.
    • UC-NTR-001-0004_case2-Hsinchu was created with the same use case on Hsinchu map and it succeeded.
    • UC-NTR-001-0005 : Failed for all parameter cases. -> Succeeded after the tolerance of the distance (to Point0) condition was modified from 0.001 to 1, the same level as other scenarios.
    • UC-NTR-001-0007 : NPC1 spawns at a different position than in the scenario (a system error).
      • UC-NTR-001-0007-Hsinchu was created with the same use case on Hsinchu map and it succeeded.
    • UC-NTR-001-0008 : NPC1 spawns at a different position than in the scenario (a system error).
      • UC-NTR-001-0008-Hsinchu was created with the same use case on Hsinchu map. Still in debugging.
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC using the space outside of its lane.
    • UC-NTR-002-0004 : Fails with the excess deceleration. -> Succeeded after "min_acc" threshold was changed from -1.5(-0.15G) to -3.92(0.4G).-> Revert to -1.5 before next week and expect future success after planning improvement.
    • UC-NTR-003-0001 : Fails with crash. -> Succeeded after "min_acc" was changed from -1.5 to -7.84(0.8G) and "D" from 5 to 3. -> Revert to -1.5 before next week and expect future success after planning improvement.
    • UC-NTR-003-0002 : Fails with excess deceleration condition when EGO detects the oncoming NPC. -> Succeeded after "min_acc" was changed from -1.5 to -7.84(0.8G).-> Revert to -1.5 before next week and expect future success after planning improvement.
    • UC-NTR-003-0003 : Fails with crash. -> Succeeded after changing "D" from 30 to 50.-> Reverted to 30 and the crash to be avoided by future planning improvement.
    • UC-NTR-004-0001 : EGO does not avoid the NPCs in the wrong direction. "max_negative_acc" was not modified,
    • UC-NTR-005-0001 : Failed as "act_slow_down_npc" trigger condition was not met. -> Succeeded after "follow_dist" was changed from 8.5m(2.2sec) to 19.2m(5 sec). -> Need to modify action1 trigger condition
      • UC-NTR-005-0001_2 was created in which Npc slows down at Point0 and see if Ego does not go too slow or come too close to npc. Reverted the time-distance condition to 3sec and the hard braking threshold to -1.5.
    • UC-VRU-001-0001_case1 : Was failing for Vp2=0 case -> Succeeded after Ped2 start condition and distance (D) were adjusted. -> Reverted D to 15 and the crash or hard braking to be avoided by future planning improvement.
    • UC-VRU-001-0002 : Was failing with excess deceleration condition when EGO detects the pedestrian (jay walker). -> Succeeded after "min_acc" was changed from -1.5(-0.15G) to -7.88(0.8G). Revert to -1.5 before next week and expect future success after planning improvement.
    • UC-VRU-002-0001_case(1|2) : Was failing with obstacle stop/timeout. -> Succeeded after "park_offset" was changed from2 to 2.5 and "min_acc" changed from -1.5(-0.15G) to -3.92(0.4G), Bicycle start condition corrected and one point added to FollowTrajectoryAction. As for case2, D was changed to (15, 17, 19). -> Revert D (in case2) and "min_acc" to -1.5 before next week and expect future success after planning improvement.
    • UC-VRU-002-0002_case(1|2) : Was failing with excess deceleration condition when EGO tries to pass the bicycle. -> Succeeded after D was changed to 14 or 18 and min_acc changed from -1.5(-0.15G) to -3.92(0.4G). -> D and min_acc reverted to original.
    • UC-VRU-002-0003_case(1|2) : Was failing with obstacle stop/timeout. -> Succeeded after "park_offset" was changed from2 to 2.5 and "min_acc" changed from -1.5(-0.15G) to -3.92(0.4G), Bicycle start condition corrected and one point added to FollowTrajectoryAction. -> Failed again but succeeded after "bicycle_side_offset" is changed from 1.5 to 2m and "keep:true" is set for RelativeDistance (longitudinal) condition. -> "min_acc" reverted to -1.5.
    • UC-VRU-002-0004_case1 : Was failing with excess deceleration condition. -> Succeeded after D, "min_negative_acc" and bicycle start condition were modified. (Higher deceleration should be allowed to void collision.) -> Failed again but succeeded after the final FollowTrajectroyAction point was set at s=140.-> Revert D and "min_acc" to -1.5 before next week and expect future success after planning improvement.
    • UC-VRU-002-0004_case2 : Was failing with excess deceleration condition. -> Succeeded after "min_negative_acc" and FollowTrajectoryAction points were modified. (Higher deceleration should be allowed to void collision.)-> Revert D and "min_acc" to -1.5 before next week and expect future success after planning improvement.
    • UC-VRU-002-0005 : Was failing with excess deceleration condition. -> Succeeded after adding Ve=5.6(20kph) and changing "RelativeDistance Type" from "lateral" to "euclidian" and changing the lateral position of Bicycle0|1. (May need to observe the safety margins when playing back.) -> Revert "min_acc" to -1.5 before next week and expect future success after planning improvement.

• On August 18, 2024, 364/427 regular and prototype scenarios succeeded.

  • 140/142 public road bus scenarios were successful. Identical result as last monday.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego is ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.

  • 186/198 pull out scenarios were successful.

    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00003_001_case05 : Failed for (D1=8, D2=4, t=-0.9) -> act_ego_nostop condition was modified to exclude the stationary time before starting.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case(01|02)_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe is not capable of it.

  • 38/87 Dense Urban scenarios were successful.

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case(1|2), UC-NTR-001-0003_case(1|2), UC-NTR-001-0005 : NPC's "take over finished" condition was not met though the rude NPS behaves as intended. -> Succeeded except for some parameter cases. For these failing cases, planning module improvement is awaited.
    • UC-NTR-001-0003_case1 : Succeeded for Vn=11.11 case. Ego starts before NPC finishes the takeover as opposed to the scenario intention. Is this dependent on LHT/RHT?
    • UC-NTR-001-0003_case2 : Succeeded after loading the latest version map (fix local diff problem). -> 1/3 failed this week as Ego started before Npc takeover was finished.
    • UC-NTR-001-0004_case2 : Ego and Npc spawn at unexpected position. Probably a map associated issue and this scenario should be replace with UC-NTR-001-0004_case2-Hsinchu (the same use case on Hsinchu map, successful.)
    • UC-NTR-001-0005 : Failed for all parameter cases. -> "act_check_ego_moved_when_overtake_over" condition is not met.
    • UC-NTR-001-0008 : NPC1 spawns at a different position than in the scenario. -> Succeeded this time.
    • UC-NTR-001-0008-Hsinchu was created with the same use case on Hsinchu map. -> Succeeded after deleting unnecessary FollowTrajectoryAction command in the scenario.
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC using the space outside of its lane.
    • UC-NTR-002-0004 : Failed with the excess deceleration.
    • UC-NTR-003-0001 : Failed with the excess deceleration.
    • UC-NTR-003-0002 : Failed with the excess deceleration.
    • UC-NTR-003-0003 : Failed with a crash. -> The crash to be avoided by future planning improvement.
    • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
    • UC-NTR-005-0001 : Failed as "act_slow_down_npc" trigger condition was not met. -> Succeeded after "follow_dist" was changed from 8.5m(2.2sec) to 19.2m(5 sec). -> Need to modify action1 trigger condition.
      • UC-NTR-005-0001_2 was created to address the issue above in which Npc slows down at Point0 and see if Ego does not go too slow or come too close to npc. The time-distance condition is set to 3sec and the hard braking threshold is -1.5. -> Succeeded.
    • UC-VRU-001-0002 : Failed with the excess deceleration.
    • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0004_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0005 : Failed with the excess deceleration.

• On August 25, 2024, 335/427 regular and prototype scenarios succeeded.

  • 110/142 public road bus scenarios were successful. The number of filed scenarios increased from 2 to 32.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego is ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 184/198 pull out scenarios were successful.

    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00003_001_case05 : Failed for (D1=8, D2=4, t=-0.9) -> act_ego_nostop condition was modified to exclude the stationary time before starting.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case(01|02)_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe is not capable of it.
    • UC-v2-F-15-00004_001_case03 : Ego stopped unnecessarily after pulling out before reaching the destination. (-> It succeeded after explicitly setting Ve=4.2.)
    • UC-v2-F-15-00006_001_case03_cmn_general : Failed for Ve=2.77 case. Ego stopped unnecessarily after pulling out before reaching the destination.

  • 41/ 90 Dense Urban scenarios were successful.

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case1 : Failed for Vn=5.55 case with the excess deceleration.
    • UC-NTR-001-0003_case1 : Failed for Vn=5.55 (excess deceleration) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-001-0003_case2 : Failed for Vn=5.55 (Ego moved before Npc takeover finish) and 8.33 (Ego stopped before destination).
    • UC-NTR-001-0005 : Failed for Vn=5.55 (excess deceleration), 11.11 (Ego moved before Npc takeover finish) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-001-0007 : NPC1 spawns at a different position than defined in the scenario (a system error).
      • UC-NTR-001-0007-Hsinchu was created with the same use case on Hsinchu map and it succeeded.
    • UC-NTR-002-0001_case(1|2) : Npc overtook Ego Successfully.
    • UC-NTR-002-0002_case1 : 2/3 cases failed with excess deceleration (<-2.499m/s2).
    • UC-NTR-002-0002_case2 : 2/3 cases failed with unnecessary stop of Ego against oncoming Npc.
    • UC-NTR-002-0004 : Failed with the excess deceleration.
    • UC-NTR-003-0001 : Failed with the excess deceleration.
    • UC-NTR-003-0002 : Failed with the excess deceleration.
    • UC-NTR-003-0003 : Failed with a crash. -> The crash to be avoided by future planning improvement.
    • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
    • UC-NTR-005-0001_2 was enhanced to test different Vs and As (2x2 = 4 patterns). (Min_distance to Npc was set to RSS safety distance.)
    • UC-VRU-001-0001_case(1|2) : 3 cases failed with stopping against pedestrians (obstacle_stop).
    • UC-VRU-001-0002 : Failed with the excess deceleration.
    • UC-VRU-002-0001_case1 : Failed with the excess deceleration.
    • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0004_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0005 : Failed with the excess deceleration.

• On September 1, 2024, 343/429 regular and prototype scenarios succeeded.

  • 111/142 public road bus scenarios were successful. Almost identical to last week result.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego is ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 185/198 pull out scenarios were successful.

    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00003_001_case05 : Failed for (D1=8, D2=4, t=-0.9) -> act_ego_nostop condition was modified to exclude the stationary time before starting.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case(01|02)_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe is not capable of it.
    • UC-v2-F-15-00006_001_case03_cmn_general : Failed for Ve=2.77 case. Ego stopped unnecessarily after pulling out before reaching the destination.

  • 47/ 89 Dense Urban scenarios were successful.

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case1 : Failed for Vn=5.55 case with the excess deceleration.
    • UC-NTR-001-0003_case1 : Failed for Vn=5.55 (excess deceleration) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-001-0003_case2 : Failed for Vn=5.55 (Ego moved before Npc takeover finish) and 8.33 (Ego stopped before destination).
    • UC-NTR-001-0005 : Failed for Vn=5.55 (excess deceleration), 11.11 (Ego moved before Npc takeover finish) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC using the space outside of its lane.
    • UC-NTR-002-0004 : Failed with the excess deceleration.
    • UC-NTR-003-0001 : Failed with the excess deceleration.
    • UC-NTR-003-0002 : Failed with the excess deceleration.
    • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
    • UC-NTR-005-0001_2 was enhanced to test different Vs and As (2x2 = 4 patterns). (Min_distance to Npc was set to RSS safety distance.)
    • UC-VRU-001-0002 : Failed with the excess deceleration.
    • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0004_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0005 : Failed with the excess deceleration.

• On September 8, 2024, 344/432 regular and prototype scenarios succeeded.

  • 111/142 public road bus scenarios were successful. Identical to the result of last week.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego is ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 182/198 pull out scenarios were successful.

    • UC-v2-F-15-00001_001_case01 : Failed with the unnecessary stop. (Looks OK after explicitly setting Ego max speed to 15km/h.)
    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00003_001_case03 : 1/4 case failed with the unnecessary stop. (Looks OK after explicitly setting Ego max speed to 15km/h.)
    • UC-v2-F-15-00003_002_case02 : 1/4 case failed with the unnecessary stop. (Looks OK after explicitly setting Ego max speed to 15km/h.)
    • UC-v2-F-15-00005_001_case01 : A random system error. (Succeeded by a simple retry.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_004_case02_cmn_general : Failed with the unnecessary stop. (Looks OK after explicitly setting Ego max speed to 15km/h.)
    • UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe is not capable of it.

  • 45/96 Dense Urban scenarios were successful.

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case1 : Failed for Vn=5.55 case with the excess deceleration.
    • UC-NTR-001-0003_case1 : Failed for Vn=5.55 (excess deceleration) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-001-0003_case2 : Failed for Vn=5.55 (Ego moved before Npc takeover finish) and 8.33 (Ego stopped before destination).
    • UC-NTR-001-0005 : Failed for Vn=5.55 (excess deceleration), 11.11 (Ego moved before Npc takeover finish) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC using the space outside of its lane.
    • UC-NTR-002-0004 : Failed with the excess deceleration.
    • UC-NTR-003-0001 : Failed with the excess deceleration.
    • UC-NTR-003-0002 : Failed with the excess deceleration.
    • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
    • UC-NTR-005-0001_2 was enhanced to test different Vs and As (2x2 = 4 patterns). (Min_distance to Npc was set to RSS safety distance.)
    • UC-VRU-001-0002 : Failed with the excess deceleration.
    • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0004_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0005 : Failed with the excess deceleration.

  • 51/92 Dense Urban scenarios were successful.

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case1 : Failed for Vn=5.55 case with the excess deceleration.
    • UC-NTR-001-0003_case1 : Failed for Vn=5.55 (excess deceleration) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-001-0003_case2 : Failed for Vn=5.55 (Ego moved before Npc takeover finish) and 8.33 (Ego stopped before destination).
    • UC-NTR-001-0005 : Failed for Vn=5.55 (excess deceleration), 11.11 (Ego moved before Npc takeover finish) and 8.33 (Ego moved before Npc takeover finish).
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC using the space outside of its lane.
    • UC-NTR-002-0004 : Failed with the excess deceleration.
    • UC-NTR-003-0001 : Failed with the excess deceleration.
    • UC-NTR-003-0002 : Failed with the excess deceleration.
    • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
    • UC-NTR-005-0001_2 was enhanced to test different Vs and As (2x2 = 4 patterns). (Min_distance to Npc was set to RSS safety distance.)
    • UC-VRU-001-0002 : Failed with the excess deceleration.
    • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0004_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0005 : Failed with the excess deceleration.

• On September 15, 2024, 296/340 regular and prototype scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of last week.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 185/198 pull out scenarios were successful.

    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00003_001_case05 : 1/4 case failed with the unnecessary stop. (Looks OK after explicitly setting Ego max speed to 15km/h.)
    • UC-v2-F-15-00003_002_case03 : 1/4 case failed with the unnecessary stop. (Looks OK after explicitly setting Ego max speed to 15km/h.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it.

  • 51/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

    • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
    • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
    • UC-NTR-001-0005 :All 3 cases failed for "Ego moved before Npc takeover finished.
    • UC-NTR-001-0008 : Failed for Vslow= 3.472 case as Ego decelerated to < 80% of Vslow.
    • UC-NTR-001-0009 : Failed for as Ego decelerated to < 80% of Vslow. (After explicitly setting Ego max speed to 4.2m/s, it then causes a crash against Npc.)
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC. (All of these look OK after explicitly setting Ego max speed to 15km/h.)
    • UC-NTR-002-0004 : Failed with the excess deceleration.
    • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
    • UC-NTR-005-0001 : "act_slow_down_npc" condition seems to be unmet.
    • UC-VRU-001-0001_case1 : Failed with unnecessary stop, or "act_check_ego_stopped_in_overtaking".
    • UC-VRU-001-0001_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
    • UC-VRU-002-0004_case2 : Failed with the excess deceleration.
    • UC-VRU-002-0005 : Failed with the excess deceleration.

• [recap of the discussions until last week]

• Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using UserDefinedValueCondition like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working on UserDefinedValueCondition.

• For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.

• @Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.

• @Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.

• Discussion (RSS distance and comfortable deceleration) pages have been added to the Use cases for Dense Urban Area 2024 presentation.

• Reached a consensus to use RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration for low speed shuttle.

• Discussion of the condition/threshold setting so that we can better track the Autoware maturity through scenario simulation.

• After the discussion, it turned out "min_acc" parameter should better be reverted to original value of -1.5 (this week).

• Current discussion with Safety Pool

  • [Map] Successfully converted a sample OpenDRIVE map from Safety Pool database to Lanelet2, but we have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead) proposed in May TSC that Autoware should consider accepting OpenDRIVE maps in a direct fashion as it solves all these issues, given that we may need to map-less autonomy, it may not worth switching map format from now.
    • Speed limit, lane change flag and LHT/RHT are lost through the conversion.
    • The converted Lanelet2 map contains a lot of unnecessary points on line strings. (e.g. 16,000points on 1 km one way straight road)
  • [Scenario] Autoware requires EGO's destination in each scenario, while all of the Safety Pool scenarios do not have the destination. Though Safety Pool once suggested to run Autoware in the “roaming” mode which does not require the destination in the scenario, it sounds unrealistic for Autoware to operate without destination as the destination is almost always an important part of the use case/scenario intention. (e.g. turning left/right).
    • In the Safety Pool Webinar, they demonstrated Autoware running with Safety Pool scenario, but they say that it requires some special settings for it and it is not scalable.
  • [Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle the parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios is not too big, we might be able to convert them from .xosc to .yaml format manually. (Currently, no automated converter is available.) If the scenario simulator v2 is installed on-premise, it will accept the .xosc file in a direct fashion.
  • [Scenario availability] After signing up for the Safety Pool database, the number of the accessible/visible scenarios are limited. 32,588 scenarios are visible but most of the scenarios are marked with "access restricted" and only 1957 scenarios are accessible. The total number of the scenario which Safety Pool claims is one million. Still waiting for an answer from Safety Pool

• After a brief hearing form TIER IV CI/CD team, it seems that the storage is becoming the bottle neck with Autoware Evaluator (for AWF organization use). There are two settings to save the storage space.

  • [Log Expiration] : Currently set to 60 days. We can revise this to 30 days, etc.
  • [Disable Logging on Success] : By disabling this, only the failed logs will be stored.

• In order to change the above settings of weekly simulations, you need to go to [Catalog] -> (ex) public road bus -> INTEGRATION tab -> select catalog -> ACTIONS -> Edit. You need the Github access token of the integrated repository.

• We can also exclude the stably succeeding scenarios from the weekly test list to save the storage space.

• After the discussion today, following settings have been applied.

  • [Log Expiration] : Changed from 60 to 45 days.
  • [Disable Logging on Success] : Not disabled for now. (We may sometimes need to play back the successful cases, too.)

• Following changes will be applied to the Autoware Evaluator settings by next week.

  • "pull over scenario suite" ("UC-F-16_ArvDpt_PullOver_zero#2") will be excluded from the weekly list as it may take some time for the speed limit/AEB issue to be addressed.
  • "test scenarios" suite to be renamed to "Low Speed Vehicles" for clarification.

• Did additional Q&A on Salvi's paper/poster "Online Identification of Operational Design Domains of Automated Driving System Features" which was presented in IV24 including the AD safety models defined in IEEE2846. His other papers are available form his personal page at Fraunhofer.

• Comments have been added to Leo Drive Scenario Catalog v0.7 - DRAFT.

  • Discussion topics include:
    • Autoware can detect animals as small as 15cm, but it requires additional learning/tuning.
    • If the detected objects fall in "unknown" category, it may lead to a sudden braking. In low speed ODDs, it may not be a big issue as the maximum speed is low (<20kph).
    • Parking module is included in current Autoware.universe.

• In addition to the discussion last week, @Igata-ctrl is investigating the cloud storage quota. It is associated to how long you would like to keep the simulation result in terms of rosbag.

• Answers to some of the questions from the meeting last week.

  • Comfortable braking threshold (deceleration and jerk) -> See the updated vehicle performance page.
  • Cloud quota for the weekly simulation -> The monthly total DURATION should not exceed 50 hours. You can check it in the Evaluator screen. -> @Igata-ctrl to double check that it is OK to simply add up the time in the DURATION column in the Evaluator Reports page.

• A TUM researcher Aniket Salvi, M. Sc. introduced his paper/poster "Online Identification of Operational Design Domains of Automated Driving System Features" which was presented in IV24.

Aniket Salvi, M. Sc.
Research Engineer
Engineering of Software-Defined Mobility
Fraunhofer-Institut für Kognitive Systeme IKS

• @Igata-ctrl joined the Reference Design WG meeting on May 29, 2024 and learned about the low speed ODDs of ROBEFF Technology's cargo vehicle and Kingwaytek's shuttle bus. -> Try to take some time to comment on this use case spreadsheet.

• Also discussed the followings:

  • Comfortable braking threshold (deceleration and jerk)
  • Cloud quota for the weekly simulation (TBC)
  • Expect @mitsudome-r to streamline the registration process to the Evaluator
  • Is there an easy way to download multiple scenarios at once? -> Unfortunately, no, as each scenario is associated to certain map and its version.

Action Item

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time