Feature Requests for Evaluator (DevOps product for Autoware from TIER IV)

[Igata-ctrl]

@mitsudome-r, @xmfcx, @armaganarsln

In ODD working group, we started discussing the feature request for the Evaluator, a CI/CD pipeline solution for Autoware from TIER IV.

As you might know, the Evaluator is a product of TIER IV and AWF is one of the users of it, while Scenario Simulator v2 and AWSIM are OSS. So I would like to assemble a prioritized wish list to discuss with TIER IV CI/CD team.

Anyone who is familiar with the Evaluator and would like provide comments, that will be truly appreciated. If you do not have the access to the Evaluator, please ask me so that I can set it up for you. You will need to sign up for your free TIER IV account first if you don' have one. then I can add you to the AFW organization and the ODD WG in the Evaluator system.

The initial discussion is mentioned in the ”2024-10-07 ODD working group weekly meeting (209)” minutes page.

I had a preliminary discussion with a TIER IV CI/CD team and got a couple piece of information as follows.

• One of the requests was the use of the more sophisticated scenario editor like AVL SCENIUS™️ Scenario Designer.
  • It turned out that TIER IV Evaluator team is working on the import function for OpenSCENARIO XML (1.x) (targeting the end of 2024) which means that you will be able to import the scenarios created by AVL SCENUS or from Safety Pool scenario database. (As for Safety Pool, other issues including the map format need to be discussed further.)
• The other request was that if we have more convenient way toe edit Lanelet2 map from with in the scenario editor.
  • This one turned that the process is somewhat simplified if you follow the steps below.
    • From the Vector Map Builder, use the Import Maps from FMS menu (the Evaluator is part of FMS, in this case.)
    • Edit the map in the Vector Map Builder
    • Export your new mp by using the Export Maps to FMS menu. (You can add version comments when you export the map.)