-
Notifications
You must be signed in to change notification settings - Fork 196
vrx_2019 phase2_retrospective
In general the Dress Rehearsal was a very productive way for teams to try out their prototype solution and get familiar with the submission process for VRX.
The Scoring Summary illustrates the scoring that was generated by each team's solution. Each team's submission was run against 36 different simulation scenarios (6 tasks and 6 trials of each task).
In preparation for Phase 2 we released 3 trials for each of the 6 tasks - a total of 18 practice simulations for teams to test their solutions: Phase 2 Task Practice. In running the Phase 2 Dress Rehearsal we executed the submitted code against 6 trials for each task; the first 3 trials were equivalent to the practice trials and 3 new trials that were different than the original 3 practice trials.
After the Phase 2 deadline, all of the worlds and models used for the 36 trials (6 tasks, 6 trials each) have been made a part of the VRX repository (see PR#207 and PR#213). Following the directions in Phase 2 Task Practice teams can use these trials for testing their code in many different environmental conditions. For example,
WORLD=stationkeeping0.world
roslaunch vrx_gazebo vrx.launch verbose:=true \
paused:=false \
wamv_locked:=true \
world:=${HOME}/vrx_ws/src/vrx/vrx_gazebo/worlds/2019_phase2/${WORLD}
In one of the practice worlds for the stationkeeping task the goal pose was far enough away from the initial WAM-V pose that the vessel couldn't reasonable transit to the goal during the 'Ready' state of the simulation. This could result in a large error for that trial of the task which could skew the results. For the finals the worlds will be setup so that the WAM-V can be expected to transit from the initial pose to near the goal pose in the ready-state-duration (typically 10 seconds). See this Forum post for more details.
Going through the submissions for Phase 2, we noticed that the teams include the P3D plugin in their sensor_config.yaml file. The P3D plugin provides a ROS published ground truth - true, noiseless pose information from Gazebo. During development it is often handy to know the true state of the system, but the system should estimate the state of the system from sensors - not use the ground truth directly in the solution.
This "sensor" is not included in the Technical Guide, but the Sensor Configuration Wiki points to the numeric.yaml file which could be interpreted to suggest that a sensor configuration that includes one P3D sensor is compliant.
For Phase 3 including the P3D sensor in the sensor_config.yaml file will result in a non-compliant configuration.
The Task Naming in the VRX Competition and Task Descriptions document describe how the six individuals tasks (stationkeeping, wayfinding, etc.) are announced via the ROS interface. In that document the "Dock" task corresponds to a Task Msg name of "scan", and the "Scan and Dock" task corresponds to a Task Msg with the name "scan_and_dock". There were two potential issues with this during Phase 2.
First, there was a bug in the practice words released prior to Phase 2: Phase 2 Task Practice. This bug caused Task Messages both the "Dock" and "Scan and Dock" tasks to have the name "scan_dock". This was corrected with during evaluation of the Phase 2 submissions so that the task names were "scan" and "scan_and_dock" as specified in the competition documents. These trials, with the correct names, have been released - see PR#207.
Second, the naming may have caused some confusion, since we probably intended to use use the Task Msg name "dock" to indicate this "Dock" task. However, we feel it is important to stick to the documentation as written which specifies the Task Msg for the "Dock" task as "scan".
For Phase 3 we will not be changing the interface, so the Dock Task (Task 5) will continue to be announced in the Task Message using the name "scan".
In the dress rehearsal the GT team noticed that we left the we failed to turn off the Scan-the-Code scoring system for the Dock Task (Task 5). This made it possible for teams that noticed the flaw to score points in the task by simply reporting the default color sequence - see Issue#186. We fixed this in PR#120.
While running the Phase 2 submissions, some of the team Docker images were not properly configured to run the vrx-docker scripts. In particular, the main problem was that the team image was not executing its expected entry point. Some teams were relying on a .bashrc that is parsed when running docker with the -it flag. However, the VRX run_trial.bash script is not using that option. See here how we use docker run with your team image. Check out this tutorial for tips and instructions on how to create your Docker images and this tutorial about how to verify your submission.
We contacted most of the teams while running Phase 2, but keep in mind that is your responsibility to guarantee that your image is properly configured for Phase 3.