How to achieve properly working Gazebo simulations for large dynamic UAV swarms?

[bconvens]

Hello,
I would like to discuss the best practices to run Gazebo simulations with large UAV swarms. More concretely, I would like to go significantly over 15 UAVs, more to something close to 50 UAVs. I consider no advanced sensing is being simulated, just GPS/RTK odometry and the drones can share positions similar as done in your mpc_tracker (not necessarily the full trajectory). Note that the goal is not to strictly simulate with a real time factor (RTF) around 1.0, it is allowed to go well below 1.0. So, all suggestions for how to get there are very welcome.

I can get around max 25 drones doing some collision avoidance tasks. But it does not work smoothly run after run. Once I was able to just spawn 30 drones, but could not manage to take off all of them, let alone make them navigate around each other. In the past I used Matlab/Simulink for these large swarm simulations and then this never leads to such issues. So why can't the Gazebo simulator not just run at a lower rate reliably?

This is what I tried up to now:

1. I invested in a decent cpu. I have a very good desktop machine running in performance mode.
   
   My graphics card is ASRock Radeon RX 5500 XT Challenger D 8G OC, more moderate. I also wonder if you think the gpu driver might be not a good fit as the Gazebo graphics also tend to produce low FPS?

2. I can simulate with a real-time factor of 1.0 for swarms of around 8 UAVs.

3. I tried to simulate with a RTF that is constant, but lower than 1.0 (i.e. 0.2) as to give my cpu some extra time. I started from your one_drone_gps example script and added ; export PX4_SIM_SPEED_FACTOR=0.2 at the end of the pre_window: and this works as expected. Also, faster than real-time I could go up to RTF = 3.5 on my machine. But when I do the same in the two_drone_gps script, I get the following error in the gazebo tmux tab:
   
   Are you aware of this issue and is there any solution as I believe this would make it possible to simulate with large number of UAVs? I continued without a solution to issue 3.

4. I then tried to change the grass_plane.world file from the default
   <max_step_size>0.004</max_step_size>
<real_time_factor>1</real_time_factor>
<real_time_update_rate>250</real_time_update_rate>
   to
   <max_step_size>0.004</max_step_size>
<real_time_factor>0.2</real_time_factor>
<real_time_update_rate>50</real_time_update_rate>, which satisfies max_step_size*real_time_update_rate=real_time_factor.
   Running the one_drone_gps or the two_drone_gps scripts with or without the change explained in 3 throws the same error as in 3. So this time also the one_uav_gps is not working, which was working under 3.

5. Then I tried running a simulation without the changes of 3. and with the default values of 4. and then manually changing the update rate in Gazebo from the default 250 to 50.
   
   For one_drone_gps this changes the RTF as expected. For two_drone_gps, but more clear to be tested for many_drone_gps, changing the update rate manually will automatically reset to the default of 250 each time a new drone spawns. This is not an issue for the 10 UAVs in many_drone_gps as they are spawned very quickly in like 30s and my real time factor is 1.0 during the whole simulation, but becomes a pain when going towards 30UAVs. where the last 10 UAVs take considerably more time to spawn and typically several uavs do not takeoff anymore.
   Moreover, I am not sure if this manual change of only the update rate in Gazebo to 50 has 100% the same effect as ; export PX4_SIM_SPEED_FACTOR=0.2?

6. When really going to larger swarms I observe the following. Spawning 24 drones goes relatively ok in 4 minutes. Once spawned I am able as described in 4. to lower the update rate and hence the RTF. The problem is that some of the uavs don't takeoff (approx. 1 out of 3). I sometimes succeed to manually make a uav takeoff, but very often the subtabs in the takeoff tab of the tmux session or not responding anymore, even with a lowered constant RTF that is very low. For those who took off, they don't seem to respond to goto commands...
   Another problem occurs when spawning over 24 uavs. When I try to spawn 30 uavs the following error is thrown almost each time I try around the same uav number (25-28). Once this error is thrown all uavs after that uav id are also not spawning anymore.
   
   
   Any idea how to overcome this?

7. Another issue is that when creating session.yaml files for 30 UAVs I bumped into this error:
   

So it seems that the scripts become too big. Therefor I used shell functions wherever possible and that allowed be to have a script for 30uavs that does not throw this error anymore. More is not possible for this and all above reasons.

Thanks a lot for your help!
Best, Bryan

[matemat13]

Hi, I see that you're trying to push Gazebo to its limits :) It seems that there are two main problems:

1. The px4 firmware doesn't like it when you set a different update rate and run it multiple times. I honestly don't know how to tackle this - it's probably related to px4 SITL or Gazebo. You can try opening an issue there.

2. There is some weird error with parameter loading when you spawn too many drones at the same time according to the screenshots. This may be because the parameter server is overloaded with requests. Maybe try spawning the drones and starting their control pipelines delayed? As in: start UAV1 + its pipeline, wait 1 second, start UAV2 + its pipeline, wait 1 second... etc.

Good luck, I hope you get this working, it sounds interesting!