GPU simulator

[bconvens]

Hello MRS team,
I have used your Gazebo simulator for several works during my PhD. It was sufficient for the purpose of safely controlling many drones (without advanced perception) in obstacle-cluttered environments. The fact that this framework seamlessly transitions from simulation to hardware is just fantastic and the main reason I prefer to stick with it. I just noticed that with the new mrs_uav_system v1.5, users are now allowed to use other than the Gazebo simulator. However, I wonder if there are any plans to integrate GPU-based simulators, such as Nvidia Isaac Sim/Gym into your system. I believe this would be great for advanced perception, learning-based navigation, and (aerial) manipulation. Recent frameworks like the aerial_gym_simulator seem to be very promising in this regard, but I would like to hear your plans/thoughts on this direction of a GPU simulator for the mrs_uav_system.
Thanks!
Bryan

[klaxalk]

Hey @bconvens, good to hear from you!

We have ongoing work on two new simulation frameworks:

1. Unreal Engine 5 -based rendered with external dynamics simulation via MRS Multirotor Simulator,
2. Unity Engine simulator+renderer (all in one this time),
   We will be rolling them out as soon as they are usable.

Regarding the aerial_gym_simulator... I don't know if the MRS System (the control pipeline) would play well in automatic learning since the system's overhead is significant. If you want to know more about our work on automatic learning, contact @penickar.

[bconvens]

Hi @klaxalk, thanks for the info!
What are the main bottlenecks? Any suggestions on where there is significant potential for us to reduce the system's overhead?
Do you have an estimated timeline for those two points?

@penickar, do you have any suggestions on automatic learning on GPUs? We are currently playing with the idea to use/extend/port the aerial_gym_simulator, which is based on Nvidia Isaac Gym, to the mrs_uav_system as a new simulator. Similar to the mrs_uav_system, we have our own ROS C++ controllers and trackers and would like to reuse them without requiring custom implementations for the GPU simulation. Note that aerial_gym_simulator is a python implementation, its controllers are very standard and minimal without the tweaks for safe real-world implementation, and it only allows to simulate parallel worlds with a single UAV. We would like to keep your tracker/controller structure with inter-agent communication in a multi-agent setting per parallel world, use C++ code everywhere (don't spend time converting to python and to keep both in sync), and have the ability to benefit from other ROS packages you developed (e.g., mapping and planning) for the mrs_uav_system, and the UAV hardware we built.

[klaxalk]

@bconvens The bottleneck is mainly the ROS ecosystem, which is not meant to be run much faster than real-time. We have already made significant steps to make the system more lightweight when needed, which translates into the ability to simulate a larger number of drones at once rather than being able to simulate one drone faster. The fastest we can get is at around 5x RTF. But I suspect that for learning (of low-level control or end-to-end control), you don't need the MRS UAV System. The control inputs that the system provides (path, trajectory) might not be suitable for learning. You may need a renderer/simulator to accept your desired control input modality. That can be partially satisfied by the https://github.com/ctu-mrs/mrs_multirotor_simulator, which can (on its own) run at ludicrous speed and/or simulate hundreds of drones simultaneously. The MRS Sim accepts various inputs, anything from individual actuator inputs to the desired position.