Skip to content

Latest commit

 

History

History
408 lines (329 loc) · 16.7 KB

README.md

File metadata and controls

408 lines (329 loc) · 16.7 KB
Raw

turludock - Robot Operating System (ROS) Docker Containers with X11 and Wayland support for Linux

N|Solid

turludock aims to provide different versions of ROS as Docker containers, but with GUI support for both X11 and Wayland! This means your local OS is no more bound to the version of ROS you are using! You can use any version of ROS with any Linux distribution, thanks to the amazing power of Docker!

TL;DR

Install tool - Python >= 3.9

pip install turludock

Build image

# Build ROS image from presets (turludock which presets)
turludock build -e noetic_mesa
# Build using a custom configuration
turludock build -c custom.yaml --tag custom_tag

# Check supported versions
turludock which ros
turludock which cuda ROS_CODENAME

All commands have a --help support.

Example YAML config

Run container

See "Running the image (as current user)" on how to run a container.

Getting Started

The idea is to have HW accelerated GUIs with Docker. Generally it has proven that this is a challenging task. Graphics card companies like NVIDIA, already provide solutions for their platform. Running X11 applications within a Docker container is proven for the last years. However, Wayland support remains still a bit challenging, but at least for our ROS containers we can assume it is supported 🍰.

Install turludock

pip install turludock

Supported tool functionality

This tool has two main functionalities:

  1. Build ROS images which result in ready-to-use containers.
  2. Generate Dockerfile and required assets for manually building the Docker images with Docker build.

Some more details with:

turludock --help

Supported GPU drivers

Currently this project supports all HW accelerated containers, assuming your GPU is supported by either mesa or NVIDIA drivers.

For older Ubuntu versions that need support for state-of-the-art GPUs like the Radeon RX 7 series, we rely on the ppa:kisak/kisak-mesa repository.

NVIDIA GPU

Both the NVIDIA drivers and the nvidia-container-toolkit are required to be installed and working.

You test this by making sure nvidia-smi works:

Docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Known Wayland limitations

ROS related GUI programs seem to working fine. Other programs like meld or vscode also seem to be working fine. There might be cases where some GUIs do not work as expected. Feel free to open a ticket so we can look into it.

For example, we know RViz is not ready for Wayland; hence we will need to use the xcb (X11) plugin instead of the one for Wayland and therefore we will use QT_QPA_PLATFORM=xcb for all QT applications. More info in section "RViz Wayland known issues".

Supported ROS Images

The idea is to try and support all currently ROS versions that are not End-of-Life (EOL). Below you can check all ROS1 and ROS2 distributions and their EOL status:

Even if ROS 1 Noetic becomes EOL we will try to support it for longer time since many people might still be relying on it 😎

To check which versions are actually supported use:

turludock which ros

This will output ROS versions with their codename.

Supported CUDA and cuDNN versions

To see the supported versions of CUDA/cuDDN for a specific ROS version use:

turludock which cuda ROS_CODENAME

ROS_CODENAME could be noetic for example.

Build desired Docker Image

Hint: For more details and supported arguments

turludock build --help
# Or
turludock generate --help

Tool is based on YAML configurations

This tool uses a specific .yaml configuration to generate Dockerfiles or build Docker images.

You can find a typical configuration in the examples folder.

Build or generate from presets

This tool already provides preconfigured .yaml files that can be used directly to generate Dockerfiles or build Docker images. This is what we call a list of presets: you might like them and use them; or hate them and create your own 😉.

For a list of available presets run:

turludock which presets

This will show the underlying ROS version, the GPU driver assumed, the CUDA/cuDNN version (if applicable) and some preinstalled packages that are part of the preset.

To build a Docker image from the presets use:

turludock build -e noetic_mesa

Or if you want to generate the Dockerfile and its required assets for manual build use:

turludock generate -e noetic_mesa FOLDER_PATH

The FOLDER_PATH now contains all necessary files to run a custom Docker build command. So you can just invoke Docker build FOLDER_PATH for example.

Build or generate from custom YAML configuration

OK, so you don't like the existing presets and you would like to build a Docker image using your own custom configuration...

No problemo, use:

turludock build -c custom.yaml --tag CUSTOM_TAG

How to create a custom yaml configuration? Check the example.

Or if you want to generate the Dockerfile and its required assets for manual build use:

turludock generate -c custom.yaml FOLDER_PATH

The FOLDER_PATH now contains all necessary files to run a custom Docker build command.

Running the image (as current user)

Mesa

🍍 Important: Make sure your YAML configuration uses: gpu_driver: mesa

X11

To run the ROS Docker container with X11 support use:

Docker run --rm -it --privileged --net=host --ipc=host \
--device=/dev/dri:/dev/dri \
-v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY \
-v $HOME/.Xauthority:/home/$(id -un)/.Xauthority -e XAUTHORITY=/home/$(id -un)/.Xauthority \
-e DOCKER_USER_NAME=$(id -un) \
-e DOCKER_USER_ID=$(id -u) \
-e DOCKER_USER_GROUP_NAME=$(id -gn) \
-e DOCKER_USER_GROUP_ID=$(id -g) \
-e ROS_IP=127.0.0.1 \
turlucode/ros-noetic:mesa-cmake-tmux-llvm-meld

Important Remarks:

  • The DOCKER_USER_* variables are used to run the container as the current user.
  • Please note that you need to pass the Xauthority to the correct user's home directory.
  • You may need to run xhost si:localuser:$USER or worst case xhost local:root if get errors like Error: cannot open display.
  • See also section "Other options" for other options.

Wayland

NOTE: Wayland support is still a bit experimental! See section "Known Wayland limitations".

ROS GUI on Wayland is still problematic and that is why we are going to use xwayland. Make sure you have installed xhost. Make also sure the user has the rights to draw to the display; more info in section "X11: Error: cannot open display".

To run the ROS Docker container with Wayland support use:

Docker run --rm -it --security-opt seccomp=unconfined \
-v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY \
-e XDG_RUNTIME_DIR=/run/user/$(id -u) \
-e WAYLAND_DISPLAY=$WAYLAND_DISPLAY \
-v $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:/run/user/$(id -u)/$WAYLAND_DISPLAY \
--device /dev/dri \
--group-add video \
-e QT_QPA_PLATFORM=xcb \
-e XDG_SESSION_TYPE=wayland \
-e GDK_BACKEND=wayland \
-e CLUTTER_BACKEND=wayland \
-e SDL_VIDEODRIVER=wayland \
-e DOCKER_USER_NAME=$(id -un) \
-e DOCKER_USER_ID=$(id -u) \
-e DOCKER_USER_GROUP_NAME=$(id -gn) \
-e DOCKER_USER_GROUP_ID=$(id -g) \
-e ROS_IP=127.0.0.1 \
turlucode/ros-noetic:mesa-cmake-tmux-llvm-meld dbus-launch terminator

Or for Ubuntu 24.04 and up, dbus-launch terminator is not needed, just use:

Docker run [...] turlucode/ros-jazzy:mesa-cmake-tmux-llvm-meld

Important Remarks:

  • The DOCKER_USER_* variables are used to run the container as the current user.
  • We need to start terminator with dbus-launch for Ubuntu versions less than 24.04, i.e. for versions < jazzy.
  • The QT_QPA_PLATFORM=xcb is for now intentionally as discussed. For this reason also the qtwayland5 package is not installed in our Docker images.
  • See also section "Other options" for other options.

NVIDIA GPU

🍍 Important: Make sure your YAML configuration uses: gpu_driver: nvidia

For machines that are using NVIDIA graphics cards we need to have the nvidia-container-toolkit installed.

X11

To run the ROS Docker container with X11 support use:

Docker run --rm -it --runtime=nvidia --gpus all --privileged --net=host --ipc=host \
-v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY \
-v $HOME/.Xauthority:/home/$(id -un)/.Xauthority -e XAUTHORITY=/home/$(id -un)/.Xauthority \
-e DOCKER_USER_NAME=$(id -un) \
-e DOCKER_USER_ID=$(id -u) \
-e DOCKER_USER_GROUP_NAME=$(id -gn) \
-e DOCKER_USER_GROUP_ID=$(id -g) \
-e ROS_IP=127.0.0.1 \
turlucode/ros-noetic:nvidia-cmake-tmux-llvm-meld

Important Remarks:

  • The DOCKER_USER_* variables are used to run the container as the current user.
  • Please note that you need to pass the Xauthority to the correct user's home directory.
  • You may need to run xhost si:localuser:$USER or worst case xhost local:root if get errors like Error: cannot open display
  • Adapt --gpus all to your needs
  • See also section "Other options" for other options.

Wayland

NOTE: Wayland support is still a bit experimental! See section "Known Wayland limitations".

ROS GUI on Wayland is still problematic and that is why we are going to use xwayland. Make sure you have installed xhost. Make also sure the user has the rights to draw to the display; more info in section "X11: Error: cannot open display".

To run the ROS Docker container with X11 support use:

Docker run --rm -it --runtime=nvidia --gpus all --privileged --net=host --ipc=host \
-v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY \
-e XDG_RUNTIME_DIR=/run/user/$(id -u) \
-e WAYLAND_DISPLAY=$WAYLAND_DISPLAY \
-v $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:/run/user/$(id -u)/$WAYLAND_DISPLAY \
-e QT_QPA_PLATFORM=xcb \
-e XDG_SESSION_TYPE=wayland \
-e GDK_BACKEND=wayland \
-e CLUTTER_BACKEND=wayland \
-e SDL_VIDEODRIVER=wayland \
-e DOCKER_USER_NAME=$(id -un) \
-e DOCKER_USER_ID=$(id -u) \
-e DOCKER_USER_GROUP_NAME=$(id -gn) \
-e DOCKER_USER_GROUP_ID=$(id -g) \
-e ROS_IP=127.0.0.1 \
turlucode/ros-noetic:nvidia-cmake-tmux-llvm-meld dbus-launch terminator

Or for Ubuntu 24.04 and up, dbus-launch terminator is not needed, just use:

Docker run [...] turlucode/ros-jazzy:nvidia-cmake-tmux-llvm-meld
  • The DOCKER_USER_* variables are used to run the container as the current user.
  • We need to now start terminator with dbus-launch
  • The QT_QPA_PLATFORM=xcb is for now intentionally as discussed. For this reason also the qtwayland5 package is not installed in our Docker images.
  • Please note that you need to pass the Xauthority to the correct user's home directory.
  • You may need to run xhost si:localuser:$USER or worst case xhost local:root if get errors like Error: cannot open display
  • Adapt --gpus all to your needs
  • See also section "Other options" for other options.

Other options

Mount your ssh-keys

For both root and custom user use:

-v $HOME/.ssh:/root/.ssh

For the custom-user the container will make sure to copy them to the right location.

Mount your local catkin_ws

To mount your local catkin_ws you can just use the following Docker feature:

# for root user
-v $HOME/<some_path>/catkin_ws:/root/catkin_ws
# for local user
-v $HOME/<some_path>/catkin_ws:/home/$(id -un)/catkin_ws

Passing a camera device

If you have a virtual device node like /dev/video0, e.g. a compatible usb camera, you pass this to the Docker container like this:

--device /dev/video0

References

Wayland in Docker references

RViz Wayland known issues

Troubleshooting

Container eating up too much memory

Is even a simple command inside Docker eating up crazy a lot of memory? This happens especially in arch-linux. Chances are you have a "miss-configured" LimitNOFILE. See here, here, here for the reported issue.

# This is what is configured
cat /usr/lib/systemd/system/containerd.service | grep LimitNOFILE
# This is what is actually being used
systemctl show containerd | grep LimitNOFILE

If containerd.service uses infinity better bound it with systemd with a new .conf file:

# /etc/systemd/system/containerd.service.d/30-override.conf
[Service]
LimitNOFILE=1048576

If the folder Docker.service.d doesn't exist, create it. Now reload service with:

sudo systemctl restart containerd && sudo systemctl daemon-reload

And you are good to go! The container feels also "snappier" now.

X11: Error: cannot open display

Assuming xhost is installed and running on your host, you may need to run xhost si:localuser:$USER or worst case xhost local:root.

Docker security risks

Are you worried about the security risks when exposing your X-server to the container? Normally you should be! 😃

There are some nice articles that explain what is going on and what might be some extra steps you can do, in order to be less exposed.

  1. Docker Security Risks: GUIs + Xorg
  2. Running a Graphical Application from a Docker Container - Simply and Securely
  3. Feel free to google/gpt the matter yourself. 😉

Most importantly you can also review the template files in order to be absolutely sure you like what is being installed in the images when using the turludock tool.

Vscode crashes

Try adding --shm-size=8G to your Docker command.

Issues and Contributing

Base images

NVIDIA

The images on this repository are based on the following work:

For developers - a quick how to

This project uses poetry for packaging and dependency management. After installing poetry you can basically create a python virtual environment for one of the supported python versions, >=3.9, and inside that new environment just install the dependencies with

poetry install

After that, you are good to go!

Hint: You can install pynev if you want to check multiple python versions.

Coding style enforcement

Check and enforce the coding style with static analysis:

poetry run isort turludock && poetry run black turludock && poetry run pflake8 turludock