GPUd is designed to ensure GPU efficiency and reliability by actively monitoring GPUs and effectively managing AI/ML workloads.
Unlike CPUs, GPU failures and issues are common and can significantly impact training and inference efficiency.
"78% of unexpected interruptions are attributed to confirmed hardware issues, such as GPU or host component failures, or suspected hardware-related issues like silent data corruption and unplanned individual host maintenance events. GPU issues are the largest category, accounting for 58.7% of all unexpected issues."
Reliability and Operational Challenges by Meta Llama team (2024)
GPUd addresses these challenges by automatically identifying, diagnosing, and repairing GPU-related issues, thereby minimizing downtime and maintaining high efficiency.
Read our announcement blog post here.
GPUd is built on years of experience operating large-scale GPU clusters at Meta, Alibaba Cloud, Uber, and Lepton AI. It is carefully designed to be self-contained and to integrate seamlessly with other systems such as Docker, containerd, Kubernetes, and Nvidia ecosystems.
- First-class GPU support: GPUd is GPU-centric, providing a unified view of critical GPU metrics and issues.
- Easy to run at scale: GPUd is a self-contained binary that runs on any machine with a low footprint.
- Production grade: GPUd is used in Lepton AI's production infrastructure.
Most importantly, GPUd operates with minimal CPU and memory overhead in a non-critical path and requires only read-only operations. See architecture for more details.
To install from the official release on Linux and amd64 (x86_64) machine:
curl -fsSL https://pkg.gpud.dev/install.sh | shNote that the install script doesn't support other architectures (arm64) and OSes (macos), yet.
Sign up at lepton.ai and get the workspace token from the "Settings" and "Tokens" page:
Copy the token and pass it to the gpud up --token flag:
sudo gpud up --token <LEPTON_AI_TOKEN>You can go to the dashboard to check the self-managed machine status.
For linux, run the following command to start the service:
sudo gpud upYou can also start with the standalone mode and later switch to the managed option:
# when the token is ready, run the following command
sudo gpud login --token <LEPTON_AI_TOKEN>To access the local web UI, open https://localhost:15132 in your browser.
If run with gpud up, you may disable this local web UI by setting FLAGS="--web-enable=false" to the /etc/default/gpud environment file and restart the service.
See gpud helm chart to deploy GPUd in your Kubernetes cluster.
To run on Mac (without systemd):
gpud runOr
nohup sudo /usr/sbin/gpud run &>> <your log file path> &sudo gpud down
sudo rm /usr/sbin/gpud
sudo rm /etc/systemd/system/gpud.service- Monitor critical GPU and GPU fabric metrics (power, temperature).
- Reports GPU and GPU fabric status (nvidia-smi parser, error checking).
- Detects critical GPU and GPU fabric errors (dmesg, hardware slowdown, NVML Xid event, DCGM).
- Monitor overall system metrics (CPU, memory, disk).
Check out components for a detailed list of components and their features.
For users looking to set up a platform to collect and process data from gpud, please refer to INTEGRATION.
GPUd collects a small anonymous usage signal by default to help the engineering team better understand usage frequencies. The data is strictly anonymized and does not contain any sensitive data. You can disable this behavior by setting GPUD_NO_USAGE_STATS=true. If GPUd is run with systemd (default option for the gpud up command), you can add the line GPUD_NO_USAGE_STATS=true to the /etc/default/gpud environment file and restart the service.
If you opt-in to log in to the Lepton AI platform, to assist you with more helpful GPU health states, GPUd periodically sends system runtime related information about the host to the platform. All these info are system workload and health info, and contain no user data. The data are sent via secure channels.
GPUd is still in active development, regularly releasing new versions for critical bug fixes and new features. We strongly recommend always being on the latest version of GPUd.
When GPUd is registered with the Lepton platform, the platform will automatically update GPUd to the latest version. To disable such auto-updates, if GPUd is run with systemd (default option for the gpud up command), you may add the flag FLAGS="--enable-auto-update=false" to the /etc/default/gpud environment file and restart the service.
