cpuset: no space left on device
#23405
Assignees
Labels
Comments
|
Hi @rodrigol-chan! Sorry to hear you're running into trouble. The error you're getting here is particularly weird:
We're writing to the |
|
It happened again just now, on a different machine. This Nomad client configuration now looks relevant: client {
gc_max_allocs = 300
gc_disk_usage_threshold = 80
}And currently have over 300 allocations: Nomad seems to be keeping a lot of tmpfss around even if the allocations aren't running anymore. I'm not sure if that's by design. |
|
For extra context: issue seems new with the 1.7.x upgrade. We've run this configuration in 1.6.x for about 8 months with no similar issues. |
|
Thanks for that extra info @rodrigol-chan. Even with that large number of allocs, I'd think you'd be ok until you get to 65535 inodes. I'll dig into that a little further to see if there's some more
The mounts are left in place until the allocation is GC'd on the client. We do that so that you can debug failed allocations. |
|
The issue still happens as of 1.8.3. Is there anything we can do to help troubleshoot this? |
|
Hi @rodrigol-chan, sorry, I haven't been able to circle back to this and I'm currently swamped trying to land some work for our 1.9 beta next week. I suspect this is platform-specific. I think you'll want to look into whether there's anything in the host configuration that could be limiting the size of those virtual FS directories. |
|
Hi @rodrigol-chan! Just wanted to check in so you don't think I've forgotten this issue. I re-read through your initial report to see if there were any clues I missed.
Even ignoring the errors you're seeing, that's got to be a bug all by itself. These should never overlap. Even though we can't write to the two files atomically, we always remove from the source first and then write to the destination. So in that tiny race you should see a missing CPU but not one counted twice. So I'll look into seeing if I can find any place where there's potentially another race condition here where that's not correctly handled.
You have other allocations on the same host that do use core constraints though? If not, we're writing an empty value to the cgroup. In which case, I found this Stack Exchange post which describes that scenario, but has no answer. 🤦 I managed to dig up a few old issues that suggest that if Also, I wanted to see if I could get this error outside of Nomad by echoing a bad input to the cgroup file, and wasn't able to get that same error.
I did get some interesting (but different) errors trying to write to the One more thing I'd like you to try is the following, to make sure we've counted the cgroups correctly when trying to figure out if its the inodes issue: |
That's correct.
Just happened again: # find /sys/fs/cgroup -depth -type d | wc -l
81
# find /sys/fs/cgroup/nomad.slice -depth -type d | wc -l
29
# head /sys/fs/cgroup/nomad.slice/cpuset.cpus /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus /sys/fs/cgroup/nomad.slice/share.slice/cpuset.cpus
==> /sys/fs/cgroup/nomad.slice/cpuset.cpus <==
0-31
==> /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus <==
0-3
==> /sys/fs/cgroup/nomad.slice/share.slice/cpuset.cpus <==
4-31Log output: It doesn't look like the CPUs overlapped this time. The number of dying descendants is curious, I wonder if it's related: # head /sys/fs/cgroup/nomad.slice/cgroup.stat /sys/fs/cgroup/nomad.slice/reserve.slice/cgroup.stat /sys/fs/cgroup/nomad.slice/share.slice/cgroup.stat
==> /sys/fs/cgroup/nomad.slice/cgroup.stat <==
nr_descendants 28
nr_dying_descendants 2356
==> /sys/fs/cgroup/nomad.slice/reserve.slice/cgroup.stat <==
nr_descendants 1
nr_dying_descendants 78
==> /sys/fs/cgroup/nomad.slice/share.slice/cgroup.stat <==
nr_descendants 25
nr_dying_descendants 2278 |
|
Can you confirm whether the |
I did look at that at failure time and from memory it was at
I can't find any I'll doublecheck |
|
Just happened again. (It has been happening strangely often lately.) Here are the values requested. $ head /sys/fs/cgroup/nomad.slice/reserve.slice/cgroup.max.descendants
max
$ head /sys/fs/cgroup/nomad.slice/reserve.slice/cgroup.stat
nr_descendants 1
nr_dying_descendants 11
$ head /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus
0-3
$ head /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.mems
$This doesn't look like it should be possible, though: It might just be an artifact of how the data is collected since I don't think it's possible to do an atomic snapshot of cgroups. All Nomad cgroups |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Nomad version
Operating system and Environment details
Running Ubuntu 22.04 on Google Cloud in an n2d-standard-32 instance.
Issue
Alerts fired due to failed allocations. Upon investigation, I noticed the following log line:
Also interesting to observe is that, unlike in our other 1.7.x clients, there's overlap between the CPUs for the reserve and share slices:
Reproduction steps
Not clear how to reproduce. This happened on a single instance. All allocations that failed are from periodic jobs, running with on the
execdriver with nocoreconstraints.Expected Result
Allocations spawn successfully.
Actual Result
Allocations failed to spawn.
Nomad Client logs (if appropriate)
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:56:52.170066+02:00","alloc_id":"7a39078a-0769-d3e9-38a0-c706ff516de8","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:58:45.224880+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:58:45.224934+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:58:45.226936+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:58:45.228745+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"processing","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:58:45.241815+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:59:45.699245+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:59:45.699331+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"processing","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:59:45.701453+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:59:45.703532+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:59:45.717471+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.588677+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.588716+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"maintenance-timer","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.594967+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.599704+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.617996+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.944803+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.944850+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.950686+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.954145+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.971800+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.280014+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.280062+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"maintenance-timer","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.284932+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.290230+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.305828+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.604525+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.604609+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.609268+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.615208+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.625727+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:01:00.615323+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:01:00.615380+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:01:00.617829+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:01:00.619727+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:01:00.633596+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:02:00.614056+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:02:00.614102+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:02:00.615991+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"} {"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:02:00.617758+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"} {"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:02:00.629413+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}Nomad client configuration
The text was updated successfully, but these errors were encountered: