Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate that HCloud servers shut down usually #1223

Open
janiskemper opened this issue Mar 22, 2024 · 7 comments
Open

Validate that HCloud servers shut down usually #1223

janiskemper opened this issue Mar 22, 2024 · 7 comments
Assignees
@kranurag7

Comments

@janiskemper
Copy link
Contributor

Description

We have code that makes sure that a server first shuts down and then gets deleted. See the references. However, we should validate that servers actually shut down properly, because we have many examples where they did not and the 2 minutes timeout was reached, which caused the server to be deleted without being shut down successfully.

Let's answer these question:
Do the HCloud servers USUALLY shut down within 2 minutes? If so, what is the average time?

If we answer this, we can improve the 2 minutes timeout and in this way improve user experience.
@apricote
Copy link
Contributor

Some background from the API docs on endpoint Shutdown a Server. This section was updated in the last months to include some more detail on what is happening:

Shuts down a Server gracefully by sending an ACPI shutdown request. The Server operating system must support ACPI and react to the request, otherwise the Server will not shut down. Please note that the action status in this case only reflects whether the action was sent to the server. It does not mean that the server actually shut down successfully. If you need to ensure that the server is off, use the poweroff action.

So it would be interesting to know why the servers do not process the shutdown within time. Are there perhaps some pods still running, volumes being unmounted, ...?

I have not seen this issue in my cluster yet.

@kranurag7
Copy link
Contributor

kranurag7 commented Apr 1, 2024 •
edited
Loading

I think the server is not shutting down under 2 minutes. For quick iteration I tried it using hcloud CLI.

Details 

$ hcloud ssh-key create --name test --public-key-from-file ~/.ssh/hetzner.pub
SSH key 20218939 created
$ hcloud server create --name=testing-demo --image=ubuntu-22.04 --type=cx21 --ssh-key=$SSH_KEY_NAME
4.7s [===================================] 100.00%
Waiting for server 45291975 to have started
 ... done
Server 45291975 created
IPv4: 95.217.214.217
IPv6: 2a01:4f9:c012:cebe::1
IPv6 Network: 2a01:4f9:c012:cebe::/64
Root password: tHPmUcuF7Ubw9iRUuxU3
$ hcloud server shutdown testing-demo
700ms [==================================] 100.00%
Sent shutdown signal to server 45291975
$ ping 95.217.214.217
PING 95.217.214.217 (95.217.214.217) 56(84) bytes of data.
64 bytes from 95.217.214.217: icmp_seq=1 ttl=44 time=245 ms
64 bytes from 95.217.214.217: icmp_seq=2 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=3 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=4 ttl=44 time=211 ms
64 bytes from 95.217.214.217: icmp_seq=5 ttl=44 time=233 ms
64 bytes from 95.217.214.217: icmp_seq=6 ttl=44 time=256 ms
64 bytes from 95.217.214.217: icmp_seq=7 ttl=44 time=278 ms
64 bytes from 95.217.214.217: icmp_seq=8 ttl=44 time=301 ms
64 bytes from 95.217.214.217: icmp_seq=9 ttl=44 time=222 ms
64 bytes from 95.217.214.217: icmp_seq=10 ttl=44 time=244 ms
64 bytes from 95.217.214.217: icmp_seq=11 ttl=44 time=268 ms
64 bytes from 95.217.214.217: icmp_seq=12 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=13 ttl=44 time=209 ms
64 bytes from 95.217.214.217: icmp_seq=14 ttl=44 time=232 ms
64 bytes from 95.217.214.217: icmp_seq=15 ttl=44 time=255 ms
64 bytes from 95.217.214.217: icmp_seq=16 ttl=44 time=278 ms
64 bytes from 95.217.214.217: icmp_seq=17 ttl=44 time=197 ms
64 bytes from 95.217.214.217: icmp_seq=18 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=19 ttl=44 time=242 ms
64 bytes from 95.217.214.217: icmp_seq=20 ttl=44 time=264 ms
64 bytes from 95.217.214.217: icmp_seq=21 ttl=44 time=287 ms
64 bytes from 95.217.214.217: icmp_seq=22 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=23 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=24 ttl=44 time=252 ms
64 bytes from 95.217.214.217: icmp_seq=25 ttl=44 time=275 ms
64 bytes from 95.217.214.217: icmp_seq=26 ttl=44 time=297 ms
64 bytes from 95.217.214.217: icmp_seq=27 ttl=44 time=220 ms
64 bytes from 95.217.214.217: icmp_seq=28 ttl=44 time=240 ms
64 bytes from 95.217.214.217: icmp_seq=29 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=30 ttl=44 time=286 ms
64 bytes from 95.217.214.217: icmp_seq=31 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=32 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=33 ttl=44 time=250 ms
64 bytes from 95.217.214.217: icmp_seq=34 ttl=44 time=273 ms
64 bytes from 95.217.214.217: icmp_seq=35 ttl=44 time=296 ms
64 bytes from 95.217.214.217: icmp_seq=36 ttl=44 time=217 ms
64 bytes from 95.217.214.217: icmp_seq=37 ttl=44 time=239 ms
64 bytes from 95.217.214.217: icmp_seq=38 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=39 ttl=44 time=284 ms
64 bytes from 95.217.214.217: icmp_seq=40 ttl=44 time=204 ms
64 bytes from 95.217.214.217: icmp_seq=41 ttl=44 time=227 ms
64 bytes from 95.217.214.217: icmp_seq=42 ttl=44 time=250 ms
64 bytes from 95.217.214.217: icmp_seq=43 ttl=44 time=273 ms
64 bytes from 95.217.214.217: icmp_seq=44 ttl=44 time=296 ms
64 bytes from 95.217.214.217: icmp_seq=45 ttl=44 time=217 ms
64 bytes from 95.217.214.217: icmp_seq=46 ttl=44 time=239 ms
64 bytes from 95.217.214.217: icmp_seq=47 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=48 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=49 ttl=44 time=205 ms
64 bytes from 95.217.214.217: icmp_seq=50 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=51 ttl=44 time=251 ms
64 bytes from 95.217.214.217: icmp_seq=52 ttl=44 time=276 ms
64 bytes from 95.217.214.217: icmp_seq=53 ttl=44 time=297 ms
64 bytes from 95.217.214.217: icmp_seq=54 ttl=44 time=218 ms
64 bytes from 95.217.214.217: icmp_seq=55 ttl=44 time=240 ms
64 bytes from 95.217.214.217: icmp_seq=56 ttl=44 time=263 ms
64 bytes from 95.217.214.217: icmp_seq=57 ttl=44 time=286 ms
64 bytes from 95.217.214.217: icmp_seq=58 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=59 ttl=44 time=229 ms
64 bytes from 95.217.214.217: icmp_seq=60 ttl=44 time=252 ms
64 bytes from 95.217.214.217: icmp_seq=61 ttl=44 time=275 ms
64 bytes from 95.217.214.217: icmp_seq=62 ttl=44 time=297 ms
64 bytes from 95.217.214.217: icmp_seq=63 ttl=44 time=217 ms
64 bytes from 95.217.214.217: icmp_seq=64 ttl=44 time=238 ms
64 bytes from 95.217.214.217: icmp_seq=65 ttl=44 time=261 ms
64 bytes from 95.217.214.217: icmp_seq=66 ttl=44 time=284 ms
64 bytes from 95.217.214.217: icmp_seq=67 ttl=44 time=204 ms
64 bytes from 95.217.214.217: icmp_seq=68 ttl=44 time=227 ms
64 bytes from 95.217.214.217: icmp_seq=69 ttl=44 time=250 ms
64 bytes from 95.217.214.217: icmp_seq=70 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=71 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=72 ttl=44 time=215 ms
64 bytes from 95.217.214.217: icmp_seq=73 ttl=44 time=239 ms
64 bytes from 95.217.214.217: icmp_seq=74 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=75 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=76 ttl=44 time=205 ms
64 bytes from 95.217.214.217: icmp_seq=77 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=78 ttl=44 time=251 ms
64 bytes from 95.217.214.217: icmp_seq=79 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=80 ttl=44 time=196 ms
64 bytes from 95.217.214.217: icmp_seq=81 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=82 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=83 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=84 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=85 ttl=44 time=208 ms
64 bytes from 95.217.214.217: icmp_seq=86 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=87 ttl=44 time=254 ms
64 bytes from 95.217.214.217: icmp_seq=88 ttl=44 time=276 ms
64 bytes from 95.217.214.217: icmp_seq=89 ttl=44 time=196 ms
64 bytes from 95.217.214.217: icmp_seq=90 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=91 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=92 ttl=44 time=265 ms
64 bytes from 95.217.214.217: icmp_seq=93 ttl=44 time=289 ms
64 bytes from 95.217.214.217: icmp_seq=94 ttl=44 time=209 ms
64 bytes from 95.217.214.217: icmp_seq=95 ttl=44 time=232 ms
64 bytes from 95.217.214.217: icmp_seq=96 ttl=44 time=254 ms
64 bytes from 95.217.214.217: icmp_seq=97 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=98 ttl=44 time=199 ms
64 bytes from 95.217.214.217: icmp_seq=99 ttl=44 time=222 ms
64 bytes from 95.217.214.217: icmp_seq=100 ttl=44 time=244 ms
64 bytes from 95.217.214.217: icmp_seq=101 ttl=44 time=267 ms
64 bytes from 95.217.214.217: icmp_seq=102 ttl=44 time=290 ms
64 bytes from 95.217.214.217: icmp_seq=103 ttl=44 time=212 ms
64 bytes from 95.217.214.217: icmp_seq=104 ttl=44 time=235 ms
64 bytes from 95.217.214.217: icmp_seq=105 ttl=44 time=255 ms
64 bytes from 95.217.214.217: icmp_seq=106 ttl=44 time=279 ms
64 bytes from 95.217.214.217: icmp_seq=107 ttl=44 time=198 ms
64 bytes from 95.217.214.217: icmp_seq=108 ttl=44 time=221 ms
64 bytes from 95.217.214.217: icmp_seq=109 ttl=44 time=245 ms
64 bytes from 95.217.214.217: icmp_seq=110 ttl=44 time=267 ms
64 bytes from 95.217.214.217: icmp_seq=111 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=112 ttl=44 time=211 ms
64 bytes from 95.217.214.217: icmp_seq=113 ttl=44 time=234 ms
64 bytes from 95.217.214.217: icmp_seq=114 ttl=44 time=257 ms
64 bytes from 95.217.214.217: icmp_seq=115 ttl=44 time=282 ms
64 bytes from 95.217.214.217: icmp_seq=116 ttl=44 time=200 ms
64 bytes from 95.217.214.217: icmp_seq=117 ttl=44 time=223 ms
64 bytes from 95.217.214.217: icmp_seq=118 ttl=44 time=245 ms
64 bytes from 95.217.214.217: icmp_seq=119 ttl=44 time=268 ms
64 bytes from 95.217.214.217: icmp_seq=120 ttl=44 time=290 ms
64 bytes from 95.217.214.217: icmp_seq=121 ttl=44 time=209 ms
64 bytes from 95.217.214.217: icmp_seq=122 ttl=44 time=232 ms
64 bytes from 95.217.214.217: icmp_seq=123 ttl=44 time=255 ms
64 bytes from 95.217.214.217: icmp_seq=124 ttl=44 time=278 ms
64 bytes from 95.217.214.217: icmp_seq=125 ttl=44 time=198 ms
64 bytes from 95.217.214.217: icmp_seq=126 ttl=44 time=223 ms
64 bytes from 95.217.214.217: icmp_seq=127 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=128 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=129 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=130 ttl=44 time=212 ms
64 bytes from 95.217.214.217: icmp_seq=131 ttl=44 time=236 ms
64 bytes from 95.217.214.217: icmp_seq=132 ttl=44 time=258 ms
64 bytes from 95.217.214.217: icmp_seq=133 ttl=44 time=281 ms
64 bytes from 95.217.214.217: icmp_seq=134 ttl=44 time=203 ms
64 bytes from 95.217.214.217: icmp_seq=135 ttl=44 time=225 ms
64 bytes from 95.217.214.217: icmp_seq=136 ttl=44 time=248 ms
64 bytes from 95.217.214.217: icmp_seq=137 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=138 ttl=44 time=292 ms
64 bytes from 95.217.214.217: icmp_seq=139 ttl=44 time=212 ms
64 bytes from 95.217.214.217: icmp_seq=140 ttl=44 time=235 ms
64 bytes from 95.217.214.217: icmp_seq=141 ttl=44 time=258 ms
64 bytes from 95.217.214.217: icmp_seq=142 ttl=44 time=281 ms
64 bytes from 95.217.214.217: icmp_seq=143 ttl=44 time=201 ms
64 bytes from 95.217.214.217: icmp_seq=144 ttl=44 time=225 ms
64 bytes from 95.217.214.217: icmp_seq=145 ttl=44 time=217 ms
64 bytes from 95.217.214.217: icmp_seq=146 ttl=44 time=216 ms
64 bytes from 95.217.214.217: icmp_seq=147 ttl=44 time=216 ms
64 bytes from 95.217.214.217: icmp_seq=148 ttl=44 time=212 ms
64 bytes from 95.217.214.217: icmp_seq=149 ttl=44 time=236 ms
64 bytes from 95.217.214.217: icmp_seq=150 ttl=44 time=261 ms
64 bytes from 95.217.214.217: icmp_seq=151 ttl=44 time=282 ms
64 bytes from 95.217.214.217: icmp_seq=152 ttl=44 time=202 ms
64 bytes from 95.217.214.217: icmp_seq=153 ttl=44 time=226 ms
64 bytes from 95.217.214.217: icmp_seq=154 ttl=44 time=248 ms
64 bytes from 95.217.214.217: icmp_seq=155 ttl=44 time=270 ms
64 bytes from 95.217.214.217: icmp_seq=156 ttl=44 time=292 ms
64 bytes from 95.217.214.217: icmp_seq=157 ttl=44 time=212 ms
64 bytes from 95.217.214.217: icmp_seq=158 ttl=44 time=235 ms
64 bytes from 95.217.214.217: icmp_seq=159 ttl=44 time=257 ms
64 bytes from 95.217.214.217: icmp_seq=160 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=161 ttl=44 time=202 ms
64 bytes from 95.217.214.217: icmp_seq=162 ttl=44 time=224 ms
64 bytes from 95.217.214.217: icmp_seq=163 ttl=44 time=247 ms
64 bytes from 95.217.214.217: icmp_seq=164 ttl=44 time=269 ms
64 bytes from 95.217.214.217: icmp_seq=165 ttl=44 time=293 ms
64 bytes from 95.217.214.217: icmp_seq=166 ttl=44 time=213 ms
64 bytes from 95.217.214.217: icmp_seq=167 ttl=44 time=236 ms
64 bytes from 95.217.214.217: icmp_seq=168 ttl=44 time=258 ms
64 bytes from 95.217.214.217: icmp_seq=169 ttl=44 time=281 ms
64 bytes from 95.217.214.217: icmp_seq=170 ttl=44 time=202 ms
64 bytes from 95.217.214.217: icmp_seq=171 ttl=44 time=225 ms
64 bytes from 95.217.214.217: icmp_seq=172 ttl=44 time=247 ms
64 bytes from 95.217.214.217: icmp_seq=173 ttl=44 time=269 ms
64 bytes from 95.217.214.217: icmp_seq=174 ttl=44 time=291 ms
64 bytes from 95.217.214.217: icmp_seq=175 ttl=44 time=213 ms
64 bytes from 95.217.214.217: icmp_seq=176 ttl=44 time=235 ms
64 bytes from 95.217.214.217: icmp_seq=177 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=178 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=179 ttl=44 time=303 ms
64 bytes from 95.217.214.217: icmp_seq=180 ttl=44 time=224 ms
64 bytes from 95.217.214.217: icmp_seq=181 ttl=44 time=248 ms
64 bytes from 95.217.214.217: icmp_seq=182 ttl=44 time=270 ms
64 bytes from 95.217.214.217: icmp_seq=183 ttl=44 time=293 ms
64 bytes from 95.217.214.217: icmp_seq=184 ttl=44 time=214 ms
64 bytes from 95.217.214.217: icmp_seq=185 ttl=44 time=237 ms
64 bytes from 95.217.214.217: icmp_seq=186 ttl=44 time=260 ms
64 bytes from 95.217.214.217: icmp_seq=187 ttl=44 time=199 ms
64 bytes from 95.217.214.217: icmp_seq=188 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=189 ttl=44 time=197 ms
64 bytes from 95.217.214.217: icmp_seq=190 ttl=44 time=249 ms
64 bytes from 95.217.214.217: icmp_seq=191 ttl=44 time=272 ms
64 bytes from 95.217.214.217: icmp_seq=192 ttl=44 time=297 ms
64 bytes from 95.217.214.217: icmp_seq=193 ttl=44 time=215 ms
64 bytes from 95.217.214.217: icmp_seq=194 ttl=44 time=238 ms
64 bytes from 95.217.214.217: icmp_seq=195 ttl=44 time=263 ms
64 bytes from 95.217.214.217: icmp_seq=196 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=197 ttl=44 time=206 ms
64 bytes from 95.217.214.217: icmp_seq=198 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=199 ttl=44 time=251 ms
64 bytes from 95.217.214.217: icmp_seq=200 ttl=44 time=276 ms
64 bytes from 95.217.214.217: icmp_seq=201 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=202 ttl=44 time=218 ms
64 bytes from 95.217.214.217: icmp_seq=203 ttl=44 time=243 ms
64 bytes from 95.217.214.217: icmp_seq=204 ttl=44 time=265 ms
64 bytes from 95.217.214.217: icmp_seq=205 ttl=44 time=287 ms
64 bytes from 95.217.214.217: icmp_seq=206 ttl=44 time=206 ms
64 bytes from 95.217.214.217: icmp_seq=207 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=208 ttl=44 time=252 ms
64 bytes from 95.217.214.217: icmp_seq=209 ttl=44 time=275 ms
64 bytes from 95.217.214.217: icmp_seq=210 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=211 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=212 ttl=44 time=198 ms
64 bytes from 95.217.214.217: icmp_seq=213 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=214 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=215 ttl=44 time=208 ms
64 bytes from 95.217.214.217: icmp_seq=216 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=217 ttl=44 time=253 ms
64 bytes from 95.217.214.217: icmp_seq=218 ttl=44 time=275 ms
64 bytes from 95.217.214.217: icmp_seq=219 ttl=44 time=297 ms
64 bytes from 95.217.214.217: icmp_seq=220 ttl=44 time=218 ms
64 bytes from 95.217.214.217: icmp_seq=221 ttl=44 time=241 ms
64 bytes from 95.217.214.217: icmp_seq=222 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=223 ttl=44 time=287 ms
64 bytes from 95.217.214.217: icmp_seq=224 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=225 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=226 ttl=44 time=253 ms
64 bytes from 95.217.214.217: icmp_seq=227 ttl=44 time=277 ms
64 bytes from 95.217.214.217: icmp_seq=228 ttl=44 time=299 ms
64 bytes from 95.217.214.217: icmp_seq=229 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=230 ttl=44 time=241 ms
64 bytes from 95.217.214.217: icmp_seq=231 ttl=44 time=264 ms
64 bytes from 95.217.214.217: icmp_seq=232 ttl=44 time=286 ms
64 bytes from 95.217.214.217: icmp_seq=233 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=234 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=235 ttl=44 time=252 ms
64 bytes from 95.217.214.217: icmp_seq=236 ttl=44 time=275 ms
64 bytes from 95.217.214.217: icmp_seq=237 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=238 ttl=44 time=218 ms
64 bytes from 95.217.214.217: icmp_seq=239 ttl=44 time=241 ms
64 bytes from 95.217.214.217: icmp_seq=240 ttl=44 time=264 ms
64 bytes from 95.217.214.217: icmp_seq=241 ttl=44 time=287 ms
64 bytes from 95.217.214.217: icmp_seq=242 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=243 ttl=44 time=229 ms
64 bytes from 95.217.214.217: icmp_seq=244 ttl=44 time=252 ms
64 bytes from 95.217.214.217: icmp_seq=245 ttl=44 time=274 ms
64 bytes from 95.217.214.217: icmp_seq=246 ttl=44 time=297 ms
64 bytes from 95.217.214.217: icmp_seq=247 ttl=44 time=217 ms
64 bytes from 95.217.214.217: icmp_seq=248 ttl=44 time=240 ms
64 bytes from 95.217.214.217: icmp_seq=249 ttl=44 time=264 ms
64 bytes from 95.217.214.217: icmp_seq=250 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=251 ttl=44 time=205 ms
64 bytes from 95.217.214.217: icmp_seq=252 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=253 ttl=44 time=251 ms
64 bytes from 95.217.214.217: icmp_seq=254 ttl=44 time=239 ms
64 bytes from 95.217.214.217: icmp_seq=255 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=256 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=257 ttl=44 time=242 ms
64 bytes from 95.217.214.217: icmp_seq=258 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=259 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=260 ttl=44 time=208 ms
64 bytes from 95.217.214.217: icmp_seq=261 ttl=44 time=200 ms
64 bytes from 95.217.214.217: icmp_seq=262 ttl=44 time=253 ms
64 bytes from 95.217.214.217: icmp_seq=263 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=264 ttl=44 time=298 ms
64 bytes from 95.217.214.217: icmp_seq=265 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=266 ttl=44 time=241 ms
64 bytes from 95.217.214.217: icmp_seq=267 ttl=44 time=263 ms
64 bytes from 95.217.214.217: icmp_seq=268 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=269 ttl=44 time=206 ms
64 bytes from 95.217.214.217: icmp_seq=270 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=271 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=272 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=273 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=274 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=275 ttl=44 time=241 ms
64 bytes from 95.217.214.217: icmp_seq=276 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=277 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=278 ttl=44 time=205 ms
64 bytes from 95.217.214.217: icmp_seq=279 ttl=44 time=228 ms
64 bytes from 95.217.214.217: icmp_seq=280 ttl=44 time=252 ms
64 bytes from 95.217.214.217: icmp_seq=281 ttl=44 time=274 ms
64 bytes from 95.217.214.217: icmp_seq=282 ttl=44 time=296 ms
64 bytes from 95.217.214.217: icmp_seq=283 ttl=44 time=216 ms
64 bytes from 95.217.214.217: icmp_seq=284 ttl=44 time=200 ms
64 bytes from 95.217.214.217: icmp_seq=285 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=286 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=287 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=288 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=289 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=290 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=291 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=292 ttl=44 time=216 ms
64 bytes from 95.217.214.217: icmp_seq=293 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=294 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=295 ttl=44 time=196 ms
64 bytes from 95.217.214.217: icmp_seq=296 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=297 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=298 ttl=44 time=251 ms
64 bytes from 95.217.214.217: icmp_seq=299 ttl=44 time=198 ms
64 bytes from 95.217.214.217: icmp_seq=300 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=301 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=302 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=303 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=304 ttl=44 time=286 ms
64 bytes from 95.217.214.217: icmp_seq=305 ttl=44 time=206 ms
64 bytes from 95.217.214.217: icmp_seq=306 ttl=44 time=229 ms
64 bytes from 95.217.214.217: icmp_seq=307 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=308 ttl=44 time=186 ms
64 bytes from 95.217.214.217: icmp_seq=309 ttl=44 time=235 ms
64 bytes from 95.217.214.217: icmp_seq=310 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=311 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=312 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=313 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=314 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=315 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=316 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=317 ttl=44 time=263 ms
64 bytes from 95.217.214.217: icmp_seq=318 ttl=44 time=196 ms
64 bytes from 95.217.214.217: icmp_seq=319 ttl=44 time=220 ms
64 bytes from 95.217.214.217: icmp_seq=320 ttl=44 time=186 ms
64 bytes from 95.217.214.217: icmp_seq=321 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=322 ttl=44 time=287 ms
64 bytes from 95.217.214.217: icmp_seq=323 ttl=44 time=208 ms
64 bytes from 95.217.214.217: icmp_seq=324 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=325 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=326 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=327 ttl=44 time=298 ms
64 bytes from 95.217.214.217: icmp_seq=328 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=329 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=330 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=331 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=332 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=333 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=334 ttl=44 time=254 ms
64 bytes from 95.217.214.217: icmp_seq=335 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=336 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=337 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=338 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=339 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=340 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=341 ttl=44 time=209 ms
64 bytes from 95.217.214.217: icmp_seq=342 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=343 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=344 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=345 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=346 ttl=44 time=222 ms
64 bytes from 95.217.214.217: icmp_seq=347 ttl=44 time=243 ms
64 bytes from 95.217.214.217: icmp_seq=348 ttl=44 time=267 ms
64 bytes from 95.217.214.217: icmp_seq=349 ttl=44 time=289 ms
64 bytes from 95.217.214.217: icmp_seq=350 ttl=44 time=209 ms
64 bytes from 95.217.214.217: icmp_seq=351 ttl=44 time=232 ms
64 bytes from 95.217.214.217: icmp_seq=352 ttl=44 time=254 ms
64 bytes from 95.217.214.217: icmp_seq=353 ttl=44 time=276 ms
64 bytes from 95.217.214.217: icmp_seq=354 ttl=44 time=196 ms
64 bytes from 95.217.214.217: icmp_seq=355 ttl=44 time=219 ms
64 bytes from 95.217.214.217: icmp_seq=356 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=357 ttl=44 time=266 ms
64 bytes from 95.217.214.217: icmp_seq=358 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=359 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=360 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=361 ttl=44 time=255 ms
64 bytes from 95.217.214.217: icmp_seq=362 ttl=44 time=278 ms
64 bytes from 95.217.214.217: icmp_seq=363 ttl=44 time=301 ms
64 bytes from 95.217.214.217: icmp_seq=364 ttl=44 time=223 ms
64 bytes from 95.217.214.217: icmp_seq=365 ttl=44 time=245 ms
64 bytes from 95.217.214.217: icmp_seq=366 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=367 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=368 ttl=44 time=213 ms
64 bytes from 95.217.214.217: icmp_seq=369 ttl=44 time=236 ms
64 bytes from 95.217.214.217: icmp_seq=370 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=371 ttl=44 time=281 ms
64 bytes from 95.217.214.217: icmp_seq=372 ttl=44 time=201 ms
64 bytes from 95.217.214.217: icmp_seq=373 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=374 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=375 ttl=44 time=268 ms
64 bytes from 95.217.214.217: icmp_seq=376 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=377 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=378 ttl=44 time=234 ms
64 bytes from 95.217.214.217: icmp_seq=379 ttl=44 time=187 ms
64 bytes from 95.217.214.217: icmp_seq=380 ttl=44 time=280 ms
64 bytes from 95.217.214.217: icmp_seq=381 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=382 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=383 ttl=44 time=247 ms
64 bytes from 95.217.214.217: icmp_seq=384 ttl=44 time=270 ms
64 bytes from 95.217.214.217: icmp_seq=385 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=386 ttl=44 time=215 ms
64 bytes from 95.217.214.217: icmp_seq=387 ttl=44 time=236 ms
64 bytes from 95.217.214.217: icmp_seq=388 ttl=44 time=259 ms
64 bytes from 95.217.214.217: icmp_seq=389 ttl=44 time=284 ms
64 bytes from 95.217.214.217: icmp_seq=390 ttl=44 time=204 ms
64 bytes from 95.217.214.217: icmp_seq=391 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=392 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=393 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=394 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=395 ttl=44 time=216 ms
64 bytes from 95.217.214.217: icmp_seq=396 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=397 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=398 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=399 ttl=44 time=205 ms
64 bytes from 95.217.214.217: icmp_seq=400 ttl=44 time=227 ms
64 bytes from 95.217.214.217: icmp_seq=401 ttl=44 time=249 ms
64 bytes from 95.217.214.217: icmp_seq=402 ttl=44 time=272 ms
64 bytes from 95.217.214.217: icmp_seq=403 ttl=44 time=296 ms
64 bytes from 95.217.214.217: icmp_seq=404 ttl=44 time=217 ms
64 bytes from 95.217.214.217: icmp_seq=405 ttl=44 time=241 ms
64 bytes from 95.217.214.217: icmp_seq=406 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=407 ttl=44 time=286 ms
64 bytes from 95.217.214.217: icmp_seq=408 ttl=44 time=207 ms
64 bytes from 95.217.214.217: icmp_seq=409 ttl=44 time=230 ms
64 bytes from 95.217.214.217: icmp_seq=410 ttl=44 time=253 ms
64 bytes from 95.217.214.217: icmp_seq=411 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=412 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=413 ttl=44 time=221 ms
64 bytes from 95.217.214.217: icmp_seq=414 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=415 ttl=44 time=267 ms
64 bytes from 95.217.214.217: icmp_seq=416 ttl=44 time=290 ms
64 bytes from 95.217.214.217: icmp_seq=417 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=418 ttl=44 time=235 ms
64 bytes from 95.217.214.217: icmp_seq=419 ttl=44 time=257 ms
64 bytes from 95.217.214.217: icmp_seq=420 ttl=44 time=283 ms
64 bytes from 95.217.214.217: icmp_seq=421 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=422 ttl=44 time=226 ms
64 bytes from 95.217.214.217: icmp_seq=423 ttl=44 time=250 ms
64 bytes from 95.217.214.217: icmp_seq=424 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=425 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=426 ttl=44 time=218 ms
64 bytes from 95.217.214.217: icmp_seq=427 ttl=44 time=240 ms
64 bytes from 95.217.214.217: icmp_seq=428 ttl=44 time=264 ms
64 bytes from 95.217.214.217: icmp_seq=429 ttl=44 time=194 ms
64 bytes from 95.217.214.217: icmp_seq=430 ttl=44 time=210 ms
64 bytes from 95.217.214.217: icmp_seq=431 ttl=44 time=231 ms
64 bytes from 95.217.214.217: icmp_seq=432 ttl=44 time=254 ms
64 bytes from 95.217.214.217: icmp_seq=433 ttl=44 time=280 ms
64 bytes from 95.217.214.217: icmp_seq=434 ttl=44 time=200 ms
64 bytes from 95.217.214.217: icmp_seq=435 ttl=44 time=222 ms
64 bytes from 95.217.214.217: icmp_seq=436 ttl=44 time=244 ms
64 bytes from 95.217.214.217: icmp_seq=437 ttl=44 time=268 ms
64 bytes from 95.217.214.217: icmp_seq=438 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=439 ttl=44 time=213 ms
64 bytes from 95.217.214.217: icmp_seq=440 ttl=44 time=233 ms
64 bytes from 95.217.214.217: icmp_seq=441 ttl=44 time=236 ms
64 bytes from 95.217.214.217: icmp_seq=442 ttl=44 time=281 ms
64 bytes from 95.217.214.217: icmp_seq=443 ttl=44 time=202 ms
64 bytes from 95.217.214.217: icmp_seq=444 ttl=44 time=192 ms
64 bytes from 95.217.214.217: icmp_seq=445 ttl=44 time=186 ms
64 bytes from 95.217.214.217: icmp_seq=446 ttl=44 time=271 ms
64 bytes from 95.217.214.217: icmp_seq=447 ttl=44 time=294 ms
64 bytes from 95.217.214.217: icmp_seq=448 ttl=44 time=214 ms
64 bytes from 95.217.214.217: icmp_seq=449 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=450 ttl=44 time=263 ms
64 bytes from 95.217.214.217: icmp_seq=451 ttl=44 time=285 ms
^C
--- 95.217.214.217 ping statistics ---
452 packets transmitted, 451 received, 0.221239% packet loss, time 451527ms
rtt min/avg/max/mdev = 185.922/230.899/302.876/34.597 ms

Iteration 2

$ hcloud server shutdown testing-demo && ping 95.217.214.217
2.2s [===================================] 100.00%
Sent shutdown signal to server 45292475
PING 95.217.214.217 (95.217.214.217) 56(84) bytes of data.
^C
--- 95.217.214.217 ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 10261ms

Iteration 3

$ hcloud server shutdown testing-demo && ping 95.217.214.217
1.1s [===================================] 100.00%
Sent shutdown signal to server 45292519
PING 95.217.214.217 (95.217.214.217) 56(84) bytes of data.
64 bytes from 95.217.214.217: icmp_seq=1 ttl=44 time=313 ms
64 bytes from 95.217.214.217: icmp_seq=2 ttl=44 time=234 ms
64 bytes from 95.217.214.217: icmp_seq=3 ttl=44 time=188 ms
64 bytes from 95.217.214.217: icmp_seq=4 ttl=44 time=189 ms
64 bytes from 95.217.214.217: icmp_seq=5 ttl=44 time=200 ms
64 bytes from 95.217.214.217: icmp_seq=6 ttl=44 time=195 ms
64 bytes from 95.217.214.217: icmp_seq=7 ttl=44 time=190 ms
64 bytes from 95.217.214.217: icmp_seq=8 ttl=44 time=191 ms
64 bytes from 95.217.214.217: icmp_seq=9 ttl=44 time=193 ms
64 bytes from 95.217.214.217: icmp_seq=10 ttl=44 time=213 ms
64 bytes from 95.217.214.217: icmp_seq=11 ttl=44 time=237 ms
64 bytes from 95.217.214.217: icmp_seq=12 ttl=44 time=260 ms
64 bytes from 95.217.214.217: icmp_seq=13 ttl=44 time=284 ms
64 bytes from 95.217.214.217: icmp_seq=14 ttl=44 time=202 ms
64 bytes from 95.217.214.217: icmp_seq=15 ttl=44 time=225 ms
64 bytes from 95.217.214.217: icmp_seq=16 ttl=44 time=250 ms
^C
--- 95.217.214.217 ping statistics ---
16 packets transmitted, 16 received, 0% packet loss, time 15016ms
rtt min/avg/max/mdev = 187.858/222.706/312.922/36.549 ms

Iteration 4

$ hcloud server create --name=testing-demo --image=ubuntu-22.04 --type=cx21 --ssh-key=$SSH_KEY_NAME
5.4s [===================================] 100.00%
Waiting for server 45292545 to have started
 ... done
Server 45292545 created
IPv4: 95.217.214.217
IPv6: 2a01:4f9:c012:cebe::1
IPv6 Network: 2a01:4f9:c012:cebe::/64
Root password: nTqX4kccWPPbbAxsXPH3
~/w/u/caph (main|✔) $ hcloud server shutdown testing-demo && ping 95.217.214.217
1.0s [===================================] 100.00%
Sent shutdown signal to server 45292545
PING 95.217.214.217 (95.217.214.217) 56(84) bytes of data.
64 bytes from 95.217.214.217: icmp_seq=1 ttl=44 time=1072 ms
64 bytes from 95.217.214.217: icmp_seq=3 ttl=44 time=267 ms
64 bytes from 95.217.214.217: icmp_seq=4 ttl=44 time=290 ms
64 bytes from 95.217.214.217: icmp_seq=5 ttl=44 time=211 ms
64 bytes from 95.217.214.217: icmp_seq=6 ttl=44 time=234 ms
64 bytes from 95.217.214.217: icmp_seq=7 ttl=44 time=257 ms
64 bytes from 95.217.214.217: icmp_seq=8 ttl=44 time=280 ms
64 bytes from 95.217.214.217: icmp_seq=9 ttl=44 time=201 ms
64 bytes from 95.217.214.217: icmp_seq=10 ttl=44 time=224 ms
64 bytes from 95.217.214.217: icmp_seq=11 ttl=44 time=246 ms
64 bytes from 95.217.214.217: icmp_seq=12 ttl=44 time=270 ms
^C
--- 95.217.214.217 ping statistics ---
12 packets transmitted, 11 received, 8.33333% packet loss, time 11050ms
rtt min/avg/max/mdev = 200.590/322.828/1072.302/238.564 ms, pipe 2

Iteration 5

$ hcloud server shutdown testing-demo && ping 95.217.214.217
1.1s [===================================] 100.00%
Sent shutdown signal to server 45292667
PING 95.217.214.217 (95.217.214.217) 56(84) bytes of data.
64 bytes from 95.217.214.217: icmp_seq=1 ttl=44 time=2275 ms
64 bytes from 95.217.214.217: icmp_seq=4 ttl=44 time=262 ms
64 bytes from 95.217.214.217: icmp_seq=5 ttl=44 time=285 ms
64 bytes from 95.217.214.217: icmp_seq=6 ttl=44 time=206 ms
64 bytes from 95.217.214.217: icmp_seq=7 ttl=44 time=230 ms
^C
--- 95.217.214.217 ping statistics ---
7 packets transmitted, 5 received, 28.5714% packet loss, time 6038ms
rtt min/avg/max/mdev = 206.444/651.509/2274.575/811.974 ms, pipe 3

After a little bit of more troubleshooting I found the reason I think why this is happening. I think acpid service is not starting well in ubuntu system most of the times. Please note that it start well in some of the times but in most cases it doesn't.

image

@guettli
Copy link
Contributor

guettli commented Apr 2, 2024

@kranurag7 thank you for finding the acpi-daemon issue. I think there should not be randomness. You see "sometimes". Could you have a look at this? In the above screenshot the service is disabled. On which machines is the acpid enabled? Maybe this is related to our custom images?

@apricote which machine image do you use? Do you use a custom hcloud machine image, or the default provided by caph? Is the acpi-daemon enabled on your nodes?

@apricote
Copy link
Contributor

apricote commented Apr 2, 2024

I build custom images with packer, inspired by templates you have in the repo. They are based on ubuntu-22.04. I only run cax/Arm servers.

Do you have some logs from CAPH or Kubernetes Events that I could search for to verify that I am not hitting the issue?

If you have a lot of volumes attached to the Nodes, this can take a while to detach, as only one operation per server may be in-flight at a time and there is some backoff involved before retrying.

@guettli
Copy link
Contributor

guettli commented Apr 2, 2024

Here it took 50 seconds to delete an empty hcloud worker node:

❯ time k delete machine caph-guettli-md-0-zvkfh-pgzv4
machine.cluster.x-k8s.io "caph-guettli-md-0-zvkfh-pgzv4" deleted

real    0m50,264s
user    0m0,049s
sys     0m0,024s

NAMESPACE                           LAST SEEN   TYPE      REASON                        OBJECT                                                                MESSAGE
...
default                             100s        Normal    HCloudServerDeleted           hcloudmachine/caph-guettli-md-0-kdnwh                                 HCloud server caph-guettli-md-0-kdnwh deleted
default                             86s         Normal    DetectedUnhealthy             machine/caph-guettli-md-0-zvkfh-pgzv4                                 Machine default/caph-guettli-md-0-unhealthy-5m/caph-guettli-md-0-zvkfh-pgzv4/caph-guettli-md-0-kdnwh has unhealthy node caph-guettli-md-0-kdnwh
default                             85s         Normal    MachineMarkedUnhealthy        machine/caph-guettli-md-0-zvkfh-pgzv4                                 Machine default/caph-guettli-md-0-unhealthy-5m/caph-guettli-md-0-zvkfh-pgzv4/ has been marked as unhealthy
default                             80s         Normal    SuccessfulCreate              machineset/caph-guettli-md-0-zvkfh                                    Created machine "caph-guettli-md-0-zvkfh-dsrmt"
default                             80s         Normal    SuccessfulDrainNode           machine/caph-guettli-md-0-zvkfh-pgzv4                                 success draining Machine's node "caph-guettli-md-0-kdnwh"
default                             80s         Normal    NodeVolumesDetached           machine/caph-guettli-md-0-zvkfh-pgzv4                                 success waiting for node volumes detaching Machine's node "caph-guettli-md-0-kdnwh"
default                             78s         Normal    SuccessfulCreate              hcloudmachine/caph-guettli-md-0-qhjfm                                 Created new server with id 45328632
default                             11s         Normal    CSRApproved                   certificatesigningrequest/csr-kn8dv                                   approved csr
default                             10s         Normal    SuccessfulSetNodeRef          machine/caph-guettli-md-0-zvkfh-dsrmt                                 caph-guettli-md-0-qhjfm
default                             10s         Normal    DetectedUnhealthy             machine/caph-guettli-md-0-zvkfh-dsrmt                                 Machine default/caph-guettli-md-0-unhealthy-5m/caph-guettli-md-0-zvkfh-dsrmt/ has unhealthy node

The acpi daemon was disabled:

# systemctl status acpid

○ acpid.service - ACPI event daemon
     Loaded: loaded (/lib/systemd/system/acpid.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● acpid.path
             ● acpid.socket
       Docs: man:acpid(8)

With other words: I could not reproduce the issue.

@batistein
Copy link
Contributor

A little background here. Volume detachment happens before a server is deleted. The cluster-api marks the hcloudmachine object to be deleted, after the node has been disabled during scheduling and all kubernetes operations have been completed. Then we trigger the reconcile delete function. So those 2 minutes and timeout are not counted from a user wanting to remove the machine, but from the actual delete process in caph.

But I guess now that we have determined that the acpid.service is not running, we have a path to follow. @apricote, is there anything similar for robot servers?

@guettli
Copy link
Contributor

guettli commented Aug 15, 2024

I have not seen that issue during the last months. But I have seen that:

capi-machine got deletionTimestamp, drain starts and hangs. Provider-machine does not get deleted because drain hangs.

In the capi-logs you see something like:

"error when evicting pods/"ccm-ccm-hetzner-86568d4cfd-skgkn" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.\n"

Back to above topic: I think we need numbers first. How long does it take on average, how large is the statistical deviation?

When we have some numbers, we can make better informed decisions here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kranurag7 kranurag7

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@guettli @apricote @batistein @janiskemper @kranurag7