Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increasing the prewarming pool size isn't working as expected #716

Open
MrSerth opened this issue Oct 13, 2024 · 0 comments
Open

Increasing the prewarming pool size isn't working as expected #716

MrSerth opened this issue Oct 13, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@MrSerth
Copy link
Member

MrSerth commented Oct 13, 2024 •
edited
Loading

All times in UTC

On October 13th, 2024 at 22:29:45, I increase the prewarming pool size for environment 29 from 15 to 30 runners. This worked fine and I saw a respective increase in runners on CodeOcean.

A few minutes later, however, I got errors POSEIDON-3W and CODEOCEAN-ZW indicating that something wasn't right. In Grafana, I saw the following:

Bildschirmfoto 2024-10-14 um 01 11 48

Poseidon logs for that time included:

Details 

time="2024-10-13T22:34:45.647306Z" level=warning msg="Prewarming Pool Alert. Reloading environment" environment_id=29 package=runner

time="2024-10-13T22:34:45.662670Z" level=debug msg="Recovered Allocation" jobID=29-55235151-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662719Z" level=debug msg="Recovered Allocation" jobID=29-411b1f4c-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662729Z" level=debug msg="Recovered Allocation" jobID=29-27aab21b-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662742Z" level=debug msg="Recovered Allocation" jobID=29-223e4424-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662753Z" level=debug msg="Recovered Allocation" jobID=29-2092bbf5-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662762Z" level=debug msg="Recovered Allocation" jobID=29-152075fd-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662771Z" level=debug msg="Recovered Allocation" jobID=29-04ac3914-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662801Z" level=debug msg="Recovered Allocation" jobID=29-e6b97961-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662809Z" level=debug msg="Recovered Allocation" jobID=29-dddcbeb1-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662819Z" level=debug msg="Recovered Allocation" jobID=29-d8efc22c-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662827Z" level=debug msg="Recovered Allocation" jobID=29-d4e50263-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662836Z" level=debug msg="Recovered Allocation" jobID=29-c9ae9635-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662845Z" level=debug msg="Recovered Allocation" jobID=29-c570a770-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662854Z" level=debug msg="Recovered Allocation" jobID=29-be5de26a-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662874Z" level=debug msg="Recovered Allocation" jobID=29-b971556d-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662893Z" level=debug msg="Recovered Allocation" jobID=29-b45314fd-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662902Z" level=debug msg="Recovered Allocation" jobID=29-abf74b2d-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662913Z" level=debug msg="Recovered Allocation" jobID=29-abe165d8-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662921Z" level=debug msg="Recovered Allocation" jobID=29-abc9df13-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662958Z" level=debug msg="Recovered Allocation" jobID=29-abb22d68-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662966Z" level=debug msg="Recovered Allocation" jobID=29-ab9b0271-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662975Z" level=debug msg="Recovered Allocation" jobID=29-ab8501dd-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.662996Z" level=debug msg="Recovered Allocation" jobID=29-ab70831e-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663012Z" level=debug msg="Recovered Allocation" jobID=29-ab55ef11-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663020Z" level=debug msg="Recovered Allocation" jobID=29-ab406330-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663028Z" level=debug msg="Recovered Allocation" jobID=29-ab268907-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663036Z" level=debug msg="Recovered Allocation" jobID=29-ab0ad05f-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663045Z" level=debug msg="Recovered Allocation" jobID=29-aaf0095a-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663054Z" level=debug msg="Recovered Allocation" jobID=29-aadf2c0c-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663062Z" level=debug msg="Recovered Allocation" jobID=29-aad16fb3-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663071Z" level=debug msg="Recovered Allocation" jobID=29-aab7e53f-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663106Z" level=debug msg="Recovered Allocation" jobID=29-aaa912fe-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663114Z" level=debug msg="Recovered Allocation" jobID=29-aa8fae5b-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663125Z" level=debug msg="Recovered Allocation" jobID=29-aa8357db-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663135Z" level=debug msg="Recovered Allocation" jobID=29-aa6e3777-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663164Z" level=debug msg="Recovered Allocation" jobID=29-aa58f381-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663172Z" level=debug msg="Recovered Allocation" jobID=29-aa40d91d-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663180Z" level=debug msg="Recovered Allocation" jobID=29-aa284cd5-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663209Z" level=debug msg="Recovered Allocation" jobID=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663217Z" level=debug msg="Recovered Allocation" jobID=29-a9fd2a00-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663229Z" level=debug msg="Recovered Allocation" jobID=29-a9f0893e-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663237Z" level=debug msg="Recovered Allocation" jobID=29-a9e1eecd-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663247Z" level=debug msg="Recovered Allocation" jobID=29-a9caffcc-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:45.663254Z" level=debug msg="Recovered Allocation" jobID=29-a98b980e-89b2-11ef-9212-fa163eb9b043 package=nomad status=running

time="2024-10-13T22:34:45.723157Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-04ac3914-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.725044Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-152075fd-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.726959Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-2092bbf5-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.728876Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-223e4424-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.731570Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-27aab21b-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.733577Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-411b1f4c-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.735469Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-55235151-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.737404Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a98b980e-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.739514Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-a9caffcc-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.742390Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a9e1eecd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.744252Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a9f0893e-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.746125Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a9fd2a00-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.749463Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.753610Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa284cd5-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.758402Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa40d91d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.764992Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa58f381-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.771958Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aa6e3777-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.781725Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa8357db-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.793106Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa8fae5b-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.801033Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aaa912fe-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.802954Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aab7e53f-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.804924Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aad16fb3-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.806821Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aadf2c0c-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.808623Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.811421Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab0ad05f-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.813463Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab268907-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.815289Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab406330-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.817134Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab55ef11-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.818948Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab70831e-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.821708Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.823546Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.825417Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abb22d68-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.827235Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abc9df13-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.829079Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abe165d8-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.831967Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abf74b2d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.833868Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-b45314fd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.835621Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-b971556d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.837611Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-be5de26a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.840600Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-c570a770-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.842535Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-c9ae9635-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.844411Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-d4e50263-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.846166Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-d8efc22c-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.848137Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-dddcbeb1-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.851040Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-e6b97961-89b2-11ef-9212-fa163eb9b043

time="2024-10-13T22:34:45.851066Z" level=info msg="Too many idle runner" environment_id=29 error="cannot scale down the environment" idle runners=30 package=environment pool size=15
time="2024-10-13T22:34:45.851102Z" level=debug msg="Destroying Runner" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-d8efc22c-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:45.851117Z" level=debug msg="Runner destroyed locally" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-d8efc22c-89b2-11ef-9212-fa163eb9b043

# ......

time="2024-10-13T22:34:51.104316Z" level=warning msg="Prewarming Pool Alert. Reloading environment" environment_id=29 package=runner

time="2024-10-13T22:34:51.115005Z" level=debug msg="Recovered Allocation" jobID=29-55235151-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115045Z" level=debug msg="Recovered Allocation" jobID=29-411b1f4c-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115054Z" level=debug msg="Recovered Allocation" jobID=29-27aab21b-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115063Z" level=debug msg="Recovered Allocation" jobID=29-223e4424-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115072Z" level=debug msg="Recovered Allocation" jobID=29-2092bbf5-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115081Z" level=debug msg="Recovered Allocation" jobID=29-152075fd-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115090Z" level=debug msg="Recovered Allocation" jobID=29-04ac3914-89b3-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115099Z" level=debug msg="Recovered Allocation" jobID=29-e6b97961-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115108Z" level=debug msg="Recovered Allocation" jobID=29-dddcbeb1-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115115Z" level=debug msg="Recovered Allocation" jobID=29-d8efc22c-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115138Z" level=debug msg="Recovered Allocation" jobID=29-d4e50263-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115150Z" level=debug msg="Recovered Allocation" jobID=29-c9ae9635-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115161Z" level=debug msg="Recovered Allocation" jobID=29-c570a770-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115171Z" level=debug msg="Recovered Allocation" jobID=29-be5de26a-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115181Z" level=debug msg="Recovered Allocation" jobID=29-b971556d-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115189Z" level=debug msg="Recovered Allocation" jobID=29-b45314fd-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115198Z" level=debug msg="Recovered Allocation" jobID=29-abf74b2d-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115205Z" level=debug msg="Recovered Allocation" jobID=29-abe165d8-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115215Z" level=debug msg="Recovered Allocation" jobID=29-abc9df13-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115224Z" level=debug msg="Recovered Allocation" jobID=29-abb22d68-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115232Z" level=debug msg="Recovered Allocation" jobID=29-ab9b0271-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115240Z" level=debug msg="Recovered Allocation" jobID=29-ab8501dd-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115249Z" level=debug msg="Recovered Allocation" jobID=29-ab70831e-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115258Z" level=debug msg="Recovered Allocation" jobID=29-ab55ef11-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115272Z" level=debug msg="Recovered Allocation" jobID=29-ab406330-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115280Z" level=debug msg="Recovered Allocation" jobID=29-ab268907-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115299Z" level=debug msg="Recovered Allocation" jobID=29-ab0ad05f-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115307Z" level=debug msg="Recovered Allocation" jobID=29-aaf0095a-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115316Z" level=debug msg="Recovered Allocation" jobID=29-aadf2c0c-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115324Z" level=debug msg="Recovered Allocation" jobID=29-aad16fb3-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115331Z" level=debug msg="Recovered Allocation" jobID=29-aab7e53f-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115340Z" level=debug msg="Recovered Allocation" jobID=29-aaa912fe-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115391Z" level=debug msg="Recovered Allocation" jobID=29-aa8fae5b-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115400Z" level=debug msg="Recovered Allocation" jobID=29-aa8357db-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115407Z" level=debug msg="Recovered Allocation" jobID=29-aa6e3777-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115416Z" level=debug msg="Recovered Allocation" jobID=29-aa58f381-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115423Z" level=debug msg="Recovered Allocation" jobID=29-aa40d91d-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115445Z" level=debug msg="Recovered Allocation" jobID=29-aa284cd5-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115462Z" level=debug msg="Recovered Allocation" jobID=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115480Z" level=debug msg="Recovered Allocation" jobID=29-a9fd2a00-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115489Z" level=debug msg="Recovered Allocation" jobID=29-a9f0893e-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.115500Z" level=debug msg="Recovered Allocation" jobID=29-a9e1eecd-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.116398Z" level=debug msg="Recovered Allocation" jobID=29-a9caffcc-89b2-11ef-9212-fa163eb9b043 package=nomad status=running
time="2024-10-13T22:34:51.116423Z" level=debug msg="Recovered Allocation" jobID=29-a98b980e-89b2-11ef-9212-fa163eb9b043 package=nomad status=running

time="2024-10-13T22:34:51.168729Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-04ac3914-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.170696Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-152075fd-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.172657Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-2092bbf5-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.174541Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-223e4424-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.177237Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-27aab21b-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.179131Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-411b1f4c-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.180979Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-55235151-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.182879Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a98b980e-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.184716Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-a9caffcc-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.187467Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a9e1eecd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.189440Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a9f0893e-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.191368Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-a9fd2a00-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.193249Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.195198Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa284cd5-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.198022Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa40d91d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.199999Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa58f381-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.201899Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aa6e3777-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.203834Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa8357db-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.206784Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aa8fae5b-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.209140Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aaa912fe-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.211583Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aab7e53f-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.213963Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-aad16fb3-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.217107Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aadf2c0c-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.223797Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.230251Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab0ad05f-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.239374Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab268907-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.250878Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab406330-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.253451Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab55ef11-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.256263Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-ab70831e-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.258464Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.260414Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.262328Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abb22d68-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.264195Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abc9df13-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.266946Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abe165d8-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.268961Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-abf74b2d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.270946Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-b45314fd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.272808Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-b971556d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.275753Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-be5de26a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.277702Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-c570a770-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.279586Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-c9ae9635-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.281598Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-d4e50263-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.283611Z" level=debug msg="Recovered Runner" isUsed=true package=runner runner_id=29-d8efc22c-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.286601Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-dddcbeb1-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.288519Z" level=debug msg="Recovered Runner" isUsed=false package=runner runner_id=29-e6b97961-89b2-11ef-9212-fa163eb9b043

time="2024-10-13T22:34:51.288545Z" level=debug msg="Creating new runners" environment_id=29 package=environment runnersRequired=0
time="2024-10-13T22:34:51.288574Z" level=debug msg="Destroying Runner" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-b971556d-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:51.288592Z" level=debug msg="Runner destroyed locally" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-b971556d-89b2-11ef-9212-fa163eb9b043

# ..............

time="2024-10-13T22:34:52.770522Z" level=debug msg="Execution canceled by context" destroy_reason="the destruction should not cause external changes" environment_id=29 package=nomad runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.770522Z" level=debug msg="Execution canceled by context" destroy_reason="the destruction should not cause external changes" environment_id=29 package=nomad runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.770567Z" level=debug code=204 duration=10.584237ms method=PATCH path=/api/v1/runners/29-aa11bcb0-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:52.771970Z" level=debug code=200 duration="66.38µs" method=POST path=/api/v1/runners/29-aa11bcb0-89b2-11ef-9212-fa163eb9b043/execute user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:52.783831Z" level=info msg="Running execution" environment_id=29 executionID=0fcb2db7-d643-4344-bb23-2f8ab9a377fc package=api runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.783948Z" level=info msg="Execution returned" environment_id=29 package=api runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.783959Z" level=warning msg="Error executing the request" environment_id=29 error="the destruction should not cause external changes" package=ws runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.785272Z" level=debug code=200 duration=1.509174ms method=GET path=/api/v1/runners/29-aa11bcb0-89b2-11ef-9212-fa163eb9b043/websocket user_agent=
time="2024-10-13T22:34:52.791315Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.792043Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.792073Z" level=debug msg="Execution terminated after SIGQUIT" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.806508Z" level=debug msg="Destroying Runner" destroy_reason="the client wants to stop the runner" package=runner runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:52.812969Z" level=debug code=204 duration=6.477056ms method=DELETE path=/api/v1/runners/29-aa11bcb0-89b2-11ef-9212-fa163eb9b043 user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:52.876298Z" level=debug msg="Runner stopped" package=runner runner_id=29-aa11bcb0-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.177151Z" level=debug msg="Execution canceled by context" destroy_reason="the destruction should not cause external changes" environment_id=29 package=nomad runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.177218Z" level=debug code=204 duration=10.017206ms method=PATCH path=/api/v1/runners/29-ab8501dd-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:55.178613Z" level=debug code=200 duration="61.31µs" method=POST path=/api/v1/runners/29-ab8501dd-89b2-11ef-9212-fa163eb9b043/execute user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:55.187821Z" level=info msg="Running execution" environment_id=29 executionID=bdb34d65-d848-4e73-8ff0-91857cf833bd package=api runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.188039Z" level=info msg="Execution returned" environment_id=29 package=api runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.188073Z" level=warning msg="Error executing the request" environment_id=29 error="the destruction should not cause external changes" package=ws runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.188781Z" level=debug code=200 duration=1.049152ms method=GET path=/api/v1/runners/29-ab8501dd-89b2-11ef-9212-fa163eb9b043/websocket user_agent=
time="2024-10-13T22:34:55.197201Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.197361Z" level=debug msg="Destroying Runner" destroy_reason="the client wants to stop the runner" package=runner runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.197582Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:55.202313Z" level=debug code=204 duration=4.989813ms method=DELETE path=/api/v1/runners/29-ab8501dd-89b2-11ef-9212-fa163eb9b043 user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:55.206259Z" level=debug msg="Runner stopped" package=runner runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.548338Z" level=debug code=204 duration="97.929µs" method=GET path=/api/v1/health user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:56.745985Z" level=debug msg="Execution canceled by context" destroy_reason="the destruction should not cause external changes" environment_id=29 package=nomad runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.746051Z" level=debug code=204 duration=10.209077ms method=PATCH path=/api/v1/runners/29-aaf0095a-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:56.749468Z" level=debug code=200 duration="48.61µs" method=POST path=/api/v1/runners/29-aaf0095a-89b2-11ef-9212-fa163eb9b043/execute user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:56.761302Z" level=info msg="Running execution" environment_id=29 executionID=a17e092c-2386-4269-a329-a50c832dfccf package=api runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.761533Z" level=info msg="Execution returned" environment_id=29 package=api runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.761565Z" level=warning msg="Error executing the request" environment_id=29 error="the destruction should not cause external changes" package=ws runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.762157Z" level=debug code=200 duration="956.842µs" method=GET path=/api/v1/runners/29-aaf0095a-89b2-11ef-9212-fa163eb9b043/websocket user_agent=
time="2024-10-13T22:34:56.774757Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.776022Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.780684Z" level=debug msg="Destroying Runner" destroy_reason="the client wants to stop the runner" package=runner runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:56.790918Z" level=debug code=204 duration=10.267007ms method=DELETE path=/api/v1/runners/29-aaf0095a-89b2-11ef-9212-fa163eb9b043 user_agent="Faraday v2.12.0"
time="2024-10-13T22:34:56.837529Z" level=debug msg="Runner stopped" package=runner runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:58.188475Z" level=info msg="the execution did not stop after SIGQUIT" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:58.188528Z" level=debug msg="Destroying Runner" destroy_reason="the execution did not stop after SIGQUIT" package=runner runner_id=29-ab8501dd-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:59.761721Z" level=info msg="the execution did not stop after SIGQUIT" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:34:59.761966Z" level=debug msg="Destroying Runner" destroy_reason="the execution did not stop after SIGQUIT" package=runner runner_id=29-aaf0095a-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:00.683819Z" level=debug code=410 duration="662.382µs" method=PATCH path=/api/v1/runners/29-ab8501dd-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:00.685871Z" level=debug code=200 duration="208.99µs" method=POST path=/api/v1/runners user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:00.702987Z" level=debug msg="Handle Allocation Event" ClientStatus=pending DesiredStatus=run NextAllocation= PrevAllocation= alloc_id=f54f1399-d389-7ee6-f250-b60ee01806f7 package=nomad
time="2024-10-13T22:35:00.723794Z" level=debug msg="Ignoring duplicate event" allocID=728c5ff9-2bd3-1e8a-b022-358cce2ce4b4 package=nomad
time="2024-10-13T22:35:00.826130Z" level=debug code=204 duration=133.970109ms method=PATCH path=/api/v1/runners/29-ab55ef11-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:00.887518Z" level=debug code=200 duration="65.05µs" method=POST path=/api/v1/runners/29-ab55ef11-89b2-11ef-9212-fa163eb9b043/execute user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:00.900176Z" level=info msg="Running execution" environment_id=29 executionID=54df1131-b3ec-4ff8-b54b-a3c61063ade5 package=api runner_id=29-ab55ef11-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:01.188137Z" level=info msg="Execution returned" environment_id=29 package=api runner_id=29-ab55ef11-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:01.188217Z" level=debug code=200 duration=288.11683ms method=GET path=/api/v1/runners/29-ab55ef11-89b2-11ef-9212-fa163eb9b043/websocket user_agent=
time="2024-10-13T22:35:01.322902Z" level=debug code=200 duration=131.705093ms method=GET path=/api/v1/runners/29-ab55ef11-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:01.536852Z" level=debug msg="Handle Allocation Event" ClientStatus=running DesiredStatus=run NextAllocation= PrevAllocation= alloc_id=f54f1399-d389-7ee6-f250-b60ee01806f7 package=nomad
time="2024-10-13T22:35:01.536900Z" level=debug msg="Runner started" package=runner runner_id=29-61ed2888-89b3-11ef-9212-fa163eb9b043 startupDuration=833.836522ms
time="2024-10-13T22:35:02.802559Z" level=debug msg="Execution canceled by context" destroy_reason="the destruction should not cause external changes" environment_id=29 package=nomad runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.802609Z" level=debug code=204 duration=8.931883ms method=PATCH path=/api/v1/runners/29-ab9b0271-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:02.803991Z" level=debug code=200 duration="85.16µs" method=POST path=/api/v1/runners/29-ab9b0271-89b2-11ef-9212-fa163eb9b043/execute user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:02.813808Z" level=info msg="Running execution" environment_id=29 executionID=1bef0c5d-b048-49a9-bc22-e3927e2779bd package=api runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.813957Z" level=info msg="Execution returned" environment_id=29 package=api runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.813978Z" level=warning msg="Error executing the request" environment_id=29 error="the destruction should not cause external changes" package=ws runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.814657Z" level=debug code=200 duration="938.322µs" method=GET path=/api/v1/runners/29-ab9b0271-89b2-11ef-9212-fa163eb9b043/websocket user_agent=
time="2024-10-13T22:35:02.821565Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.823516Z" level=debug msg="Execution canceled by context" environment_id=29 package=nomad runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.823540Z" level=debug msg="Execution terminated after SIGQUIT" destroy_reason="the destruction should not cause external changes" package=runner runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.830408Z" level=debug msg="Destroying Runner" destroy_reason="the client wants to stop the runner" package=runner runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:02.834709Z" level=debug code=204 duration=4.32503ms method=DELETE path=/api/v1/runners/29-ab9b0271-89b2-11ef-9212-fa163eb9b043 user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:02.930067Z" level=debug msg="Runner stopped" package=runner runner_id=29-ab9b0271-89b2-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:04.289400Z" level=debug code=410 duration="295.271µs" method=PATCH path=/api/v1/runners/29-aaf0095a-89b2-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:04.290693Z" level=debug code=200 duration="59.539µs" method=POST path=/api/v1/runners user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:04.386859Z" level=debug msg="Ignoring duplicate event" allocID=f54f1399-d389-7ee6-f250-b60ee01806f7 package=nomad
time="2024-10-13T22:35:04.390793Z" level=debug msg="Handle Allocation Event" ClientStatus=pending DesiredStatus=run NextAllocation= PrevAllocation= alloc_id=7583c73f-8d22-5995-e2d7-8fef71efbb7c package=nomad
time="2024-10-13T22:35:04.410773Z" level=debug code=204 duration=113.901149ms method=PATCH path=/api/v1/runners/29-61ed2888-89b3-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:04.454428Z" level=debug code=200 duration="57.229µs" method=POST path=/api/v1/runners/29-61ed2888-89b3-11ef-9212-fa163eb9b043/execute user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:04.465968Z" level=info msg="Running execution" environment_id=29 executionID=94476f91-93bd-4d7c-99b4-f98c5b4b6957 package=api runner_id=29-61ed2888-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:04.736965Z" level=info msg="Execution returned" environment_id=29 package=api runner_id=29-61ed2888-89b3-11ef-9212-fa163eb9b043
time="2024-10-13T22:35:04.737012Z" level=debug code=200 duration=271.093897ms method=GET path=/api/v1/runners/29-61ed2888-89b3-11ef-9212-fa163eb9b043/websocket user_agent=
time="2024-10-13T22:35:04.835387Z" level=debug code=204 duration="90.55µs" method=GET path=/api/v1/health user_agent="check_http/v2.3.1 (monitoring-plugins 2.3.1)"
time="2024-10-13T22:35:04.866776Z" level=debug code=200 duration=127.584662ms method=GET path=/api/v1/runners/29-61ed2888-89b3-11ef-9212-fa163eb9b043/files user_agent="Faraday v2.12.0"
time="2024-10-13T22:35:05.254853Z" level=debug msg="Handle Allocation Event" ClientStatus=running DesiredStatus=run NextAllocation= PrevAllocation= alloc_id=7583c73f-8d22-5995-e2d7-8fef71efbb7c package=nomad
time="2024-10-13T22:35:05.254913Z" level=debug msg="Runner started" package=runner runner_id=29-64133775-89b3-11ef-9212-fa163eb9b043 startupDuration=864.085859ms

# .........

CodeOcean did not show interesting logs. A few minutes after the issue occurred, I checked Nomad, too. There, the number of runners was expected (at 47, which are 30 idle runners and 17 used runners). We should investigate what happened and fix the problem.
@MrSerth MrSerth added the bug Something isn't working label Oct 13, 2024
@MrSerth MrSerth mentioned this issue Oct 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MrSerth