Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stopping the agent results in a -2 worker exit #466

Open
roberth opened this issue Nov 16, 2022 · 0 comments
Open

Stopping the agent results in a -2 worker exit #466

roberth opened this issue Nov 16, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@roberth
Copy link
Member

roberth commented Nov 16, 2022

Description

Stopping the agent may result in -2 worker exit in builds.

To Reproduce

systemctl stop hercules-ci-agent.service while builds are running.

Expected behavior

Worker exit status -2 should be treated as a transient error. The job is queued again.

Logs

Small extract. The task ids may not match up, but that's because many more

Nov 16 01:05:47 hx90 systemd[1]: Started hercules-ci-agent.service.
Nov 16 01:05:47 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:47][Agent][Info][hx90][PID 645568][ThreadId 27][agent-version:0.9.7][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:115:19] Agent online.
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 39][agent-version:0.9.7][task:0c1d764e-3311-4e2a-8751-0801f3fe4b38][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 43][agent-version:0.9.7][task:0ce24793-99d7-4f43-856a-d8ac0f264c8c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 47][agent-version:0.9.7][task:0d0a0088-0c47-4717-a118-85ad31ca08e5][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 48][agent-version:0.9.7][task:0d500f24-10a1-4fb4-bcde-37766a408b54][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 56][agent-version:0.9.7][task:0d750aec-fa83-4060-820e-0c46ae70154c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 60][agent-version:0.9.7][task:0ec977c8-4a58-4fbd-a9d2-367195ec2741][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:48 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:48][Agent][Info][hx90][PID 645568][ThreadId 65][agent-version:0.9.7][task:0f294b1b-d071-4c93-9dc2-d890ee89df7b][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 69][agent-version:0.9.7][task:0f42617f-44ab-46b3-b489-3fd06327922b][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 73][agent-version:0.9.7][task:1079eb47-bab5-4d4b-b9fd-89f0a2d79ed0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 77][agent-version:0.9.7][task:10879f28-bf93-4fe6-b07b-54ada677234c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 82][agent-version:0.9.7][task:0a3f4183-d82c-4a2b-bef7-d80d03b6282c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 86][agent-version:0.9.7][task:0a85d79c-3cd3-49d2-a2bd-220020c9339a][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 90][agent-version:0.9.7][task:0ab6c30b-82f8-460a-b3fe-2da43294bf7a][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 95][agent-version:0.9.7][task:11d23872-cbf6-4e7c-bc12-70f2a292345f][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 99][agent-version:0.9.7][task:137f4774-3b0f-477a-8c8d-46d82e8ebf23][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:49 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:49][Agent][Info][hx90][PID 645568][ThreadId 103][agent-version:0.9.7][task:13e14184-3aa9-42c7-bd5e-4362a01385b8][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:52 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:52][Agent][Info][hx90][PID 645568][ThreadId 56][agent-version:0.9.7][task:0d750aec-fa83-4060-820e-0c46ae70154c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:175:9] Completed task successfully
Nov 16 01:05:53 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:53][Agent][Info][hx90][PID 645568][ThreadId 73][agent-version:0.9.7][task:1079eb47-bab5-4d4b-b9fd-89f0a2d79ed0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:175:9] Completed task successfully
Nov 16 01:05:54 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:54][Agent][Info][hx90][PID 645568][ThreadId 129][agent-version:0.9.7][task:15150195-cd00-4b22-9dcd-cf2b79552115][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:05:54 hx90 hercules-ci-agent[645568]: [2022-11-16 09:05:54][Agent][Info][hx90][PID 645568][ThreadId 133][agent-version:0.9.7][task:15c1de66-b732-4f5e-b4b8-0b4e42020b8d][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sent signal SIGTERM to main process 645568 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645624 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645655 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645686 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645689 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645781 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645814 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645816 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645887 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645910 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645924 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645962 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 645966 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 646003 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 646076 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 646564 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Sending signal SIGTERM to process 646593 (hercules-ci-age) on client request.
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 60][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0ec977c8-4a58-4fbd-a9d2-367195ec2741][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 86][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0a85d79c-3cd3-49d2-a2bd-220020c9339a][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 39][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0c1d764e-3311-4e2a-8751-0801f3fe4b38][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 133][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:15c1de66-b732-4f5e-b4b8-0b4e42020b8d][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 48][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0d500f24-10a1-4fb4-bcde-37766a408b54][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 90][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0ab6c30b-82f8-460a-b3fe-2da43294bf7a][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 129][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:15150195-cd00-4b22-9dcd-cf2b79552115][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 77][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:10879f28-bf93-4fe6-b07b-54ada677234c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 69][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0f42617f-44ab-46b3-b489-3fd06327922b][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 95][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:11d23872-cbf6-4e7c-bc12-70f2a292345f][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 65][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0f294b1b-d071-4c93-9dc2-d890ee89df7b][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 43][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0ce24793-99d7-4f43-856a-d8ac0f264c8c][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 99][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:137f4774-3b0f-477a-8c8d-46d82e8ebf23][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 103][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:13e14184-3aa9-42c7-bd5e-4362a01385b8][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: [2022-11-16 09:06:39][Agent][Error][hx90][PID 645568][ThreadId 47][agent-version:0.9.7][exception:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][message:FatalError {fatalErrorMessage = "Worker failed: ExitFailure (-2)"}][task:0d0a0088-0c47-4717-a118-85ad31ca08e5][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:182:11] Exception in task
Nov 16 01:06:39 hx90 hercules-ci-agent[645568]: NixInfo {nixLibVersion = "2.9.2", nixPlatforms = ["i686-linux","x86_64-linux","x86_64-v1-linux","x86_64-v2-linux","x86_64-v3-linux"], nixSystemFeatures = ["benchmark","big-parallel","kvm","nixos-test"], nixSubstituters = [], nixTrustedPublicKeys = [], nixNarinfoCacheNegativeTTL = 0, nixNetrcFile = Just "/etc/nix/netrc"}
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Deactivated successfully.
Nov 16 01:06:39 hx90 systemd[1]: hercules-ci-agent.service: Consumed 42.574s CPU time, received 1006.9K IP traffic, sent 3.1M IP traffic.

Platform / Version

agent 0.9.7, nix 2.9.2
@roberth roberth added the bug Something isn't working label Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@roberth