hGetBufSome: illegal operation (handle is closed) in journalctl

[quinn-dougherty]

I'm running the agent as a service in my nixos config. I use the flake as a module to install 0.9.7.

Here is some of my journalctl -u hercules-ci-agent.service

Aug 29 22:11:44 fw hercules-ci-agent[2553]: [2022-08-29 14:11:44][Agent][Info][fw][PID 2553][ThreadId 518][agent-version:0.9.7][task:6469e163-e3d0-439e-9cbc-b24c743797a3][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:193:7] Starting task
Aug 29 22:11:45 fw hercules-ci-agent[2553]: NixInfo {nixLibVersion = "2.10.3", nixPlatforms = ["i686-linux","x86_64-linux","x86_64-v1-linux","x86_64-v2-linux","x86_64-v3-linux","x86_64-v4-linux"], nixSystemFeatures = ["benchmark","big-parallel","kvm","nixos-test"], nixSubstituters = ["https://quantified-uncertainty.cachix.org","https://miso-haskell.cachix.org","https://jsprojects.cachix.org","https://effective-altruism.cachix.org","https://cache.nixos.org/","https://iohk.cachix.org","https://hydra.iohk.io","https://quantified-uncertainty.cachix.org","https://effective-altruism.cachix.org","https://cache.nixos.org/"], nixTrustedPublicKeys = ["cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=","quantified-uncertainty.cachix.org-1:6Dmk4ZKzEPai4o0FovuUyeZ0lZcf3/4v+MTHWaxczzc=","miso-haskell.cachix.org-1:6N2DooyFlZOHUfJtAx1Q09H0P5XXYzoxxQYiwn6W1e8=","jsprojects.cachix.org-1:udVvXpLhh6N12arVzOlElJRv01jvWPtMavyqXywG69o=","effective-altruism.cachix.org-1:mybF0dFfu2Bxbc+LuNm7rb46MFY4FGG2GhwdHuQPWxU=","iohk.cachix.org-1:DpRUyj7h7V830dp/i6Nti+NEO2/nhblbov/8MW7Rqoo=","hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ=","quantified-uncertainty.cachix.org-1:6Dmk4ZKzEPai4o0FovuUyeZ0lZcf3/4v+MTHWaxczzc=","effective-altruism.cachix.org-1:mybF0dFfu2Bxbc+LuNm7rb46MFY4FGG2GhwdHuQPWxU="], nixNarinfoCacheNegativeTTL = 0, nixNetrcFile = Just "/etc/nix/netrc"}
Aug 29 22:11:45 fw hercules-ci-agent[2553]: hercules-ci-agent: fd:49: hGetBufSome: illegal operation (handle is closed)
Aug 29 22:11:48 fw hercules-ci-agent[2553]: [2022-08-29 14:11:48][Agent][Info][fw][PID 2553][ThreadId 518][agent-version:0.9.7][task:6469e163-e3d0-439e-9cbc-b24c743797a3][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:175:9] Completed task successfully

Critically, the only tasks being completed are evaluations. builds are stalled out completely. I don't think it's a cachix issue, as my caches are working fine when I do github action usage of them.

Something happened a few weeks ago that made all herc ci basically stall, whereas before it was working perfectly. I thought it was on the cachix end, but no longer think that.