RoadRunner silently crashes leaving defunct process

[egonbraun]

Hi, I have a backend application written in PHP 8.1 and Laravel v9.19.0. We were using NginX+FPM and now we are trying to migrate to NginX with Octane+RoadRunner but we are running into an issue that I need your help with.

So, we are using RoadRunner 2.10.6 and our controller function is running fine, however when the controller function returns the Illuminate\Http\Response object RoadRunner crashes and NginX returns a 500 error. No logs, no stack traces, nothing. I just see the 500 error in access logs.

I also noticed that RoadRunner processes are becoming defunct (zombie). When I run ps auxww I see dozens of processes like this one:

root     12701  0.0  0.0      0     0 ?        Z    14:47   0:00 [rr] <defunct>

This is how the RoadRunner process is being executed by Octane:

/usr/bin/rr -c /app/.rr.yaml -o version=2.7 -o http.address=127.0.0.1:8000 -o http.access_logs=true -o http.pool.debug=true -o http.pool.max_jobs=500 -o http.pool.num_workers=0 -o http.pool.supervisor.exec_ttl=300s -o logs.mode=development -o logs.level=info -o logs.output=stderr -o logs.err_output=stderr -o logs.encoding=console -o rpc.listen=tcp://:6001 -o server.command=/usr/bin/php8.1 /app/vendor/bin/roadrunner-worker -o server.user=www-data -o server.group=www-data -o server.relay=pipes serve

I could use some ideas on how to further troubleshoot this issue.

Thank you very much.

[rustatian]

Hey @egonbraun 👋🏻

-o http.pool.debug=true this option should be set to true only for the development. For some reason, processes exited before the wait4 syscall and became zombies in that case.

[rustatian]

Could you please remove that option?

[egonbraun]

Sure, doing it now :)

[egonbraun]

I believe that did it! Thanks, it is still failing but now I see why it is failing and I gotta do some digging on my end to find the root cause. Thank you very much @rustatian!

[rustatian]

And create a bug for this.

[rustatian]

I guess that on the 500 status from the worker, the octane kills it because RR doesn't interrupt the worker if it returns any status code, but without error.

[rustatian]

And this is not an ERROR in the RR terms. I'm not a PHP dev, but could you please check what is octane doing on the 500 returned from the worker?

[egonbraun]

Sure, I will do that. I get the feeling that it also might be something in our codebase this is why I first came to the Q&A section. I tried reproducing the issue using a fresh new Laravel+Octane installation but I couldn't.

I will see if I can create a project so you guys can reproduce it over there.

[egonbraun]

And this is not an ERROR in the RR terms. I'm not a PHP dev, but could you please check what is octane doing on the 500 returned from the worker?

Yep, I am also not a PHP Dev so I am trying to get someone to help me run a debug session to see where exactly this fails.

Thanks for your help @rustatian, I appreciate your prompt response!