A slightly better way I came up with for doing this is a variant of panicwrap (but has no dependencies to deal with). The container runtime execs Retina with some flag (for example, --supervise) that says it should supervise itself, on startup Retina then execs itself with the same arguments (after removing --supervise). The supervising parent retina process monitors STDERR from the child as panicwrap does, then forwards any panics it finds to App Insights with the appropriate metadata.

Care needs to be taken to properly handle the executable path (as well as injecting it into the args of the child), and also properly handling exit codes so that the orchestrator isn't confused by what we're doing.

This is a PoC that shows the technique described: https://github.com/timraymond/dontpanic/blob/master/main.go

Also, from @matmerr :

...we use the apiserver, podname and nodename dimensions... we'd need that information to be attached to the panic telemetry as well

@timraymond @huntergregory I've had a quick look and I've a question on the legacy operator.
This is the set up for it below:

@mereta yeah, that's... not great. It will probably work to just replace those os.Exit calls with return as a quick win. However, I'm not sure if it will return with the exit code of 1 to keep parity with here... some testing is needed. This is why it's nice to just return errors... it keeps the semantics pretty straightforward (but that ship has sailed at this point for this logic). Unfortunately, deferring os.Exit just has the effect of putting that invocation on the defer stack, so if there's anything behind it in the stack, it won't run.