[Epic] Application annotations for Cryostat Agent integrations

[andrewazores]

Describe the feature

There should be some defined annotation prefix (like io.cryostat/) and keys like inject-agent, inject-tls-cert. Users should be able to add these annotations to their applications (on the Deployment or Pod) within any Namespace associated with a Cryostat instance. When the Operator observes that an application has such an annotation, an integration feature should be enabled:

• Cryostat Agent injection. The Operator mounts a volume to the annotated application, containing the Cryostat Agent JAR.
  • if the Operator can detect the base image or framework that the application is using, and if we determine some safe method to do so (maybe with another optional annotation), the Operator will also modify the application's Deployment to automatically instrument the application with the Agent JAR, ie. by setting the -javaagent JVM flag as well as setting environment variables to point the Agent at the correct Cryostat instance
• TLS cert injection. This would be used primarily for [Story] Separate TLS keystore/truststore from host application cryostat-agent#138 . If the user has requested Cryostat Agent injection to their application, they can use this second annotation to also request a TLS cert. This requires cert-manager integration. The Operator uses cert-manager to generate a TLS cert and places that into a ConfigMap or Secret and mounts that to the application. The Operator can also automatically add configuration to the application so that the injected Agent instance uses this new cert for its webserver.
  • the same TLS cert ConfigMap/Secret should also be mounted to the associated Cryostat server instance and included in its TLS truststore
• the associated Cryostat instance's TLS certificate should also be mounted to the application so that the Agent's HTTP client can trust the certificate

This way we can automate end-to-end TLS in both directions between Cryostats and their Agent instances, with the user doing as little work as possible to enable this security feature.

Tasks:

• [Story] Generate self-signed certificates for agents #936
• [Story] Operator should handle target namespace deletion #941
• [Story] Allow agents to authenticate with Cryostat using certificates #952
• [Story] Attempt to discover DNS name in Kubernetes cryostat-agent#525

[andrewazores]

Some more thoughts about the Agent injection/integration.

1. We want our Operator to be able to supply the Agent JAR to the application - the user should be able but not required to rebuild their application image containing our Agent.
2. Configuring the Agent should be as simple and automatic as possible - ideally, the user just flips one switch to opt in, which the Operator sees and takes care of the rest.

---

For point 1, packaging the Agent JAR version that corresponds to the Cryostat version which the Operator will deploy seems relatively simple. As discussed previously, the Operator can put this into some resource like a ConfigMap or Secret and mount it to the application Deployment.

If the Operator is already modifying the Deployment then it is trivial to add environment variables for configuring the Agent, to point it at the correct Cryostat instance, use the Pod IP for the callback URL, etc.

---

But how to actually attach the Agent to the application JVM? We have talked about trying to do some detection of the application's framework or base image and manipulating entry points, environment variables, etc. to add the -javaagent switch. This seems error-prone and conflict-prone.

We have also experimented with running an Attach Agent as a sidecar process and connecting to the main container, but this might be tricky to put together in practice and probably runs into a lot of security constraints.

What if we can do the equivalent of a kubectl exec on containers from within the Operator controller?

operator-framework/operator-sdk#4302

https://github.com/zalando/postgres-operator/blob/master/pkg%2Fcluster%2Fexec.go#L18-L44

operator-framework/operator-sdk#1561

If the Agent JAR is already available in the application container (via mounted ConfigMap), then there is an easy solution when combined with agent dynamic attach:

cryostatio/cryostat-agent#234

Just exec the Agent process in PID autodetect mode.

[andrewazores]

If we can do an equivalent to kubectl cp to copy the Agent JAR from the Operator into the application container, instead of mounting a ConfigMap, then this can even be done without causing an application restart. Fully dynamic and online.

[andrewazores]

https://stackoverflow.com/questions/51686986/how-to-copy-file-to-container-with-kubernetes-client-go

[andrewazores]

While chatting with Chris, another interesting idea came up. There could be another annotation that the user can add to their Deployment so that the Agent is configured such that it creates a Flight Recording on the attached JVM immediately. This would save the user the additional step of either going to the Cryostat UI to manually create a recording, or else creating an Automated Rule on their Cryostat instance to trigger on matching Agent instances. After adding this annotation to their application and waiting a few moments, the user could go to the Cryostat UI and find that there is already JFR data available for analysis.

I think this could be done by supplying the Agent instance with an agentmain argument for Smart Triggers:

https://github.com/cryostatio/cryostat-agent?tab=readme-ov-file#smart-triggers

JAVA_OPTIONS="-javaagent:-Dcryostat.agent.baseuri=http://cryostat.local!/deployments/app/cryostat-agent-${CRYOSTAT_AGENT_VERSION}.jar=[ProcessCpuLoad>0.2]~profile

If the trigger condition was simply true, then the trigger should fire immediately and start a recording.

Either this annotation would need to have a few configuration options, or we should be careful to pick some sane defaults (event template, todisk/maxsize, etc.)