Service identification within J2EE/EAP containers

[nosnhojbob]

Along with modern microservices, we also have a mix of legacy systems that use J2EE/EAP containers, all in need of Distributed Trace. Often we need to utilize an auto-instrumentation/agent approach to provide trace data, but we are left with overly broad service.name for our installation. When the agent is applied to the entire container, the service.name is essentially the container itself, but when you have many EAR/WAR/EJB or even vendor-specific deployments, it is hard or impossible to distinguish app1.war from app2.war in trace data. Has anyone else had this need? I am hoping to solve or get some guidance at least in proper tagging/identification of these sub-modules.

I have looked into service semantic conventions, but these appear to be primarily intended for set-it-once-per-JVM use and doesn't provide the granularity I am looking for.

If there was an attribute similar to the following, it could help the downstream systems filter accurately and consistently for the sub-module of interest.

Attribute	Type	Description	Example	Required
service.module	string	A sub-module contained within a parent service.name.	app1.war, custom-service.tar	No

Having an additional attribute such as this would help unlock a lot of possibilities in downstream systems (visualization, alerting), giving them a consistent strategy to perform a click-down for that next level bit of information. Eg. you would continue to see an ambiguous bloom of latency in "j2ee-server-sample", but if you grouped the data by module, you could see which specific WAR within the server was experiencing a problem. That could get directed to a completely different team concerned with just that specific WAR.

[jkwatson]

Hi @nosnhojbob . I don't think there's anything stopping you from adding a custom attribute to your spans to track this. If you're looking to define a standard span attribute that would track this, I recommend opening an issue in the specification repository, as that's where the standard semantic attributes are defined.

[ghost]

Current approach (name + namespace, set via JVM attributes) is great for dockerized deployments (single JVM per service).
I believe it's a valid point (adding a standardised attribute for modules). Then it'll be more probable that backend providers will support it. There is also a coupled discussion - how to set such attributes, so that each deployment module (say - a WAR in Tomcat) gets a unique value with own spans? It would probably make sense for @nosnhojbob to describe real world scenarios that should be supported (servers and deployment modules). Then we can perhaps discuss possible solutions.

[kenfinnigan]

I wonder if this is something that needs work from an agent perspective? To enable the service.name separation?

[nosnhojbob]

First scenario:

• JBoss EAP 6.x, 7.x app server (named my-eap)
• Deployments include app1.war, app2.war, mdb1.jar (a servlet web app, a spring mvc app and a message-driven bean EJB deployment)
• No guarantee of classloader isolation between the three modules
• Use OpenTelemetry Java Agent as part of the JBoss standalone.sh script to add:
  • -javaagent:path/to/opentelemetry-javaagent-all.jar -Dotel.resource.attributes=service.name=my-eap
• Java Agent approach taken here due to inability to modify individual modules (or for simplicity and minimal work by app teams).

Expectation:

• Java Agent provides hooks to the configured SDK which determine the J2EE module dynamically and add decoration which provides the service.module attribute.

Here is a abbreviated potential Zipkin export of spans flowing through this EAP container to visualize the expected output.

[
  {
    "traceId": "traceId 1",
    "id": "spanId 1",
    "name": "servlet-span",
    "localEndpoint": {
      "serviceName": "my-eap"
    },
    "tags": {
      "service.module": "app1"
    }
  },
  {
    "traceId": "traceId 2",
    "id": "spanId 2",
    "name": "springmvc-span",
    "localEndpoint": {
      "serviceName": "my-eap"
    },
    "tags": {
      "service.module": "app2"
    }
  },
  {
    "traceId": "traceId 3",
    "id": "spanId 3",
    "name": "mdb-span",
    "localEndpoint": {
      "serviceName": "my-eap"
    },
    "tags": {
      "service.module": "mdb1"
    }
  }
]

[nosnhojbob]

All three traces are completely independent, but they flow through the same EAP server. Given the constraints of an agent on the command-line for the EAP container, each trace is associated to my-eap, but also tagged more specifically to the module within that EAP container. My hope would be that the tag/attribute name was formalized into a semantic convention, so that if a second, manual approach (where perhaps you had better control of classloader isolation), you could set these values yourself:

...
        .setTracerProvider(SdkTracerProvider.builder()
            .setResource(Resource.getDefault()
                .merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "my-eap", 
                    ResourceAttributes.SERVICE_NAMESPACE, "jboss-applications"),  // Group all of my JBoss containers
                    ResourceAttributes.SERVICE_MODULE, "app1")...