[BUG] Broken Distributed Tracing in ASP.NET Core 3.1 Razor Pages with Elastic APM Agent ≥ 1.26.0

[Accudraft]

Environment:

• Elastic APM Agent Version 1.26.0 - 1.28.4
• Operating System Windows Server 2016 (IIS 10) / Win 10 (Visual Studio IIS Express)
• .NET Framework/Core name and version NET Core 3.1.31
• Application Target Framework netcoreapp3.1

Bug Description:
Distributed tracing is not functioning as expected in ASP.NET Core 3.1 Razor Pages projects when using the Elastic APM Agent version 1.28.4. While MVC and API projects seem unaffected, Razor Pages projects fail to generate end-to-end traces for requests involving other microservices or external services, even when those services utilize the same APM agent and server. Another issue I found to only affect Razor pages is the Transaction Name. The route could be something like /customer/details/abc-123, but the transaction name only comes through as /details. Again, this doesn’t seem to affect MVC or API projects.

Steps to Reproduce:

1. Create a new ASP.NET Core 3.1 Razor Pages application.
2. Install the Elastic APM Agent version 1.26.0 or greater
3. Implement functionality within the Razor Pages application to communicate with other microservices using HttpClient (ensure these microservices also have the Elastic APM agent installed), and external services such as Dropbox API, Twilio, or Mailgun.
4. Generate traffic to the Razor Pages application, triggering requests to both internal microservices and external services.
5. Observe the behavior in Elastic Observability, noting the lack of end-to-end traces, incomplete service map, and missing transaction details.

Expected Behavior:

• Complete end-to-end traces should be captured and visible in Elastic Observability, accurately reflecting the flow of requests across all involved services.
• The service map should display the relationships and dependencies between services based on the collected traces.
• Individual transactions should show associated traces to other services, providing detailed insights into the request flow.

Actual Behavior:

• End-to-end traces are broken or incomplete.
• The service map does not accurately represent service dependencies.
• Transactions lack associated traces, showing only generic "dependencies" with no detailed information.

Additional Information:

• The issue was resolved by rolling back the Elastic APM Agent version to 1.25.0.
• Upgrading to 1.25.3 still produced the expected behavior, but the issues reappeared with version 1.26.0.
• A review of the 1.26.0 release notes did not reveal any obvious changes that might have caused this behavior.
• I reviewed the Elastic APM documentation to understand the specific data requirements within transactions, spans, and traces for successful distributed tracing, but I found the information limited and unclear. Simply changing versions resolves the issue without any configuration changes at all.

[stevejgordon]

Thanks for raising this, @Accudraft. This is surprising and certainly something we need to investigate. I can't see any changes in the diff that would immediately explain it. I'll attempt to reproduce locally and identify why this isn't working as expected.

Can you gather trace level logs from two instrumented services that form a distributed trace? If so, could you email us at [email protected]? I can then provide a secure link to send the log files. Our logging should reveal why the distributed tracing may not work.

[stevejgordon]

I've successfully reproduced it and now have the logs I need to proceed. I have a theory that this could be due to the inclusion of http.request.cookies, which we updated after implementing the spec around cookie redaction. There have been some issues with mapping this data in older versions of APM server. I'll try to verify that and then look at the best solutions.

[Accudraft]

Great, I'm glad you were able to reproduce it. I wouldn't have been able to get you the required logs until next week otherwise. Thanks for looking into this.

[stevejgordon]

After digging a bit deeper, I see that this is likely due to the cookie handling changes we made in 1.26.0. Although strictly speaking, the agent is now doing the "right" thing per our apm specification. The issue in my repro is due to dropped transactions when cookies are prefixed with a period character, which fails to be indexed by the server. I'm discussing this with the server team to see what we can/should do about this. @Accudraft, would you be able to check if you use any cookies with a period character at the start, which may confirm my current theory?

[Accudraft]

Yes, all of my cookies start with ".AspNetCore"

Session, Identity, Antiforgery, etc...

Edit: I should note, these are the only cookies my applications use, and I believe they are just the default cookies created by the Microsoft.AspNetCore.Authentication.OpenIdConnect package we use.

[stevejgordon]

Thanks, @Accudraft. Yeah, those are some of the built-in cookies. Technically, it's possible to configure them with different names, but ultimately, any cookie starting with a . will be a problem. We've decided to partially revert the cookie change from 1.26.0 which I'm starting work on this week.

[stevejgordon]

Fixed by #2444