APM Tracing in .NET Native AOT apps

[zlangner]

I'm using .NET8 in a custom image running on AWS Lambda. In its current state I have logs and APM traces going to datadog using Datadog.Trace v2.50.0. This has been working great so far.

In order to get faster run time, I'm exploring using Native AOT. The first and only hurdle I hit was using Datadog.Trace. After removing this package I was able to update the app to use Native AOT and the result is that it's about 10x faster. However, I do like the visibility that I get with APM so I'd like to have it back.

I did try to use Datadog.Trace.Trimming with Datadog.Trace but it didn't seem to matter. The error I got while publishing the app was:

Generating native code
/root/.nuget/packages/microsoft.identitymodel.tokens/6.35.0/lib/net6.0/Microsoft.IdentityModel.Tokens.dll : warning IL2104: Assembly 'Microsoft.IdentityModel.Tokens' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/src/MyProject.csproj]
/root/.nuget/packages/datadog.trace/2.50.0/lib/net6.0/Datadog.Trace.dll : warning IL2104: Assembly 'Datadog.Trace' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/src/MyProject.csproj]
EXEC : error : Failed to load assembly 'Microsoft.AspNetCore.Http.Abstractions' [/src/MyProject.csproj]

This app does not reference or use Microsoft.AspNetCore.Http.Abstractions so I suspect it's coming from Datadog.Trace.

Is there any plan to support Native AOT compiled dotnet apps in the future?

[kevingosse]

There are two distinct scenarios to support:

• Custom instrumentation, where you manually call the Datadog.Trace APIs to create spans
• Automatic instrumentation, where we dynamically rewrite relevant methods to automatically create spans

We are actively working on supporting automatic instrumentation on NativeAOT, but it is a huge undertaking, requiring to rewrite the core of our instrumentation engine. We might reach the status of closed beta by the end of the quarter (June), but I can't make any promise at this point.

As a stop-gap measure, I believe custom instrumentation should mostly work with NativeAOT, even though we don't actively support it. I tried publishing a simple app, and interestingly I only got the Microsoft.AspNetCore.Http.Abstractions error when Datadog.Trace.Trimming is added. Without it, I managed to publish the application just fine.
If you decide to go this (unsupported) way, make sure to never set DiagnosticSourceEnabled to true if you reconfigure the tracer. This path relies on emitting IL, so it will break on NativeAOT.

[zlangner]

Thank you for the quick reply! I should have time to try it again without Datadog.Trace.Trimming later today. This particular application doesn't do anything requiring automatic instrumentation so only having custom instrumentation should be fine.

Is there any thought of splitting those up into 2 packages? Something like Datadog.Trace.Core for custom instrumentation, and Datadog.Trace referencing that and building on it to add automatic instrumentation? If it were architected that way, use cases that only need/want custom instrumentation should be leaner and easier to manage.

[zlangner]

You're correct @kevingosse, by only using Datadog.Trace it does compile using NatvieAOT but the behavior is the same as not including it. I still get logs but there are no traces appearing in APM. Nothing from custom instrumentation nor automatic instrumentation.

[Dreamescaper]

Is it possible to enable built-in OTEL metrics with DataDog for NativeAOT?

[toupswork]

Hi, @zlangner.
I wanted to see if you were already aware that you have a few other build options that will get you pretty close to Native AOT.

The default build option is tiered compilation, but you have the option to also have your assemblies trimmed; perform much (but not all) of the JIT native compilation ahead of time so that when deployed, IL will be small; and/or single assembly deployment.

For example, Ready to Run, Profile Guided Optimization, and Crossgen2 can convert much of the IL to native at build time.

For a short-lived app, start-up time and memory footprint are important. .NET 8 gives you the ability to customize the GC, too, for example, limit the managed heap size and thus Working Set memory.

@kevingosse, do you know if there are DD limitations/incompatibilities with those?

Best, Tom

[kevinkemp]

Any updates on a timeline for this work? Is this the place I should go to check on status? @kevingosse