Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] The APM Agent does not control the length of the SQL query text it receives in OTEL attributes via ActivitySource #2450

Closed
OlegUfaev opened this issue Sep 19, 2024 · 2 comments · Fixed by #2461
Labels
bug Something isn't working community triage

Comments

@OlegUfaev
Copy link

APM Agent version

1.29.0

Environment

Operating system and version: Linux, Windows

.NET Framework/Core name and version : .NET 8

Application Target Framework(s): .NET 8

Describe the bug

In our application we are using Npgsql EF Core Provider and when we execute commands to the database - it creates an Activity via ActivitySource and add OTEL-attributes to it via tags:
https://github.com/npgsql/npgsql/blob/6990cceffbca2d2de4c5f12df32729bc78bbeafb/src/Npgsql/NpgsqlActivitySource.cs#L50

        var activity = Source.StartActivity(activityName, ActivityKind.Client);
        if (activity is not { IsAllDataRequested: true })
            return activity;

        activity.SetTag("db.statement", commandText); // This tag will contain FULL command text

        if (dbOperation != null)
            activity.SetTag("db.operation", dbOperation);
        if (dbSqlTable != null)
            activity.SetTag("db.sql.table", dbSqlTable);

        return activity;

APM Agent gets reacts to Activity creation/stop events in the ElasticActivityListener class and retrieves OTEL-attributes from the tags of the Activity instance as they are written there, without any modifications.

private static void UpdateOTelAttributes(Activity activity, OTel otel)
{
if (!activity.TagObjects.Any()) return;
otel.Attributes ??= new Dictionary<string, object>();
foreach (var (key, value) in activity.TagObjects)
otel.Attributes[key] = value;
}

Then when in PayloadSenderV2 class the event is serialized into JSON for transmission to APM Server, the OTEL attribute values also remain unchanged, no matter how long they are (below, the db.statement value has been manually trimmed to avoid display issues).

{
  "span": {
    "context": {
      "destination": {
        "service": {
          "name": "",
          "resource": "postgresql/postgres",
          "type": ""
        }
      },
      "service": {
        "target": {
          "type": "postgresql",
          "name": "postgres"
        }
      }
    },
    "duration": 703.9279,
    "id": "5b3ab303ff8f0921",
    "name": "postgres",
    "outcome": "unknown",
    "parent_id": "6fa9d132a67115e5",
    "links": [],
    "sample_rate": 1.0
    "subtype": "postgresql",
    "timestamp": 1726759475306925,
    "trace_id": "153bbc814bcd35a2f74603119a662a18",
    "transaction_id": "8cc7a2369faba86e",
    "type": "db",
    "otel": {
      "attributes": {
        "db.system": "postgresql",
        "db.connection_string": "Host=localhost;Port=5432;Database=postgres;Username=postgres",
        "db.user": "postgres",
        "db.name": "postgres",
        "db.statement": "select 0; select 1; select 2; ......  select 34998; select 34999;",
        "db.connection_id": 5686,
        "net.transport": "ip_tcp",
        "net.peer.ip": "::1",
        "net.peer.name": "localhost",
        "otel.status_code": "OK"
      },
      "spanKind": "Client"
    }
  }
}

Obviously, the data size of this event can easily exceed the allowed value, which is set on the APM Server side in the max_event_size parameter (defaults to 307200 bytes). When trying to transfer such event data to APM Server - the error event exceeded the allowed size will occur.

A long query text can be generated by EF Core during bulk insert/update operations, which is what we observe in our application.

To Reproduce

Steps to reproduce the behavior:

  1. Get source code: https://github.com/elastic/apm-agent-dotnet/tree/v1.29.0
  2. Open project WebApiExample
  3. In appsettings.json add required ElasticApm configuration for your APM Server instance (ServerUrl, SecretToken, etc. )
  4. Replace WeatherForecastController implementation (see code snippet below)
  5. Launch the project and open the link in your browser: https://localhost:64661/WeatherForecast
  6. Error event exceeded the allowed size will appear in the log
using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;

namespace WebApiExample.Controllers;

[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
	private static readonly string[] Summaries =
	[
		"Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
	];

	[HttpGet(Name = "GetWeatherForecast")]
	public IEnumerable<WeatherForecast> Get()
	{
		// Constuct Activity the same way:
		// https://github.com/npgsql/npgsql/blob/6990cceffbca2d2de4c5f12df32729bc78bbeafb/src/Npgsql/NpgsqlActivitySource.cs#L50C57-L50C58
		using var activity = new ActivitySource("Npgsql", "0.1.0").StartActivity("test", ActivityKind.Client);
		activity?.SetTag("db.statement", new string('1', 400_000));

		return Enumerable.Range(1, 5)
			.Select(index => new WeatherForecast
			{
				Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
				TemperatureC = Random.Shared.Next(-20, 55),
				Summary = Summaries[Random.Shared.Next(Summaries.Length)]
			})
			.ToArray();
	}
}

Expected behavior

Values of OTEL-attributes that by specification contain SQL command text (db.query.text, db.statement) should be truncated by the same rules, as it is done now for the Statement property in the Database class. Maybe you should apply such a policy to all OTEL-attributes.

public class Database
{
public const string TypeElasticsearch = "elasticsearch";
public const string TypeSql = "sql";
public string Instance { get; set; }
[MaxLength(10_000)]
public string Statement { get; set; }
public string Type { get; set; }
}

Actual behavior

The SQL query text that comes in the tags of the Activity instance is passed in OTEL attributes to APM Server as is, which results in an error event exceeded the allowed size when the query text is huge.

@OlegUfaev OlegUfaev added the bug Something isn't working label Sep 19, 2024
@OlegUfaev OlegUfaev changed the title [BUG] The APM Agent does not control the length of the SQL queries it receives in OTEL attributes via ActivitySource [BUG] The APM Agent does not control the length of the SQL query text it receives in OTEL attributes via ActivitySource Sep 19, 2024
@OlegUfaev
Copy link
Author

@Mpdreamz, please look into this bug

@Mpdreamz
Copy link
Member

Mpdreamz commented Oct 7, 2024

Opened #2461 to address this, thanks for bringing this to our attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants