Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: move attributes.log to body field for filelog/monitored_pods #188

Merged
merged 2 commits into from
Oct 30, 2024

Conversation

joschi
Copy link
Contributor

@joschi joschi commented Oct 30, 2024

Refined log messages from Kubernetes pods/containers were only available in the log attribute but were never copied into the log record body.

This PR adds the necessary move operator to move the cleaned up log message from attributes.log to body.

The operator is also used in the recommended OpenTelemetry Collector configuration for Kubernetes:
https://github.com/open-telemetry/opentelemetry.io/blob/2024.09/content/en/docs/kubernetes/collector/components.md?plain=1#L328-L330

Refs open-telemetry/opentelemetry.io#3073

@joschi joschi self-assigned this Oct 30, 2024
Copy link
Member

@mmanciop mmanciop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joschi I remember this used to work, but it might be an oversight. Can you share a before/after?

@joschi
Copy link
Contributor Author

joschi commented Oct 30, 2024

Can you share a before/after?

@mmanciop Good that you've mentioned it I just found another discrepancy in the regular expressions for parser-crio and parser-containerd in the template. 😄

If you mean a before/after view of the produced logs:

Before:

{
  "resourceLogs": [
    {
      "schemaUrl": "https://opentelemetry.io/schemas/1.6.1",
      "scopeLogs": [
        {
          "logRecords": [
            {
              "attributes": [
                {
                  "key": "log",
                  "value": {
                    "stringValue": " Starting Foobar component ..."
                  }
                },
                {
                  "key": "log.file.path",
                  "value": {
                    "stringValue": "/var/log/pods/foobar-c6694c779-l42h8_6e8e8909-88ad-4cf9-bd45-9babedea17e9/foobar/0.log"
                  }
                },
                {
                  "key": "log.file.record_number",
                  "value": {
                    "intValue": "1"
                  }
                },
                {
                  "key": "log.iostream",
                  "value": {
                    "stringValue": "stdout"
                  }
                },
                {
                  "key": "logtag",
                  "value": {
                    "stringValue": "F"
                  }
                }
              ],
              "body": {
                "stringValue": "2024-10-30T13:15:51.659623514Z stdout F Starting Foobar component ..."
              },
              "observedTimeUnixNano": "1730294151744298369",
              "severityNumber": 0,
              "timeUnixNano": "1730294151659623514"
            }
          ]
        }
      ]
    }
  ]
}

After (also pay attention that the leading whitespace in the log message has been removed):

{
  "resourceLogs": [
    {
      "schemaUrl": "https://opentelemetry.io/schemas/1.6.1",
      "scopeLogs": [
        {
          "logRecords": [
            {
              "attributes": [
                {
                  "key": "log.file.path",
                  "value": {
                    "stringValue": "/var/log/pods/foobar-c6694c779-l42h8_6e8e8909-88ad-4cf9-bd45-9babedea17e9/foobar/0.log"
                  }
                },
                {
                  "key": "log.file.record_number",
                  "value": {
                    "intValue": "1"
                  }
                },
                {
                  "key": "log.iostream",
                  "value": {
                    "stringValue": "stdout"
                  }
                },
                {
                  "key": "logtag",
                  "value": {
                    "stringValue": "F"
                  }
                }
              ],
              "body": {
                "stringValue": "Starting Foobar component ..."
              },
              "observedTimeUnixNano": "1730294151744298369",
              "severityNumber": 0,
              "timeUnixNano": "1730294151659623514"
            }
          ]
        }
      ]
    }
  ]
}

Copy link

sonarcloud bot commented Oct 30, 2024

@joschi joschi requested a review from mmanciop October 30, 2024 15:04
@basti1302 basti1302 merged commit 5217e88 into main Oct 30, 2024
9 checks passed
@basti1302 basti1302 deleted the fix-otel-collector-logs-body branch October 30, 2024 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants