Loki output: set tenant header based on k8s label instead of static value

[atamgp]

Describe the solution you'd like
I want to define 1 default Loki ClusterOutput but not with a static tenant.
With this, tenants don't get to see the config (it's cluster level) and for 100 tenant's I don't need then to create 100 namespaced output's with the same loki endpoint and auth config with is unsecure and harder to maintain if something changes.

e.g.:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
 name: default
spec:
 loki:
   url: http://loki:3100
   tenant_from_label: "capsule.clastix.io/tenant"
   configure_kubernetes_labels: true
   buffer: ...

Additional context
We create multi tenant k8s clusters and use capsule to create Tenant's in the cluster.
It's not relevant to know how Capsule works internally, only that namespaces and pods have a certain label with their tenant id.

For logging, I need to extract this label to be forwarded as the tenant header (X-Scope-OrgId) to Loki.

I see in the code that something like this is available for kafka (headers_from_record in the crds):

              kafka:
                properties:
                  ack_timeout:
                    type: integer
...
                  headers_from_record:
                    additionalProperties:
                      type: string
                    type: object

More important: Fluentd also supports this:
https://docs.fluentd.org/output/http#headers_from_placeholders

For non-loki users it might be interesting to also add this feature (with a more generic name: headers_from_record) to the http output

[atamgp]

Update:

It seems logging-framework uses the loki output from fluentd.
fluentd does not yet support tenant_id_key which fluentbit does support:
https://docs.fluentbit.io/manual/v/1.8/pipeline/outputs/loki

I opened a ticket with fluend to also support this, then the logging-framework needs to pass through only?
fluent/fluentd#3992

[aslafy-z]

I believe this is already supported by the fluentd loki plugin, see https://grafana.com/docs/loki/latest/clients/fluentd/#tenant.

The tenant field also supports placeholders, so it can dynamically change based on tag and record fields. Each placeholder must be added as a buffer chunk key. The following is an example of setting the tenant based on a k8s pod label:

<match **>
  @type loki
  url "https://logs-prod-us-central1.grafana.net"
  tenant ${$.kubernetes.labels.tenant}
  # ...
  <buffer $.kubernetes.labels.tenant>
    @type memory
    flush_interval 5s
  </buffer>
</match>

Also, the associated logging-operator output field is already passed through to the fluentd configuration: https://github.com/banzaicloud/logging-operator/blob/42402497f2116f014f14ba867fdce13aba0bf171/pkg/sdk/logging/model/output/loki.go#L74

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

thanks @aslafy-z for clarifying this! @fekete-robert is there a good place in the docs to add this?

[fekete-robert]

I think we can do two things:

• either add this example to the documentation of the Loki output (this is generated from the source code https://github.com/kube-logging/logging-operator/blob/master/pkg/sdk/logging/model/output/loki.go - everything has to be added as code comments, so it's not very convenient and prone to formatting errors),
• or we can create a separate page in the docs repo (under https://kube-logging.github.io/docs/examples/ ) that describes the scenario and how to do it.

[pepov]

I would opt for the separate page, but first someone should actually try this out whether it works correctly or not. I closed this prematurely, sorry.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[ralgozino]

Hi, sorry for commenting on a closed issue, but I dedicated some time testing this, so I figure I'll share my findings for others coming to the issue. Not being familiar with fluentd's configuration syntax, getting this working wasn't straight forward to me.

Note

Tested with Logging Operator v4.5.6, but I guess it should be the same for newer versions.

Let's say you want to use the value of the tenant label to identify the Loki tenant the Pod's logs should be sent to.

You will need to add to your ClusterOutput or Output definition the tenant property and the buffer.tags property like this:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: external-loki
  namespace: logging
spec:
  loki:
    tenant: ${$.kubernetes.label.tenat}
    # .. the rest of your config, like your Loki URL, etc.
    buffer:
      tags:  tag,time,$.kubernetes.labels.tenant

Warning

Setting the buffer.tags value is very important, otherwise fluentd won't replace the placeholder ${$.kubernetes.label.tenat} with the label's value and you will get an error like this:

failed to write post to http://loki-gateway.loki-single.svc.cluster.local/loki/api/v1/push (400 Bad  Request tenant ID '${$.kubernetes.labels.tenant}' contains unsupported character '$'

Also, you need to keep at least tag in the tags, if you put only tags: $.kubernetes.labels.tenant it does not work.

Once that you added the needed properties and applied to the cluster, give it some time to the operator to refresh fluentd's config.

To verify that everything is working, you can then check the logs of fluentd and you should see some lines like these:

infra-fluentd-1 fluentd 2024-10-02 14:02:00 +0000 [debug]: #0 [clusterflow:logging:kubernetes:clusteroutput:logging:external-loki] Created new chunk chunk_id="6237ee1c2572a286b6b414671dae335d" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1727877720, tag="kubernetes.var.log.containers.counter_default_count-8b9e74cd8ba457482c99b795b17297988a07466ccbec7a573dde21e01857b41f.log", variables={:"$.kubernetes.labels.tenant"=>"counter"}, seq=0>

Note

See that the variables field has been detected and its value calculated. You may need to set fluentd to debug to see this message. To change fluentd to debug:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: <your stack's name>
spec:
  fluentd:
    logLevel: debug

and

infra-fluentd-1 fluentd 2024-10-02 14:02:13 +0000 [debug]: #0 [clusterflow:logging:kubernetes:clusteroutput:logging:external-loki] sending 4594 bytes to loki (tenant: "counter")

Note

See that the tenant has correctly been set to the label's value at the end of the line.