Add updated fluentd chart remove old fluentd chart (#27)

### PR Description This PR replaces the old unmaintained fluentd helm chart See: helm/charts#21235 with an updated fluentd helm charm located here : https://github.com/fluent/helm-charts/tree/main/charts/fluentd. I've broken out changes into three separate commits: 1) Replaces old helm chart from stable with new chart 2) Updates version to phrg1 3) Add's support for nodeports (see: fluent/helm-charts#231)
powerhome · Nov 30, 2023 · f1c181f · f1c181f
1 parent e39bfa9
commit f1c181f
Show file tree

Hide file tree

Showing 29 changed files with 4,422 additions and 648 deletions.
diff --git a/charts/fluentd/.helmignore b/charts/fluentd/.helmignore
@@ -14,8 +14,10 @@
 *.swp
 *.bak
 *.tmp
+*.orig
 *~
 # Various IDEs
 .project
 .idea/
 *.tmproj
+.vscode/
diff --git a/charts/fluentd/Chart.yaml b/charts/fluentd/Chart.yaml
@@ -1,13 +1,16 @@
-apiVersion: v1
-description: DEPRECATED A Fluentd Elasticsearch Helm chart for Kubernetes.
-icon: https://raw.githubusercontent.com/fluent/fluentd-docs/master/public/logo/Fluentd_square.png
+apiVersion: v2
 name: fluentd
-version: 2.5.3-phrg1
-appVersion: v2.4.0
+description: A Helm chart for Kubernetes
+# type: application
+version: 0.5.0-phrg1
+appVersion: v1.16.2
+icon: https://www.fluentd.org/images/miscellany/fluentd-logo_2x.png
 home: https://www.fluentd.org/
 sources:
-- https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image
-- https://quay.io/repository/coreos/fluentd-kubernetes
-- https://github.com/coreos/fluentd-kubernetes-daemonset
-- https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
-deprecated: true
+  - https://github.com/fluent/fluentd/
+  - https://github.com/fluent/fluentd-kubernetes-daemonset
+maintainers:
+  - name: edsiper
+    email: [email protected]
+  - name: dioguerra
+    email: [email protected]
diff --git a/charts/fluentd/README.md b/charts/fluentd/README.md
@@ -1,121 +1,187 @@
-# ⚠️ Repo Archive Notice
+# Fluentd Helm Chart
 
-As of Nov 13, 2020, charts in this repo will no longer be updated.
-For more information, see the Helm Charts [Deprecation and Archive Notice](https://github.com/helm/charts#%EF%B8%8F-deprecation-and-archive-notice), and [Update](https://helm.sh/blog/charts-repo-deprecation/).
+[Fluentd](https://www.fluentd.org/) is an open source data collector for unified logging layer. Fluentd allows you to unify data collection and consumption for a better use and understanding of data.
 
-# fluentd
+## Installation
 
-[Fluentd](https://www.fluentd.org/) collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the Unified Logging Layer).
+To add the `fluent` helm repo, run:
 
-## DEPRECATION NOTICE
-
-This chart is deprecated and no longer supported.
+```sh
+helm repo add fluent https://fluent.github.io/helm-charts
+helm repo update
+```
 
-## TL;DR;
+To install a release named `fluentd`, run:
 
-```console
-$ helm install stable/fluentd
+```sh
+helm install fluentd fluent/fluentd
 ```
+## Upgrading
 
-## Introduction
+### To 0.4.0
 
-This chart bootstraps an fluentd deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.
+Although the services will deploy and generally work, version 0.4.0 introduces some changes that are considered _breaking changes_. To upgrade, you should do the following to avoid any potential conflicts or problems:
 
-## Installing the Chart
+- Add the `mountVarLogDirectory` and `mountDockerContainersDirectory` values and set them to the values you need; to follow the previous setup where these were mounted by default, set the values to `true`, e.g. `mountVarLogDirectory: true`
+- If you have the `varlog` mount point defined and enabled under both `volumes` and `volumeMounts`, set `mountVarLogDirectory` to true
+- If you have the `varlibdockercontainers` mount point defined and enabled under both `volumes` and `volumeMounts`, set `mountDockerContainersDirectory` to true
+- Remove the previous default volume and volume mount definitions - `etcfluentd-main`, `etcfluentd-config`, `varlog`, and `varlibdockercontainers`
+- Remove the `FLUENTD_CONF` entry from the `env:` list
 
-To install the chart with the release name `my-release`:
+## Chart Values
 
-```console
-$ helm install stable/fluentd --name my-release
+```sh
+helm show values fluent/fluentd
 ```
 
-The command deploys fluentd on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
+## Value Details
+
+### default-volumes
+
+The default configurations bellow are required for the fluentd pod to be able to read the hosts container logs. The second section is responsible for  allowing the user to load the "extra" configMaps either defined by the `fileConfigs` contained objects or, in addition, loaded externally and indicated by `configMapConfigs`.
+
+```yaml
+- name: varlog
+  hostPath:
+    path: /var/log
+- name: varlibdockercontainers
+  hostPath:
+    path: /var/lib/docker/containers
+---
+- name: etcfluentd-main
+  configMap:
+    name: fluentd-main
+    defaultMode: 0777
+- name: etcfluentd-config
+  configMap:
+    name: fluentd-config
+    defaultMode: 0777
+```
 
-## Uninstalling the Chart
+### default-volumeMounts
 
-To uninstall/delete the `my-release` deployment:
+The default configurations bellow are required for the fluentd pod to be able to read the hosts container logs. They should not be removed unless for some reason your container logs are accessible through a different path
 
-```console
-$ helm delete my-release
+```yaml
+- name: varlog
+  mountPath: /var/log
+- name: varlibdockercontainers
+  mountPath: /var/lib/docker/containers
+  readOnly: true
 ```
 
-The command removes all the Kubernetes components associated with the chart and deletes the release.
-
-## Autoscaling
-
-By enabling autoscaling the chart will use statefulset with hpa instead of deployment with PVC.
-Please be noted to [statefulset limitation](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations)
-The autoscaling is disabled by default for backward compatibility
-
-## Configuration
-
-The following table lists the configurable parameters of the fluentd chart and their default values.
-
-Parameter | Description | Default
---- | --- | ---
-`useStatefulSet` | Deploy as a StatefulSet regardless of whether autoscaling is enabled | `nil`
-`affinity` | node/pod affinities | `{}`
-`configMaps` | Fluentd configuration | See [values.yaml](values.yaml)
-`output.host` | output host | `elasticsearch-client.default.svc.cluster.local`
-`output.port` | output port | `9200`
-`output.scheme` | output scheme | `http`
-`output.sslVersion` | output ssl version | `TLSv1`
-`output.buffer_chunk_limit` | output buffer chunk limit | `2M`
-`output.buffer_queue_limit` | output buffer queue limit | `8`
-`deployment.labels` | Additional labels for pods | `{}`
-`image.pullPolicy` | Image pull policy | `IfNotPresent`
-`image.repository` | Image repository | `gcr.io/google-containers/fluentd-elasticsearch`
-`image.tag` | Image tag | `v2.4.0`
-`imagePullSecrets` | Specify image pull secrets | `nil` (does not add image pull secrets to deployed pods)
-`extraEnvVars` | Adds additional environment variables to the deployment (in yaml syntax) | `{}` See [values.yaml](values.yaml)
-`extraVolumeMounts` | Mount extra volumes (in yaml syntax) | `` See [values.yaml](values.yaml)
-`extraVolumes` | Extra volumes (in yaml syntax) | `` See [values.yaml](values.yaml)
-`ingress.enabled` | enable ingress | `false`
-`ingress.labels` | list of labels for the ingress rule | See [values.yaml](values.yaml)
-`ingress.annotations` | list of annotations for the ingress rule | `kubernetes.io/ingress.class: nginx` See [values.yaml](values.yaml)
-`ingress.hosts` | host definition for ingress | See [values.yaml](values.yaml)
-`ingress.tls` | tls rules for ingress | See [values.yaml](values.yaml)
-`nodeSelector` | node labels for pod assignment | `{}`
-`replicaCount` | desired number of pods | `1` ???
-`resources` | pod resource requests & limits | `{}`
-`plugins.enabled` | Enable Plugins Installation | `false`
-`plugins.pluginsList` | List of plugins to install | `[]`
-`rbac.create` | Specifies whether RBAC resources should be created | `true`
-`serviceAccount.create` | Specifies whether a service account should be created. | `true`
-`serviceAccount.name` | Name of the service account.
-`priorityClassName` | priorityClassName | `nil`
-`service.loadBalancerIP` | If `service.type` is `LoadBalancer` set custom IP load balancer IP address | `nil`
-`service.ports` | port definition for the service | See [values.yaml](values.yaml)
-`service.type` | type of service | `ClusterIP`
-`service.annotations` | list of annotations for the service | `{}`
-`tolerations` | List of node taints to tolerate | `[]`
-`persistence.enabled` | Enable buffer persistence | `false`
-`persistence.accessMode` | Access mode for buffer persistence | `ReadWriteOnce`
-`persistence.size` | Volume size for buffer persistence | `10Gi`
-`autoscaling.enabled` | Set this to `true` to enable autoscaling | `false`
-`autoscaling.minReplicas` | Set minimum number of replicas | `2`
-`autoscaling.maxReplicas` | Set maximum number of replicas | `5`
-`autoscaling.metrics` | metrics used for autoscaling | See [values.yaml](values.yaml)
-`terminationGracePeriodSeconds` | Optional duration in seconds the pod needs to terminate gracefully | `30`
-`metrics.enabled`                         | Set this to `true` to enable Prometheus metrics HTTP endpoint                         | `false`
-`metrics.service.port`                    | Prometheus metrics HTTP endpoint port                                                 | `24231`
-`metrics.serviceMonitor.enabled`          | Set this to `true` to create ServiceMonitor for Prometheus operator                   | `false`
-`metrics.serviceMonitor.additionalLabels` | Additional labels that can be used so ServiceMonitor will be discovered by Prometheus | `{}`
-`metrics.serviceMonitor.namespace`        | Optional namespace in which to create ServiceMonitor                                  | `nil`
-`metrics.serviceMonitor.interval`         | Scrape interval. If not set, the Prometheus default scrape interval is used           | `nil`
-`metrics.serviceMonitor.scrapeTimeout`    | Scrape timeout. If not set, the Prometheus default scrape timeout is used             | `nil`
-
-Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
-
-```console
-$ helm install stable/fluentd --name my-release \
-  --set=image.tag=v0.0.2,resources.limits.cpu=200m
+The section bellow is responsible for allowing the user to load the "extra" configMaps either defined by the `fileConfigs` contained objects or otherwise load externally and indicated by `configMapConfigs`.
+
+```yaml
+- name: etcfluentd-main
+  mountPath: /etc/fluent
+- name: etcfluentd-config
+  mountPath: /etc/fluent/config.d/
+  ```
+
+### default-fluentdConfig
+
+The `fileConfigs` section is organized by sources -> filters -> destinations. Flow control must be configured using fluentd routing with tags or labels to guarantee that the configurations are executed as intended. Alternatively you can use numeration on your files to control the configurations loading order.
+
+```yaml
+01_sources.conf: |-
+  <source>
+    @type tail
+    @id in_tail_container_logs
+    @label @KUBERNETES
+    path /var/log/containers/*.log
+    pos_file /var/log/fluentd-containers.log.pos
+    tag kubernetes.*
+    read_from_head true
+    <parse>
+      @type multi_format
+      <pattern>
+        format json
+        time_key time
+        time_type string
+        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
+        keep_time_key false
+      </pattern>
+      <pattern>
+        format regexp
+        expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
+        time_format '%Y-%m-%dT%H:%M:%S.%NZ'
+        keep_time_key false
+      </pattern>
+    </parse>
+    emit_unmatched_lines true
+  </source>
+
+02_filters.conf: |-
+  <label @KUBERNETES>
+    <match kubernetes.var.log.containers.fluentd**>
+      @type relabel
+      @label @FLUENT_LOG
+    </match>
+
+    # <match kubernetes.var.log.containers.**_kube-system_**>
+    #   @type null
+    #   @id ignore_kube_system_logs
+    # </match>
+
+    <filter kubernetes.**>
+      @type record_transformer
+      enable_ruby
+      <record>
+        hostname ${record["kubernetes"]["host"]}
+        raw ${record["log"]}
+      </record>
+      remove_keys $.kubernetes.host,log
+    </filter>
+
+    <match **>
+      @type relabel
+      @label @DISPATCH
+    </match>
+  </label>
+
+03_dispatch.conf: |-
+  <label @DISPATCH>
+    <filter **>
+      @type prometheus
+      <metric>
+        name fluentd_input_status_num_records_total
+        type counter
+        desc The total number of incoming records
+        <labels>
+          tag ${tag}
+          hostname ${hostname}
+        </labels>
+      </metric>
+    </filter>
+
+    <match **>
+      @type relabel
+      @label @OUTPUT
+    </match>
+  </label>
+
+04_outputs.conf: |-
+  <label @OUTPUT>
+    <match **>
+      @type elasticsearch
+      host "elasticsearch-master"
+      port 9200
+      path ""
+      user elastic
+      password changeme
+    </match>
+  </label>
 ```
 
-Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example,
+## Backwards Compatibility - v0.1.x
 
-```console
-$ helm install stable/fluentd --name my-release -f values.yaml
-```
+The old fluentd chart used the ENV variables and the default fluentd container definitions to set-up automatically many aspects of fluentd. It is still possible to trigger this behaviour by removing this charts current `.Values.env` configuration and replace by:
 
-> **Tip**: You can use the default [values.yaml](values.yaml)
+```yaml
+env:
+- name: FLUENT_ELASTICSEARCH_HOST
+  value: "elasticsearch-master"
+- name: FLUENT_ELASTICSEARCH_PORT
+  value: "9200"
+```