Sending logs to Graylog #145
Comments
|
I also want to add this daemonset as a default log collector for Graylog helm chart I am developing - https://github.com/imubit/graylog-helm-chart |
|
Hi! I noticed that the master nodes are the ones with that error message, if you look at the other pods logs, you will notice that they have successful outputs. UDP is the default protocol set as default: https://github.com/bodhi-space/fluent-plugin-gelf-hs/blob/master/lib/fluent/plugin/out_gelf.rb I was able to change it successfully to TCP by adding an attribute: <match **>
@type gelf
@id out_graylog
log_level info
include_tag_key true
host "#{ENV['FLUENT_GRAYLOG_HOST']}"
port "#{ENV['FLUENT_GRAYLOG_PORT']}"
protocol "#{ENV['FLUENT_GRAYLOG_PROTOCOL']}"
buffer_chunk_limit 4096K
buffer_queue_limit 512
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 8
</match>However, even when fluentd is able to generate the connection (graylog shows 8 connections which is the amount of pods I have set in the daemonset), no inputs show up. I created a TCP GELF input. Is TCP supported? |
|
Also, increase the resource limit to at least 400! Otherwise it will be getting killed every-time without you noticing. |
|
Update, TCP is working properly, I just had to change the attribute shared and switch the GELF input in graylog to TCP. My RBAC version: ---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: yourdockerpath/your:image
env:
- name: FLUENT_GRAYLOG_HOST
value: "tcp.log.com"
- name: FLUENT_GRAYLOG_PORT
value: "12201"
- name: FLUENT_GRAYLOG_PROTOCOL
value: "tcp"
resources:
limits:
# ===========
# Less memory leads to child process problems.
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
securityContext:
privileged: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
|
|
Encountered the same problem, but on slave nodes too. Error is followed by: |
|
I think there is a problem with encoding, some values from the payload sent to graylog have â which is not being able to be converted to UTF-8 when JSON.parse is trying to do its thing. Check your graylog, that pod should be sending logs, as far as I see, it only doesnt send a payload when it fails to convert. What I do not know is if this is being worked on. |
|
having the same issue: |
|
Great ! Thanks @miguelortize Here is my fluentd-kubernetes-daemonset. I forked the official https://github.com/fluent/fluentd-kubernetes-daemonset and made some changes:
|
|
To clarify what @repeatedly sent, I managed to get rid of the conversion error by making a custom build and installing an extra plugin:
I then edited the fluent.conf to convert all incoming messages to UTF-8: |
|
i am facing a issue for graylog integration, ,this is my k8s pod logs. fluent.conf: | |
|
I'm running into this error as well but only on nodes that are running tiller pods. I've tried the approaches suggested in FAQ but nothing helps. Error messages in fluentd logs: The buffer content is kept in the following gist |
|
@harshal-shah It looks like encoding conversion problem. I made custom image and added plugin to fix it as suggested by @bjaworski3 above. My repo is here: https://github.com/dekses/fluentd-kubernetes-daemonset |
|
@miguelortize Can you explain why you needed to add privileged here? Other examples doesn't use this. Just curious. ...
securityContext:
privileged: true |
|
Hello all, It will keep like that and restart service in a infinity loop. Tks in advance, |
|
@thodquach use I'll submit my graylog working example to this repo very soon. |
Tks @shinebayar-g , Hope to see you soon :D |
|
Submitted PR #342 @thodquach |
Tks so much @shinebayar-g =))) ( BTW, the icon Tyrannosaurus Rex looks funny ) |
|
@miguelortize any chance to get this working with TLS? |
|
@huegelc Create follow-up issue regarding TLS support: #355. I'll be working on it this week, can pull you in for reviews as well if you like 👍 |
There is no example of Graylog DaemonSet, I tried to create my own by modifying elasticsearch example. But I receive the following error:
[out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog"I also noticed that
protocolis not configured in fluentd.conf, so I tested UDP and TCP with the same result.fluentd-daemonset-graylog.yamlI created:Any idea?
Thanks
The text was updated successfully, but these errors were encountered: