unable to see Kubernetes pod logs in vmui

[slopesaka]

Is your question request related to a specific component?

Victoria-logs

Describe the question in detail

the documentation says to set vector.enabled to true to enable vector logs shipper but I see no logs in vmui when using this values.yaml file to install Victoria-logs-single using helm

vector:
  # -- Enable deployment of vector
  enabled: true
  podMonitor:
    enabled: true
  customConfig:
    sinks:
      console:
        type: console
        inputs: [ parser ]
        encoding:
          codec: json

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[AndrewChubatiuk]

hey @slopesaka
could you please leave only custom chart values in a description?
what is your kubernetes version?
how do you setup it?
also migrating this issue to helm-charts

[AndrewChubatiuk]

hey @slopesaka
do you still experience issues?

[trudzkimedo]

Hey.
Possibly related.

When used victoria-logs-single-0.8.1 / v0.41.0 with default config with vector set to enable on azure k8s, every json message that was generated by pod was not logged in but in it's place got:

missing _msg field; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field

It's same for today (victoria-logs-single-0.8.2 / v1.0.0) release.

this is easily to see when using istio ingress as it's logs are missing.

[trudzkimedo]

Quick update.
In my situation I misunderstood some concept here (especially that in fluent-bit it behaved bit differently)
My json message is parsed into log dict but since there was no message/msg/_msg field inside json payload default message (as above) is injected.
In fluent-bit the whole json payload was still left as _msg field.
Not sure if OP issue is same.

[AndrewChubatiuk]

hey @trudzkimedo
you can try to update processing pipeline to write a whole unparsed json into message field if neither msg nor message are found

vector:
  customConfig:
    transforms:
      parser:
        source: |
          message = parse_json(.message) ?? {}
          if is_nullish(message) || (is_null(message.msg) && is_null(message.message)) {
             message = .message
          }
          .log = message
          del(.message)

or add expected field names to VL-Msg-Field

vector:
  customConfig:
    sinks:
      vlogs:
        request:
          headers:
            VL-Msg-Field: message,msg,_msg,log.msg,log.message,log,<expected-value>

[trudzkimedo]

Hey,
I've removed
del(.message) and it's working as expected for me (it's first option in VL-Msg-Field), but yes, above example is even more cleaner.
I probably will use Yours one and add custom message (like "Message was processed and stored under log dictionary") as it will reflect true nature of what happened and it's more informative. Perhaps something You could consider to add to VL template if it will be causing more confiusion.

Anyway, for me it's working now fine, hopefully @slopesaka will solve her/his issue.

[slopesaka]

Hi @AndrewChubatiuk ,

Not sure I get what you mean by "leave only custom chart values in a description" ..do you mean the properties I have changed in the values.yaml file? if yes then

here are the changes I made

  ingress:
    enabled: true
    annotations:
       kubernetes.io/ingress.class: nginx
       cert-manager.io/cluster-issuer: cluster-vault-issuer
    hosts:
       - name: vmlogs.host-server.com
         path:
           - /select
           - /insert
         port: https
vector:
  # -- Enable deployment of vector
  enabled: true
  role: Agent
  dataDir: /vector-data-dir
  resources: {}
  podMonitor:
    enabled: true
  customConfig:
    sinks:
      console:
       type: console
       inputs: [parser]
       encoding:
         codec: json
      

I using Kubernetes version v1.27.16+k3s1
Made the setup by installing Helm chart

[AndrewChubatiuk]

hey @slopesaka
do you see any errors in vector logs?

[slopesaka]

Hi @AndrewChubatiuk,
Here are the snippet of logs I see in the logs

2024-11-20T09:27:15.653868Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:16.948234Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:17.248677Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:18.343329Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:19.426908Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:19.506242Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:20.273535Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:21.377880Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:22.357645Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:29.452427Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:32.775718Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:35.340291Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:35.927623Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-20T09:27:36.093239Z ERROR kube_client::client::builder: failed with error error trying to connect: Connection reset by peer (os error 104)
2024-11-20T09:27:36.093292Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchStartFailed(HyperError(hyper::Error(Connect, ConnectError { error: Error { code: ErrorCode(5), cause: Some(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) }, verify_result: X509VerifyResult { code: 0, error: "ok" } })))
2024-11-20T09:27:53.702436Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchError(ErrorResponse { status: "Failure", message: "too old resource version: 72551125 (72551185)", reason: "Expired", code: 410 })
2024-11-20T09:28:06.561115Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchError(ErrorResponse { status: "Failure", message: "too old resource version: 72551121 (72551185)", reason: "Expired", code: 410 })
2024-11-23T06:04:14.460754Z WARN kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-11-23T06:04:14.471804Z WARN kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-11-23T06:04:14.473379Z WARN kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-11-23T06:04:15.904145Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:15.904960Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:15.905474Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:15.916629Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:16.201643Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:16.296627Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:17.921326Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:18.137715Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:18.302447Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:18.657945Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:20.285760Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:20.298648Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:23.588932Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:25.726952Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:28.927378Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:33.197044Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:33.684467Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:37.328907Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:42.028247Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:04:52.389417Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(Api(ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }))
2024-11-23T06:05:12.004297Z ERROR kube_client::client::builder: failed with error error trying to connect: tcp connect error: Connection refused (os error 111)
2024-11-23T06:05:12.004348Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchStartFailed(HyperError(hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))))
2024-11-23T06:05:14.868896Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchError(ErrorResponse { status: "Failure", message: "too old resource version: 73029261 (73029263)", reason: "Expired", code: 410 })
2024-11-23T06:05:25.732416Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchError(ErrorResponse { status: "Failure", message: "too old resource version: 73029261 (73029263)", reason: "Expired", code: 410 })

[AndrewChubatiuk]

this is a setup specific issue, as vector is unable to get logs due to certificates issue
there are many kubernetes_logs source certificate related issues similar to this invector, like this one vectordotdev/vector#19224
unfortunately we don't have such setup to be able to reproduce this issue easily to help with a solution

[slopesaka]

hi @AndrewChubatiuk

ok ....so does this mean that the latest version of Victoria-logs helm chart which specifies use of vector instead of the previous fluentd was not tested hence "no such setup" ?

[AndrewChubatiuk]

it was tested on multiple k8s version, it was tested on k3s as well. regarding an issue you're having it's not related to vector or to victorialogs. It's configuration related issue as it's required to mount certificate inside vector container and specify a path to it in vector configuration, but due to a reason that I cannot reproduce your setup I cannot help you with exact parameters you need to change to make vector working for you

[slopesaka]

oh ok thanks for the clarification just needed to understand, perhaps the default chart values without my changes should work ?

[AndrewChubatiuk]

i'm not sure that with default values it should work, there's nothing special in your values, which could lead to issues you're having. how different is your k3s setup from a default one, which is provided by k3s install script? I've tested chart on default k3s of your version and it works without issues