parse_vector VRL function?

[davidpellcb]

I'm looking for a way to parse the logs emitted by Vector itself. AFAICT there isn't a parse_* function for this.

Example log:

2024-07-11T17:56:52.024579Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "pods is forbidden: User \"system:serviceaccount:local-utility:vector-agent\" cannot list resource \"pods\" in API group \"\" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io \"vector-agent\" not found", reason: "Forbidden", code: 403 })

Is there a recommended approach for parsing Vector's logs? Just parse_grok or parse_regex? Seems like this may be hard to do. Compare this other WARN log:

2024-07-16T02:01:33.917564Z  WARN sink{component_kind="sink" component_id=proxy component_type=http}:request{request_id=1532040}: vector::sinks::util::retries: Retrying after response. reason=Http Status: 500 Internal Server Error internal_log_rate_limit=true

[jszwedko]

That's true, we could add this. Typically we recommend users configure Vector to log as JSON if they want to process the logs though, by setting the environment variable VECTOR_LOG_FORMAT=json.

[davidpellcb]

VECTOR_LOG_FORMAT=json

TIL, doh, sorry for missing that. Thanks so much! I will plan to use that and we could probably close this unless you think there's any use in being able to parse the non-JSON logs.

[jszwedko]

I think it's a reasonable enough request, though probably low-priority given the ability to emit structured JSON events.