Couldn't set log file full path as receiver ID

[QI-D]

I was trying to set a log file full path "/var/log/abc.log" as receiver ID in the config.ymal, then I got this following error:
[2022/03/03 22:57:39] [ warn] [output:stackdriver:stackdriver.0] error { "error": { "code": 400, "message": "Received unexpected value parsing name \"projects/qd-mar3/logs//var/log/abc.log\": abc.log. Expected the form projects/[PROJECT_ID]/logs/[ID]", "status": "INVALID_ARGUMENT" } }.
AWS CloudWatch Agent allows users to set file full path as the log stream name/ log name. I'm wondering if users can set log file full path as log name/ receiver ID with Ops Agent.

[yfuruyama]

Hi @QI-D I'm not sure if I understand your intention, but if your intention is sending the logs of /var/log/abc.log to Cloud Logging, you can write the following config.

logging:
  receivers:
    abc:
      type: files
      include_paths:
        - /var/log/abc.log
  service:
    pipelines:
      abc:
        receivers:
          - abc

[ridwanmsharif]

<img width="1232" alt="image" src="https://github.com/GoogleCloudPlatform/ops-agent/assets/18472685/18a15b54-a02a-45ad-88ea-6d8ae757f711">

I was able to reproduce this. We shouldn't allow special characters in the receiver ID as they make up the log name in Cloud Logging.

Cloud logging states that it must follow this rule:

[LOG_ID] must be URL-encoded within logName. Example: "organizations/1234567890/logs/cloudresourcemanager.googleapis.com%2Factivity".

[LOG_ID] must be less than 512 characters long and can only include the following characters: upper and lower case alphanumeric characters, forward-slash, underscore, hyphen, and period.

We need to have the same contract and report a config error if this is violated. I'll assign this to myself to do just that. See https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry for more info.

However, I do think you should use the record_log_file_path field in https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration#logging-receiver-structure. If set to true, then the path to the specific file from which the log record was obtained appears in the output log entry as the value of the agent.googleapis.com/log_file_path label. When using a wildcard, only the path of the file from which the record was obtained is recorded.

[quentinmit]

I think the bug here is that fluent-bit should be escaping the log name if necessary, and then this configuration should just work.

[github-actions]

This issue was marked stale due to lack of activity. It will be closed in 14 days.