Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Improved and consistent logging for Filebeat & Filestream input status reporting #39775

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

[WIP] Improved and consistent logging for Filebeat & Filestream input status reporting #39775

wants to merge 5 commits into from
+108 −34

Conversation

belimawr
Copy link
Contributor

@belimawr belimawr commented May 30, 2024 •
edited
Loading

Proposed commit message

This commit improves Filebeat's input lifecycle logging as well as Filestream logs. All logs related to input starting/stopping now contain the input_id key, with the value coming from the defined input ID in the configuration file or sent by the Elastic-Agent.

The input_id is also propagated to all Filestream sub-components, and their logging is now enhanced with a consistent set of fields about the file being ingested, those fields are:

  • os_id: The file ID as identified by the OS, the pair [inode, device ID]
  • state_id: The file state ID, it is part of the ID used in the registry log and checkpoint related to the file. When using fingerprint file identity it contains the fingerprint, when using native file identity it contains the [inode, device ID] pair.
  • source_file The key used in the registry (in-memory, log and checkpoint). This contains the input type and input ID
  • path: The full file path from the file.

The input status reprorting when running under Elastic-Agent is also implemented for the Filestream input.

Full logs example from before and after are in the logs section

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Filebeat

filebeat.inputs:
  - type: filestream
    paths:
      - /tmp/logs/1*
    id: ID-Native
    close.on_state_change.inactive: 5s

  - type: filestream
    paths:
      - /tmp/logs/2*
    id: ID-Fingerprint
    file_identity.fingerprint: ~
    close.on_state_change.inactive: 5s
    prospector.scanner.fingerprint:
      enabled: true

path.home: /tmp/filebeat/

output:
  discard:
    enabled: true

queue.mem:
  flush.min_events: 0
  flush.timeout: 1s

logging:
  to_files: true
  level: debug
  selectors:
    - centralmgmt
    - file_watcher
    - filestream
    - input.filestream
    - metric_registry
    - prospector

Elastic-Agent

outputs:
  default:
    type: elasticsearch
    hosts:
      - http://localhost:9200
    username: "elastic"
    password: "changeme"
    preset: latency

inputs:
  - type: filestream
    id: Input-ID
    streams:
      - id: ID-Native
        paths:
          - /tmp/logs/1*
        close.on_state_change.inactive: 5s
      - id: ID-Fingerprint
        paths:
          - /tmp/logs/2*
        file_identity.fingerprint: ~
        close.on_state_change.inactive: 5s
        prospector.scanner.fingerprint:
          enabled: true

agent.logging:
  level: debug
  selectors:
    - "*"

Related issues

Use cases

Screenshots

Logs

The fields not relevant to this PR are redacted.

Single log example for file identity native and fingerprint:
Fingerprint:

{
  "@timestamp": "2024-05-31T16:54:45.340-0400",
  "log.level": "debug",
  "log.logger": "input.filestream",
  "message": "End of file reached: /tmp/logs/2.ndjson; Backoff now.",
  "input_id": "ID-Fingerprint",
  "os_id": "43116-34",
  "state_id": "fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d",
  "source_file": "filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d",
  "path": "/tmp/logs/2.ndjson"
}

Native:

{
  "@timestamp": "2024-05-31T16:54:45.338-0400",
  "log.level": "debug",
  "log.logger": "input.filestream",
  "message": "End of file reached: /tmp/logs/1.ndjson; Backoff now.",
  "input_id": "ID-Native",
  "os_id": "43127-34",
  "state_id": "native::43127-34",
  "source_file": "filestream::ID-Native::native::43127-34",
  "path": "/tmp/logs/1.ndjson"
}

Full logs from two Filestrams input running:

Logs before the change 

{"@timestamp":"2024-05-31T17:02:38.532-0400","log.level":"info","log.logger":null,"message":"Home path: [/home/tiago/devel/beats/x-pack/filebeat] Config path: [/home/tiago/devel/beats/x-pack/filebeat] Data path: [/home/tiago/devel/beats/x-pack/filebeat/data] Logs path: [/home/tiago/devel/beats/x-pack/filebeat/logs]","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.556-0400","log.level":"info","log.logger":null,"message":"Beat ID: df6ff8fd-80c9-4cec-833e-1f47d7cc8d3f","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.588-0400","log.level":"info","log.logger":"seccomp","message":"Syscall filter successfully installed","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.588-0400","log.level":"info","log.logger":"beat","message":"Beat info","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.588-0400","log.level":"info","log.logger":"beat","message":"Build info","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.588-0400","log.level":"info","log.logger":"beat","message":"Go runtime info","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.589-0400","log.level":"info","log.logger":"beat","message":"Host info","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.589-0400","log.level":"info","log.logger":"beat","message":"Process info","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.589-0400","log.level":"info","log.logger":null,"message":"Setup Beat: filebeat; Version: 8.15.0","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.591-0400","log.level":"info","log.logger":"discard","message":"Initialized discard output","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.591-0400","log.level":"info","log.logger":"publisher","message":"Beat name: millennium-falcon","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.591-0400","log.level":"info","log.logger":"modules","message":"Enabled modules/filesets: ","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.591-0400","log.level":"warn","log.logger":null,"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.591-0400","log.level":"info","log.logger":"monitoring","message":"Starting metrics logging every 30s","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.591-0400","log.level":"info","log.logger":null,"message":"filebeat start running.","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.598-0400","log.level":"info","log.logger":null,"message":"Finished loading transaction log file for '/home/tiago/devel/beats/x-pack/filebeat/data/registry/filebeat'. Active transaction id=0","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.605-0400","log.level":"info","log.logger":null,"message":"Finished loading transaction log file for '/home/tiago/devel/beats/x-pack/filebeat/data/registry/filebeat'. Active transaction id=0","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.605-0400","log.level":"warn","log.logger":null,"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.605-0400","log.level":"info","log.logger":"registrar","message":"States Loaded from registrar: 0","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.605-0400","log.level":"info","log.logger":"crawler","message":"Loading Inputs: 2","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.605-0400","log.level":"info","log.logger":"crawler","message":"starting input, keys present on the config: [filebeat.inputs.0.close.on_state_change.inactive filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"crawler","message":"Starting input (ID: 1843310979670678291)","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"crawler","message":"starting input, keys present on the config: [filebeat.inputs.1.close.on_state_change.inactive filebeat.inputs.1.id filebeat.inputs.1.paths.0 filebeat.inputs.1.prospector.scanner.fingerprint.enabled filebeat.inputs.1.type]","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' starting","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"metric_registry","message":"registering","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting prospector","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"crawler","message":"Starting input (ID: 15291939220401959268)","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"crawler","message":"Loading and starting Inputs completed. Enabled inputs: 2","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' starting","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"info","log.logger":"metric_registry","message":"registering","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting prospector","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"A new file /tmp/logs/1.ndjson has been found","id":"ID-Native","os_id":"43127-34","source_name":"native::43127-34","source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"A new file /tmp/logs/2.ndjson has been found","id":"ID-Fingerprint","os_id":"43116-34","source_name":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":null}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting harvester for file","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"newLogFileReader with config.MaxBytes:10485760","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting harvester for file","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:38.606-0400","log.level":"debug","log.logger":"input.filestream","message":"newLogFileReader with config.MaxBytes:10485760","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:38.610-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/1.ndjson; Backoff now.","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:38.611-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/2.ndjson; Backoff now.","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:40.619-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/2.ndjson; Backoff now.","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:40.619-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/1.ndjson; Backoff now.","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:44.623-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/2.ndjson; Backoff now.","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:44.623-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/1.ndjson; Backoff now.","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:48.607-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:48.607-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:48.607-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:48.607-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:52.629-0400","log.level":"info","log.logger":"input.filestream","message":"Reader was closed. Closing. Path='/tmp/logs/2.ndjson'","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:52.629-0400","log.level":"debug","log.logger":"input.filestream","message":"Stopped harvester for file","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:52.629-0400","log.level":"info","log.logger":"input.filestream","message":"Reader was closed. Closing. Path='/tmp/logs/1.ndjson'","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:52.629-0400","log.level":"debug","log.logger":"input.filestream","message":"Closing reader of filestream","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d"}
{"@timestamp":"2024-05-31T17:02:52.629-0400","log.level":"debug","log.logger":"input.filestream","message":"Stopped harvester for file","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:52.629-0400","log.level":"debug","log.logger":"input.filestream","message":"Closing reader of filestream","id":"ID-Native","os_id":null,"source_name":null,"source_file":"filestream::ID-Native::native::43127-34"}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":"service","message":"Received signal \"terminated\", stopping","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":null,"message":"Stopping filebeat","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":null,"message":"Stopping Crawler","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":null,"message":"Stopping 2 inputs","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":"crawler","message":"Stopping input: 15291939220401959268","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":"crawler","message":"Stopping input: 1843310979670678291","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"debug","log.logger":"input.filestream","message":"Prospector has stopped","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":"metric_registry","message":"unregistering","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (goroutine)","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (runner)","id":"ID-Fingerprint","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.427-0400","log.level":"debug","log.logger":"input.filestream","message":"Prospector has stopped","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":"metric_registry","message":"unregistering","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (goroutine)","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (runner)","id":"ID-Native","os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":null,"message":"Crawler stopped","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":null,"message":"Stopping filebeat","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":"registrar","message":"Stopping Registrar","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":"registrar","message":"Ending Registrar","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.428-0400","log.level":"info","log.logger":"registrar","message":"Registrar stopped","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.439-0400","log.level":"info","log.logger":"monitoring","message":"Total metrics","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.439-0400","log.level":"info","log.logger":"monitoring","message":"Uptime: 18.918011727s","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.439-0400","log.level":"info","log.logger":"monitoring","message":"Stopping metrics logging.","id":null,"os_id":null,"source_name":null,"source_file":null}
{"@timestamp":"2024-05-31T17:02:57.439-0400","log.level":"info","log.logger":null,"message":"filebeat stopped.","id":null,"os_id":null,"source_name":null,"source_file":null}

Logs after the change 

{"@timestamp":"2024-05-31T16:54:45.241-0400","log.level":"info","log.logger":null,"message":"Home path: [/home/tiago/devel/beats/x-pack/filebeat] Config path: [/home/tiago/devel/beats/x-pack/filebeat] Data path: [/home/tiago/devel/beats/x-pack/filebeat/data] Logs path: [/home/tiago/devel/beats/x-pack/filebeat/logs]","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.264-0400","log.level":"info","log.logger":null,"message":"Beat ID: eec2bc7a-0384-45a5-a7c2-0ce85c3d2df4","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.317-0400","log.level":"info","log.logger":"seccomp","message":"Syscall filter successfully installed","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.317-0400","log.level":"info","log.logger":"beat","message":"Beat info","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.317-0400","log.level":"info","log.logger":"beat","message":"Build info","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.317-0400","log.level":"info","log.logger":"beat","message":"Go runtime info","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.318-0400","log.level":"info","log.logger":"beat","message":"Host info","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.318-0400","log.level":"info","log.logger":"beat","message":"Process info","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.318-0400","log.level":"info","log.logger":null,"message":"Setup Beat: filebeat; Version: 8.15.0","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.320-0400","log.level":"info","log.logger":"discard","message":"Initialized discard output","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.320-0400","log.level":"info","log.logger":"publisher","message":"Beat name: millennium-falcon","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.320-0400","log.level":"info","log.logger":"modules","message":"Enabled modules/filesets: ","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.320-0400","log.level":"warn","log.logger":null,"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.320-0400","log.level":"info","log.logger":"monitoring","message":"Starting metrics logging every 30s","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.320-0400","log.level":"info","log.logger":null,"message":"filebeat start running.","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.328-0400","log.level":"info","log.logger":null,"message":"Finished loading transaction log file for '/home/tiago/devel/beats/x-pack/filebeat/data/registry/filebeat'. Active transaction id=0","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.334-0400","log.level":"info","log.logger":null,"message":"Finished loading transaction log file for '/home/tiago/devel/beats/x-pack/filebeat/data/registry/filebeat'. Active transaction id=0","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"warn","log.logger":null,"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"registrar","message":"States Loaded from registrar: 0","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"crawler","message":"Loading Inputs: 2","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"crawler","message":"starting input, keys present on the config: [filebeat.inputs.0.close.on_state_change.inactive filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"prospector","message":"file identity is set to native","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"crawler","message":"Starting input (ID: 1843310979670678291)","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"crawler","message":"starting input, keys present on the config: [filebeat.inputs.1.close.on_state_change.inactive filebeat.inputs.1.id filebeat.inputs.1.paths.0 filebeat.inputs.1.prospector.scanner.fingerprint.enabled filebeat.inputs.1.type]","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"prospector","message":"file identity is set to fingerprint","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' starting","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"crawler","message":"Starting input (ID: 15291939220401959268)","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"metric_registry","message":"registering","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"crawler","message":"Loading and starting Inputs completed. Enabled inputs: 2","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' starting","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"info","log.logger":"metric_registry","message":"registering","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting prospector","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting prospector","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"A new file /tmp/logs/1.ndjson has been found","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":null,"path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"A new file /tmp/logs/2.ndjson has been found","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":null,"path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting harvester for file","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"newLogFileReader with config.MaxBytes:10485760","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"Starting harvester for file","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.335-0400","log.level":"debug","log.logger":"input.filestream","message":"newLogFileReader with config.MaxBytes:10485760","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.338-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/1.ndjson; Backoff now.","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:45.340-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/2.ndjson; Backoff now.","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:47.339-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/1.ndjson; Backoff now.","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:47.348-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/2.ndjson; Backoff now.","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:51.343-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/1.ndjson; Backoff now.","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:51.349-0400","log.level":"debug","log.logger":"input.filestream","message":"End of file reached: /tmp/logs/2.ndjson; Backoff now.","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:55.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:55.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:55.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:55.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:54:59.363-0400","log.level":"info","log.logger":"input.filestream","message":"Reader was closed. Closing. Path='/tmp/logs/1.ndjson'","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:59.363-0400","log.level":"debug","log.logger":"input.filestream","message":"Stopped harvester for file","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:59.363-0400","log.level":"debug","log.logger":"input.filestream","message":"Closing reader of filestream","input_id":"ID-Native","os_id":"43127-34","state_id":"native::43127-34","source_file":"filestream::ID-Native::native::43127-34","path":"/tmp/logs/1.ndjson"}
{"@timestamp":"2024-05-31T16:54:59.363-0400","log.level":"info","log.logger":"input.filestream","message":"Reader was closed. Closing. Path='/tmp/logs/2.ndjson'","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:59.363-0400","log.level":"debug","log.logger":"input.filestream","message":"Stopped harvester for file","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:54:59.363-0400","log.level":"debug","log.logger":"input.filestream","message":"Closing reader of filestream","input_id":"ID-Fingerprint","os_id":"43116-34","state_id":"fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","source_file":"filestream::ID-Fingerprint::fingerprint::fea2922758a60439a9f7e683783331f0d07fce9fec955fc46f4e60e2f83bb36d","path":"/tmp/logs/2.ndjson"}
{"@timestamp":"2024-05-31T16:55:05.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:05.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:05.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:05.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:15.322-0400","log.level":"info","log.logger":"monitoring","message":"Non-zero metrics in the last 30s","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:15.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:15.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:15.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:15.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:25.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:25.336-0400","log.level":"debug","log.logger":"file_watcher","message":"Start next scan","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:25.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:25.336-0400","log.level":"debug","log.logger":"file_watcher","message":"File scan complete","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"service","message":"Received signal \"terminated\", stopping","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":null,"message":"Stopping filebeat","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"crawler","message":"Stopping Crawler","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"crawler","message":"Stopping 2 inputs","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"crawler","message":"Stopping input: 15291939220401959268","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"debug","log.logger":"input.filestream","message":"Prospector has stopped","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"metric_registry","message":"unregistering","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"crawler","message":"Stopping input: 1843310979670678291","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (goroutine)","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (runner)","input_id":"ID-Fingerprint","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"debug","log.logger":"input.filestream","message":"Prospector has stopped","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"metric_registry","message":"unregistering","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (goroutine)","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"input.filestream","message":"Input 'filestream' stopped (runner)","input_id":"ID-Native","os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"crawler","message":"Crawler stopped","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":null,"message":"Stopping filebeat","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"registrar","message":"Stopping Registrar","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"registrar","message":"Ending Registrar","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.907-0400","log.level":"info","log.logger":"registrar","message":"Registrar stopped","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.913-0400","log.level":"info","log.logger":"monitoring","message":"Total metrics","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.913-0400","log.level":"info","log.logger":"monitoring","message":"Uptime: 41.67822786s","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.913-0400","log.level":"info","log.logger":"monitoring","message":"Stopping metrics logging.","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}
{"@timestamp":"2024-05-31T16:55:26.913-0400","log.level":"info","log.logger":null,"message":"filebeat stopped.","input_id":null,"os_id":null,"state_id":null,"source_file":null,"path":null}

@belimawr
Make logging fields consistent for Filestream input
0073209 
This commit makes all components from the Filestream input to contain
its ID in the logs and makes some file identifier fields consistent
and present on all components, the fields are:
 - `os_id`: OS specific ID for the file, on Linux it  is
 <inode>-<device ID>
 - `state_id`: Identifier for the file state in the store. It depends
 on the file_identity set
 - `path`: The full path of the file being harvested
 - `source_file`: The full store identifier containing the input type,
 input ID and file_identifier
@belimawr belimawr added skip-ci Skip the build in the CI but linting backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels May 30, 2024
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 30, 2024
@belimawr belimawr force-pushed the improved-and-consistent-logging-for-filebeat branch from 4d18bac to a500131 Compare May 30, 2024 21:33
@belimawr belimawr changed the title [WIP] Improved and consistent logging for filebeat [WIP] Improved and consistent logging for Filebeat & Filestream input status reporting Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@belimawr belimawr

Labels
backport-skip Skip notification from the automated backport with mergify skip-ci Skip the build in the CI but linting Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@belimawr