Skip to content

Commit

Permalink
Recommend debug logs while migrating from log to filestream input (#3…
Browse files Browse the repository at this point in the history
…6302)

All the migration-related logs are reported at the debug level. This
makes it hard to investigate if something goes wrong and the user has
not enabled the debug level. The migration guide is now updated to
mention the need of debug-level logs.
  • Loading branch information
rdner authored Aug 14, 2023
1 parent 241b459 commit 6151ae8
Showing 1 changed file with 10 additions and 4 deletions.
14 changes: 10 additions & 4 deletions filebeat/docs/howto/migrate-to-filestream.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -97,13 +97,18 @@ defined `log` input, we need to add `take_over: true` to each new `filestream`.
that the new `filestream` inputs will continue ingesting files from the same offset where the `log`
inputs stopped.

NOTE: It's recommended to enable debug-level logs for Filebeat in order to follow the migration process.
After the first run with `take_over: true` the setting can be removed.

WARNING: The `take over` mode is in beta.

IMPORTANT: If this parameter is not set, all the files will be re-ingested from the beginning
and this will lead to data duplication. Please, double-check that this parameter is set.

[source,yaml]
----
logging:
level: debug
filebeat.inputs:
- type: filestream
enabled: true
Expand Down Expand Up @@ -235,9 +240,10 @@ If for whatever reason you'd like to revert the configuration after running the
and return to old `log` inputs the files that were taken by `filestream` inputs, you need to do the following:

1. Stop Filebeat as soon as possible
2. Find your <<configuration-global-options,`registry.path/filebeat` directory>>
3. Find the created backup files, they have the `<timestamp>.bak` suffix. If you have multiple backups for the same file, choose the one with the more recent timestamp.
4. Replace the files with their backups, e.g. `log.json` should be replaced by `log.json-1674152412247684000.bak`
5. Run Filebeat with the old configuration (no `filestream` inputs with `take_over: true`).
2. Save its debug-level logs for further investigation
3. Find your <<configuration-global-options,`registry.path/filebeat` directory>>
4. Find the created backup files, they have the `<timestamp>.bak` suffix. If you have multiple backups for the same file, choose the one with the more recent timestamp.
5. Replace the files with their backups, e.g. `log.json` should be replaced by `log.json-1674152412247684000.bak`
6. Run Filebeat with the old configuration (no `filestream` inputs with `take_over: true`).

NOTE: Reverting to backups might cause some events to repeat, depends on the amount of time the new configuration was running.

0 comments on commit 6151ae8

Please sign in to comment.