[8.15] [Data Forge] Add artificial delay feature (#187901) #188333
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Summary
This PR adds a new setting, `indexing.artificialIndexDelay`, to the
indexing configuration to control how much artificial delay to add to
the timestamps. This PR also adds a "final" ingest pipeline to each data
source along with injecting a new base `component_template` which
includes the `event.ingested` field.
The artificial delay is useful for testing transforms on data that has a
significant delays. It also allows us to test if we miss data when
syncing on the transforms using `event.ingested`.
- Installs default ingest pipeline to add event.ingested to each
document
- Adds final_pipeline to each install_index_template
- Inject base component_template to each index_template at install time
- Add artificial delay for "current" events, historical events are
ingested without delay.
- Change index math to produce monthly indices
### How to test:
Copy the following to `fake_logs.delayed.yaml`:
```YAML
---
elasticsearch:
installKibanaUser: false
kibana:
installAssets: true
host: "http://localhost:5601/kibana"
indexing:
dataset: "fake_logs"
eventsPerCycle: 100
artificialIndexDelay: 300000
schedule:
- template: "good"
start: "now-1h"
end: false
eventsPerCycle: 100
```
Then run `node x-pack/scripts/data_forge.js --config
fake_logs.delayed.yaml`. This should index an hour of data immediately,
then add a 300s delay when indexing in "real time". The logs will look
like:
```
info Starting index to http://localhost:9200 with a payload size of 10000 using 5 workers to index 100 events per cycle
info Installing index templates (fake_logs)
info Installing components for fake_logs (fake_logs_8.0.0_base,fake_logs_8.0.0_event,fake_logs_8.0.0_log,fake_logs_8.0.0_host,fake_logs_8.0.0_metricset)
info Installing index template (fake_logs)
info Indexing "good" events from 2024-07-09T16:23:36.803Z to indefinitely
info Delaying 100 by 300000ms
info Waiting 60000ms
info { took: 2418721239, latency: 541, indexed: 6000 } Indexing 6000 documents.
...
```
Then after `300s`, it will index another `100` documents every `60s`.
You can also inspect the delay per minute using the following ES|QL in
Discover:
```
FROM kbn-data-forge-fake_logs.fake_logs-* | eval diff=DATE_DIFF("seconds", @timestamp, event.ingested) | STATS delay=AVG(diff) by timestamp=BUCKET(@timestamp, 1 minute)
```
This should give you a chart that looks something like this:
<img width="1413" alt="image"
src="https://github.com/elastic/kibana/assets/41702/2f48cb85-a410-487e-8f3b-41311ff95186">
There should also be a 5 minute gap at the end in Discover:
<img width="1413" alt="image"
src="https://github.com/elastic/kibana/assets/41702/660acc87-6958-4ce9-a544-d66d56f805dd">
---------
Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 2fac5e8)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.15:Questions ?
Please refer to the Backport tool documentation