[Fetch Migration] Monitoring script for Data Prepper

[kartg]

Description

This change adds a monitor.py Python script that can monitor the running Data Prepper pipeline and shut it down once the target document count has been reached (as reported by Data Prepper's documentsSuccess Prometheus metric) and an idle pipeline is detected.

• Category: New feature

---

Testing

Unit Tests

$ python -m coverage report --omit "*/tests/*"
Name                  Stmts   Miss  Cover
-----------------------------------------
endpoint_info.py          6      0   100%
index_operations.py      35      0   100%
main.py                 121      1    99%
monitor.py               52      0   100%
utils.py                 13      0   100%
-----------------------------------------
TOTAL                   227      1    99%

Integration Test

1. Setup step (see the Index Configuration Tool readme):

python index_configuration_tool/main.py -r <input_pipeline_yaml> <output_yaml_path>

2. Data Prepper container kickoff:

docker run -p 4900:4900 -v <output_yaml_path>:/usr/share/data-prepper/pipelines/pipelines.yaml opensearch-data-prepper:2.4.0-SNAPSHOT

3. After Data Prepper server has started, monitoring process kickoff:

python index_configuration_tool/monitor.py https://localhost:4900 <target_doc_count_from_report>

4. Verified that the monitoring process invoked the /shutdown endpoint once the target doc count was reached

---

Limitations

• Though the monitoring script correctly stops the Data Prepper server, the Docker container does not terminate. The current theory is that this is due to [BUG] Docker container does not stop running after Data Prepper shuts down data-prepper#3141
• Currently the endpoint auth information is hard-coded. The fix for this is tracked in JIRA task MIGRATIONS-1285
• The monitoring script only tracks Data Prepper metrics to identify shutdown state. It does not check the target cluster to validate that the target document count has been reached. Since index readers need to refresh to make the newly ingested documents searchable, there is often a delay (~5 minutes from my local testing) in the document counts reflecting in the _cat/indices count of the target cluster. Handling this scenario is tracked in JIRA task MIGRATIONS-1228

---

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #264 (361edef) into main (92405b8) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #264   +/-   ##
=========================================
  Coverage     62.14%   62.14%           
  Complexity      619      619           
=========================================
  Files            82       82           
  Lines          3141     3141           
  Branches        292      292           
=========================================
  Hits           1952     1952           
  Misses         1013     1013           
  Partials        176      176           

Flag	Coverage Δ
unittests	62.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[kartg]

The codecov diff hit seems inaccurate since it's looking at Java code which isn't a part of this change.

EDIT: Resolved after rebase on 8/21

[mikaylathompson]

Biggest issue is around error handling -- if any of the api calls fail, even transiently, the entire thing will crash.

[mikaylathompson]

How high is the risk that a doc fails to be written for whatever reason, and we're stuck here forever because doc_count is never >= target? Should we also have a timeout value that if doc_count doesn't change for X seconds/minutes, we give up?

[kartg]

I think that's a good callout. I'll capture this as a follow up item in JIRA - https://opensearch.atlassian.net/browse/MIGRATIONS-1303