[index configuration tool] Change to parsing Data Prepper pipeline YAML as input #215
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kartik Ganesh <[email protected]>
…ML as input This change removes support for Logstash input files and the use of the Lark library. Instead, index_configuration_tool now accepts a Data Prepper pipeline YAML file as input. Signed-off-by: Kartik Ganesh <[email protected]>
kartg
requested review from
chelma,
gregschohn,
lewijacn,
mikaylathompson,
okhasawn and
sumobrian
as code owners
Codecov Report
@@ Coverage Diff @@
## main #215 +/- ##
=======================================
Coverage 91.51% 91.51%
=======================================
Files 25 25
Lines 1155 1155
Branches 122 122
=======================================
Hits 1057 1057
Misses 83 83
Partials 15 15 |
This allows easy updating of the stored pickle output file if/when the test pipeline YAML file is updated. This code is commented out for now since it will only be triggered manually. Signed-off-by: Kartik Ganesh <[email protected]>
kartg
force-pushed
the
ict-data-prepper-yaml
branch
from
ea32d3b
to
4e6d990
Compare
mikaylathompson
approved these changes
Jun 27, 2023
gregschohn
added a commit
to gregschohn/opensearch-migrations
that referenced
this pull request
Jun 28, 2023
* main: (57 commits) [Index configuration tool] Add support for insecure HTTPS endpoint (opensearch-project#218) Refactor inner classes from ExpiringTrafficStreamMap into separate classes. One other refactorings to simplify data encapsulation between the main map itself and each ExpiringKeyQueue. Move ExpiringTrafficStreamMap file to a new package to accommodate refactoring in the next commit. Make getOrCreateNodeMap() less pessimistic and add a TODO for work that needs to be done to remove (or at least describe) an obscure and rare race condition. [index configuration tool] Change to parsing Data Prepper pipeline YAML as input (opensearch-project#215) changing argument and variable names + updating description run on any .py changes within the top cluster_migration_core directory run lint on all .py changes Removing accidentally included kafka.properties file Migrations-1150 - Enhance Setting Kafka Properties for Kafka Puller Disabling python related workflows for non-python directories Using wildcards to add coverage report files instead of adding individual files removing finalized renaming workflow + no longer double executing tests Introduce Copilot Deployments (opensearch-project#201) Wildcard isn't supported - add each cov file manually run gradle build instead of assemble Update codecov version Add github workflow Bump aws-cdk-lib in /deployment/cdk/opensearch-service-migration (opensearch-project#206) ... Signed-off-by: Greg Schohn <[email protected]> # Conflicts: # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/Accumulation.java # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/ExpiringTrafficStreamMap.java # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/CapturedTrafficToHttpTransactionAccumulator
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change consists of 3 commits:
index_configuration_toolto now accept a Data Prepper pipeline YAML file as input.test_constants.pyto enable updating of the stored pickle output file if/when the test pipeline YAML file is updated. This code is commented out since it will only be triggered manually.Issues Resolved
resolves #164
Testing
All unit tests pass.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.