[8.14] [BK] Migrate es-forward (+add versions.json dependent triggering) (#184018) #187987
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…astic#184018) ## Goal We'd like to introduce a way to run pipelines that have a dependency on the currently active branch set (managed in [versions.json](./versions.json)). With this, we'd like to migrate over the `es-forward` pipelines (currently: [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility), and [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility)) to the new buildkite infra. ## Summary This PR introduces a new pipeline: https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs (through [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)). The purpose of this new pipeline is to take the name of a "pipelineSet" that refers to a pipeline, and based on the `versions.json` file, work out what are the branches on which the referred pipeline should be triggered. ### Example: `Trigger ES forward compatibility tests` - a scheduled run on [kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs) with the env var `TRIGGER_PIPELINE_SET=es-forward` runs - the pipeline implementation for `kibana-trigger-version-dependent-jobs` works out (looking at `versions.json`), that the `es-forward` set should trigger https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to the PR) for (7.17+8.14) and (7.17+8.15) - the pipeline implementation uploads two trigger steps, running https://buildkite.com/elastic/kibana-es-forward in two instances with the relevant parameterization. Since the trigger parameters are derived from the `versions.json` file, if we move on and close `8.14`, and open up `8.16`, this will follow, without having to update the pipeline resources or schedules. ## Changes - 2 pipelines created: [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml), [kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml) - [x] add kibana-es-forward.yml - implementation for `trigger-version-dependent-jobs` added - branch configuration removed from pipelines (kibana-artifacts-staging, kibana-artifacts-snapshot, kibana-artifacts-trigger) - added a script for checking RREs validity (moved a few files) ## Verification I've used the migration staging pipeline (*) to run this: - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130 - Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"` - Result: [(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806) it triggered for 8.14 only (as expected) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131 - Env: `TRIGGER_PIPELINE_SET="es-forward"` - Result: (success): it generated 2 trigger steps, but since the es-forward pipeline doesn't exist, the upload step failed - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132 - Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"` - Result: (success): it triggered jobs for all 3 open branches (main/8.14/7.17) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134 - Env: `TRIGGER_PIPELINE_SET="artifacts-staging"` - Result: (success): it triggered 8.14 / 7.14, but not for main (*note: this migration staging pipeline will come in handy even after the migration, to stage newly created pipelines without creating the resource up-front) (cherry picked from commit ea25099) # Conflicts: # .buildkite/pipeline-resource-definitions/_templates/_new_pipeline.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-snapshot.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-staging.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-trigger.yml # .buildkite/pipeline-resource-definitions/locations.yml # .buildkite/pipeline-resource-definitions/scripts/fix-location-collection.ts
1 task
|
This will need to wait for #188027 |
💚 Build Succeeded
Metrics [docs]
|
Ikuni17
approved these changes
Jul 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.14:Questions ?
Please refer to the Backport tool documentation