Skip to content

Commit

Permalink
[8.14] [BK] Migrate es-forward (+add versions.json dependent triggeri…
Browse files Browse the repository at this point in the history
…ng) (elastic#184018) (elastic#187987)

# Backport

This will backport the following commits from `main` to `8.14`:
- [[BK] Migrate es-forward (+add versions.json dependent triggering)
(elastic#184018)](elastic#184018)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Alex
Szabo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-07-10T10:08:16Z","message":"[BK]
Migrate es-forward (+add versions.json dependent triggering)
(elastic#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines
that have a dependency on\r\nthe currently active branch set (managed
in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to
migrate over the `es-forward`
pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto
the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new
pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe
purpose of this new pipeline is to take the name of a
\"pipelineSet\"\r\nthat refers to a pipeline, and based on the
`versions.json` file, work\r\nout what are the branches on which the
referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger
ES forward compatibility tests`\r\n- a scheduled run
on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith
the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline
implementation for\r\n`kibana-trigger-version-dependent-jobs` works out
(looking at\r\n`versions.json`), that the `es-forward` set should
trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't
exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the
pipeline implementation uploads two trigger steps,
running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two
instances with\r\nthe relevant parameterization.\r\n\r\nSince the
trigger parameters are derived from the `versions.json` file,\r\nif we
move on and close `8.14`, and open up `8.16`, this will
follow,\r\nwithout having to update the pipeline resources or
schedules.\r\n\r\n## Changes\r\n- 2 pipelines
created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n
- [x] add kibana-es-forward.yml\r\n - implementation for
`trigger-version-dependent-jobs` added\r\n- branch configuration removed
from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot,
kibana-artifacts-trigger)\r\n - added a script for checking RREs
validity (moved a few files)\r\n\r\n## Verification\r\nI've used the
migration staging pipeline (*) to run
this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n-
Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit
triggered for 8.14 only (as
expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n
- Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it
generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't
exist, the upload step
failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result:
(success): it triggered jobs for all 3 open
branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result:
(success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note:
this migration staging pipeline will come in handy even after\r\nthe
migration, to stage newly created pipelines without creating
the\r\nresource
up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","backport:prev-minor","v8.15.0","v8.16.0"],"number":184018,"url":"https://github.com/elastic/kibana/pull/184018","mergeCommit":{"message":"[BK]
Migrate es-forward (+add versions.json dependent triggering)
(elastic#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines
that have a dependency on\r\nthe currently active branch set (managed
in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to
migrate over the `es-forward`
pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto
the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new
pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe
purpose of this new pipeline is to take the name of a
\"pipelineSet\"\r\nthat refers to a pipeline, and based on the
`versions.json` file, work\r\nout what are the branches on which the
referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger
ES forward compatibility tests`\r\n- a scheduled run
on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith
the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline
implementation for\r\n`kibana-trigger-version-dependent-jobs` works out
(looking at\r\n`versions.json`), that the `es-forward` set should
trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't
exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the
pipeline implementation uploads two trigger steps,
running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two
instances with\r\nthe relevant parameterization.\r\n\r\nSince the
trigger parameters are derived from the `versions.json` file,\r\nif we
move on and close `8.14`, and open up `8.16`, this will
follow,\r\nwithout having to update the pipeline resources or
schedules.\r\n\r\n## Changes\r\n- 2 pipelines
created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n
- [x] add kibana-es-forward.yml\r\n - implementation for
`trigger-version-dependent-jobs` added\r\n- branch configuration removed
from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot,
kibana-artifacts-trigger)\r\n - added a script for checking RREs
validity (moved a few files)\r\n\r\n## Verification\r\nI've used the
migration staging pipeline (*) to run
this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n-
Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit
triggered for 8.14 only (as
expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n
- Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it
generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't
exist, the upload step
failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result:
(success): it triggered jobs for all 3 open
branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result:
(success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note:
this migration staging pipeline will come in handy even after\r\nthe
migration, to stage newly created pipelines without creating
the\r\nresource
up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/184018","number":184018,"mergeCommit":{"message":"[BK]
Migrate es-forward (+add versions.json dependent triggering)
(elastic#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines
that have a dependency on\r\nthe currently active branch set (managed
in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to
migrate over the `es-forward`
pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto
the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new
pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe
purpose of this new pipeline is to take the name of a
\"pipelineSet\"\r\nthat refers to a pipeline, and based on the
`versions.json` file, work\r\nout what are the branches on which the
referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger
ES forward compatibility tests`\r\n- a scheduled run
on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith
the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline
implementation for\r\n`kibana-trigger-version-dependent-jobs` works out
(looking at\r\n`versions.json`), that the `es-forward` set should
trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't
exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the
pipeline implementation uploads two trigger steps,
running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two
instances with\r\nthe relevant parameterization.\r\n\r\nSince the
trigger parameters are derived from the `versions.json` file,\r\nif we
move on and close `8.14`, and open up `8.16`, this will
follow,\r\nwithout having to update the pipeline resources or
schedules.\r\n\r\n## Changes\r\n- 2 pipelines
created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n
- [x] add kibana-es-forward.yml\r\n - implementation for
`trigger-version-dependent-jobs` added\r\n- branch configuration removed
from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot,
kibana-artifacts-trigger)\r\n - added a script for checking RREs
validity (moved a few files)\r\n\r\n## Verification\r\nI've used the
migration staging pipeline (*) to run
this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n-
Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit
triggered for 8.14 only (as
expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n
- Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it
generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't
exist, the upload step
failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result:
(success): it triggered jobs for all 3 open
branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n
- Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result:
(success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note:
this migration staging pipeline will come in handy even after\r\nthe
migration, to stage newly created pipelines without creating
the\r\nresource
up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}}]}]
BACKPORT-->
  • Loading branch information
delanni authored Jul 15, 2024
1 parent 655f25e commit b16d258
Show file tree
Hide file tree
Showing 8 changed files with 367 additions and 1 deletion.
45 changes: 45 additions & 0 deletions .buildkite/pipeline-utils/utils.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

/* eslint-disable @typescript-eslint/no-unused-expressions */

import { expect } from 'chai';
import { getKibanaDir, getVersionsFile } from './utils';
import fs from 'fs';

// TODO: replace mocha with jest, and write tests that mock FS

describe('getKibanaDir', () => {
it('should return the kibana directory', () => {
const kibanaDir = getKibanaDir();

expect(kibanaDir).to.be.ok;
expect(fs.existsSync(kibanaDir)).to.be.true;
});
});

describe('getVersionsFile', () => {
it('should return the versions file', () => {
const versionsFile = getVersionsFile();

expect(versionsFile).to.be.ok;
expect(versionsFile.versions).to.be.an('array');
});

it('should correctly find prevMajor and prevMinor versions', () => {
const versionsFile = getVersionsFile();

expect(versionsFile.prevMajors).to.be.an('array');
expect(versionsFile.prevMajors.length).to.eql(1);
expect(versionsFile.prevMajors[0].branch).to.eql('7.17');

expect(versionsFile.prevMinors).to.be.an('array');
});

// TODO: write more tests with mocking...
});
42 changes: 41 additions & 1 deletion .buildkite/pipeline-utils/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
*/

import { execSync } from 'child_process';
import fs from 'fs';
import path from 'path';

const getKibanaDir = (() => {
let kibanaDir: string | undefined;
Expand All @@ -21,4 +23,42 @@ const getKibanaDir = (() => {
};
})();

export { getKibanaDir };
export interface Version {
branch: string;
version: string;
}
export interface VersionsFile {
versions: Array<
{
previousMajor?: boolean;
previousMinor?: boolean;
currentMajor?: boolean;
currentMinor?: boolean;
} & Version
>;
}
const getVersionsFile = (() => {
let versions: VersionsFile & {
prevMinors: Version[];
prevMajors: Version[];
current: Version;
};
const versionsFileName = 'versions.json';
try {
const versionsJSON = JSON.parse(
fs.readFileSync(path.join(getKibanaDir(), versionsFileName)).toString()
);
versions = {
versions: versionsJSON.versions,
prevMinors: versionsJSON.versions.filter((v: any) => v.previousMinor),
prevMajors: versionsJSON.versions.filter((v: any) => v.previousMajor),
current: versionsJSON.versions.find((v: any) => v.currentMajor && v.currentMinor),
};
} catch (error) {
throw new Error(`Failed to read ${versionsFileName}: ${error}`);
}

return () => versions;
})();

export { getKibanaDir, getVersionsFile };
6 changes: 6 additions & 0 deletions .buildkite/scripts/common/util.sh
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,9 @@ npm_install_global() {
download_artifact() {
retry 3 1 timeout 3m buildkite-agent artifact download "$@"
}

print_if_dry_run() {
if [[ "${DRY_RUN:-}" =~ ^(1|true)$ ]]; then
echo "DRY_RUN is enabled."
fi
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/bin/bash

set -euo pipefail

ts-node .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
/* eslint-disable @typescript-eslint/no-unused-expressions */

import { getVersionsFile } from '#pipeline-utils';
import { expect } from 'chai';

import {
getArtifactBuildTriggers,
getArtifactSnapshotPipelineTriggers,
getESForwardPipelineTriggers,
getArtifactStagingPipelineTriggers,
} from './pipeline';

const versionsFile = getVersionsFile();

describe('pipeline trigger combinations', () => {
it('should trigger the correct pipelines for "es-forward"', () => {
const esForwardTriggers = getESForwardPipelineTriggers();
// tests 7.17 against 8.x versions
const targets = versionsFile.versions.filter((v) => v.currentMajor === true);

expect(esForwardTriggers.length).to.eql(targets.length);

expect(esForwardTriggers.every((trigger) => trigger.build?.branch === '7.17')).to.be.true;

const targetedManifests = esForwardTriggers.map((t) => t.build?.env?.ES_SNAPSHOT_MANIFEST);
targets.forEach((t) =>
expect(targetedManifests).to.include(
`https://storage.googleapis.com/kibana-ci-es-snapshots-daily/${t.version}/manifest-latest-verified.json`
)
);
});

it('should trigger the correct pipelines for "artifacts-snapshot"', () => {
const snapshotTriggers = getArtifactSnapshotPipelineTriggers();
// triggers for all open branches
const branches = versionsFile.versions.map((v) => v.branch);

expect(snapshotTriggers.length).to.eql(branches.length);

branches.forEach((b) => {
expect(snapshotTriggers.some((trigger) => trigger.build?.branch === b)).to.be.true;
});
});

it('should trigger the correct pipelines for "artifacts-trigger"', () => {
const triggerTriggers = getArtifactBuildTriggers();
// all currentMajor+prevMinor branches
const branches = versionsFile.versions
.filter((v) => v.currentMajor === true && v.previousMinor === true)
.map((v) => v.branch);

expect(triggerTriggers.length).to.eql(branches.length);
branches.forEach((b) => {
expect(triggerTriggers.some((trigger) => trigger.build?.branch === b)).to.be.true;
});
});

it('should trigger the correct pipelines for "artifacts-staging"', () => {
const stagingTriggers = getArtifactStagingPipelineTriggers();
// all branches that are not currentMajor+currentMinor
const branches = versionsFile.versions
.filter((v) => !v.currentMajor || !v.currentMinor)
.map((v) => v.branch);

expect(stagingTriggers.length).to.eql(branches.length);
branches.forEach((b) => {
expect(stagingTriggers.some((trigger) => trigger.build?.branch === b)).to.be.true;
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
#!/usr/bin/env ts-node
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { getVersionsFile, BuildkiteTriggerStep } from '#pipeline-utils';

const pipelineSets = {
'es-forward': 'kibana-es-forward-compatibility-testing',
'artifacts-snapshot': 'kibana-artifacts-snapshot',
'artifacts-staging': 'kibana-artifacts-staging',
'artifacts-trigger': 'kibana-artifacts-trigger',
};

/**
* This pipeline is used to emit trigger steps onto different pipelines, based on dynamic parameters retrieved from the repository.
*/
async function main() {
const pipelineSetNames = Object.keys(pipelineSets);
const pipelineSetName: string | undefined = process.env.TRIGGER_PIPELINE_SET;
const pipelineSteps: BuildkiteTriggerStep[] = [];

if (!pipelineSetName) {
throw new Error(
`Env var TRIGGER_PIPELINE_SET is required, and can be one of: ${pipelineSetNames}`
);
} else if (!pipelineSetNames.includes(pipelineSetName)) {
throw new Error(
`Invalid value for TRIGGER_PIPELINE_SET (${pipelineSetName}), can be one of: ${pipelineSetNames}`
);
}

switch (pipelineSetName) {
case 'es-forward': {
pipelineSteps.push(...getESForwardPipelineTriggers());
break;
}
case 'artifacts-snapshot': {
pipelineSteps.push(...getArtifactSnapshotPipelineTriggers());
break;
}
case 'artifacts-staging': {
pipelineSteps.push(...getArtifactStagingPipelineTriggers());
break;
}
case 'artifacts-trigger': {
pipelineSteps.push(...getArtifactBuildTriggers());
break;
}
default: {
throw new Error(`Unknown pipeline set: ${pipelineSetName}`);
}
}

emitPipeline(pipelineSteps);
}

/**
* This pipeline is testing the forward compatibility of Kibana with different versions of Elasticsearch.
* Should be triggered for combinations of ([email protected] + [email protected] {current open branches on the same major})
*/
export function getESForwardPipelineTriggers(): BuildkiteTriggerStep[] {
const versions = getVersionsFile();
const kibanaPrevMajor = versions.prevMajors[0];
const targetESVersions = [versions.prevMinors, versions.current].flat();

return targetESVersions.map(({ version }) => {
return {
trigger: pipelineSets['es-forward'],
async: true,
label: `Triggering Kibana ${kibanaPrevMajor.version} + ES ${version} forward compatibility`,
build: {
message: process.env.MESSAGE || `ES forward-compatibility test for ES ${version}`,
branch: kibanaPrevMajor.branch,
commit: 'HEAD',
env: {
ES_SNAPSHOT_MANIFEST: `https://storage.googleapis.com/kibana-ci-es-snapshots-daily/${version}/manifest-latest-verified.json`,
DRY_RUN: process.env.DRY_RUN,
},
},
} as BuildkiteTriggerStep;
});
}

/**
* This pipeline creates Kibana artifact snapshots for all open branches.
* Should be triggered for all open branches in the versions.json: 7.x, 8.x
*/
export function getArtifactSnapshotPipelineTriggers() {
// Trigger for all named branches
const versions = getVersionsFile();
const targetVersions = [versions.prevMajors, versions.prevMinors, versions.current].flat();

return targetVersions.map(({ branch }) => {
return {
trigger: pipelineSets['artifacts-snapshot'],
async: true,
label: `Triggering snapshot artifact builds for ${branch}`,
build: {
message: process.env.MESSAGE || `Snapshot artifact build for ${branch}`,
branch,
commit: 'HEAD',
env: {
DRY_RUN: process.env.DRY_RUN,
},
},
} as BuildkiteTriggerStep;
});
}

/**
* This pipeline creates Kibana artifacts for branches that are not the current main.
* Should be triggered for all open branches in the versions.json: 7.x, 8.x, but not main.
*/
export function getArtifactStagingPipelineTriggers() {
// Trigger for all branches, that are not current minor+major
const versions = getVersionsFile();
const targetVersions = [versions.prevMajors, versions.prevMinors].flat();

return targetVersions.map(({ branch }) => {
return {
trigger: pipelineSets['artifacts-staging'],
async: true,
label: `Triggering staging artifact builds for ${branch}`,
build: {
message: process.env.MESSAGE || `Staging artifact build for ${branch}`,
branch,
commit: 'HEAD',
env: {
DRY_RUN: process.env.DRY_RUN,
},
},
} as BuildkiteTriggerStep;
});
}

/**
* This pipeline checks if there are any changes in the incorporated $BEATS_MANIFEST_LATEST_URL (beats version)
* and triggers a staging artifact build.
* Should be triggered only for the active minor versions that are not `main` and not `7.17`.
*
* TODO: we could basically do the check logic of .buildkite/scripts/steps/artifacts/trigger.sh in here, and remove kibana-artifacts-trigger
*/
export function getArtifactBuildTriggers() {
const versions = getVersionsFile();
const targetVersions = versions.prevMinors;

return targetVersions.map(
({ branch }) =>
({
trigger: pipelineSets['artifacts-trigger'],
async: true,
label: `Triggering artifact build for ${branch}`,
build: {
message: process.env.MESSAGE || `Artifact build for ${branch}`,
branch,
commit: 'HEAD',
env: {
DRY_RUN: process.env.DRY_RUN,
},
},
} as BuildkiteTriggerStep)
);
}

function emitPipeline(pipelineSteps: BuildkiteTriggerStep[]) {
console.log(JSON.stringify(pipelineSteps, null, 2));
}

if (require.main === module) {
main().catch((error) => {
console.error(error);
process.exit(1);
});
}
5 changes: 5 additions & 0 deletions .buildkite/scripts/steps/artifacts/cloud.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ set -euo pipefail
source "$(dirname "$0")/../../common/util.sh"
source .buildkite/scripts/steps/artifacts/env.sh

if [[ "${DRY_RUN:-}" =~ ^(true|1)$ ]]; then
echo "--- Nothing to do in DRY_RUN mode"
exit 0
fi

echo "--- Push docker image"
mkdir -p target

Expand Down
Loading

0 comments on commit b16d258

Please sign in to comment.