[Create Environment] Adding ec module #1742

gurevichdmitry · 2023-12-26T06:46:19Z

Summary of your changes

This PR introduces a new internal EC module that deploys the basic ELK stack. Additionally, the ability to override the image has been implemented as part of this change.

Screenshot/Data

Related Issues

Checklist

I have added tests that prove my fix is effective or that my feature works
I have added the necessary README/documentation (if appropriate)

Introducing a new rule?

Generate rule metadata using this script
Add relevant unit tests
Generate relevant rule templates using this script, and open a PR in elastic/packages/cloud_security_posture

mergify · 2023-12-26T06:46:55Z

This pull request does not have a backport label. Could you fix it @gurevichdmitry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

… dg-upgrade-ec-dep

amirbenun

Nice!

amirbenun · 2024-01-04T10:13:20Z

.github/workflows/test-environment.yml

+          # Check if the version contains a commit hash, remove it
+          if [[ $stack_version =~ -[a-f0-9]+ ]]; then
+            cleaned_version=$(echo $stack_version | awk -F"-" '{print $1}')
+            echo "STACK_VERSION=$cleaned_version" >> $GITHUB_ENV


Remove it's declaration on line 115

Updates the declaration to support the working of the entire flow retroactively. The version with the hash is only required for Terraform to pin the version of the stack to be deployed.

amirbenun · 2024-01-04T10:23:48Z

deploy/cloud/modules/ec/main.tf

+  es_docker_image                  = lookup(var.docker_image, "elasticsearch", "")
+  es_docker_image_tag_override     = lookup(var.docker_image_tag_override, "elasticsearch", "")
+  kibana_docker_image              = lookup(var.docker_image, "kibana", "")
+  kibana_docker_image_tag_override = lookup(var.docker_image_tag_override, "kibana", "")
+  apm_docker_image                 = lookup(var.docker_image, "apm", "")
+  apm_docker_image_tag_override    = lookup(var.docker_image_tag_override, "apm", "")


Do we really have a usecase of overriding these?

The main use case is to support pinned BC versions. As described in this task, if we provide just the version (e.g., 8.12.0) and a new BC candidate is released, the stack will be updated with the newest version underneath. To prevent that, we should provide the exact version to be deployed. Unfortunately, the ec provider does not support the direct deployment of versions with a hash. To work around this, we need to override the docker_image for all stack components (ElasticSearch, Kibana, and Integrations Server). In the test environments' main.tf, we provide docker_image_tag_override, which is then passed to the ec module and used to deploy the custom image.

why do we even need the apm component?

The default manual deployment on the cloud includes the Integrations server component.
We are installing the Integrations server component, which was introduced in version 8.x and consists of APM and Fleet Server. Fleet is necessary to centrally manage all agents, integrations, and APM is part of Integrations server component.

amirbenun · 2024-01-04T10:27:35Z

deploy/cloud/modules/ec/output.tf

+  description = "The Elasticsearch password"
+}
+
+output "stack_version" {


Nice, we should start using it instead of the "requested" version.

amirbenun · 2024-01-04T10:28:10Z

deploy/cloud/modules/ec/variables.tf

+    "elasticsearch" = "",
+    "kibana"        = "",
+    "apm"           = "",


Do we need those?

Yes, each stack component may use a different image tag

amirbenun · 2024-01-04T10:29:54Z

deploy/test-environments/main.tf

  region        = var.ess_region
-  stack_version = var.stack_version
+  stack_version = local.cleaned_version


So we don't even deploy the requested version? Where do we use the commit sha?

In the case of a released or snapshot version, stack_version will not be modified and remains the same. However, in the case of a BC version, we need to remove the hash from stack_version. This is essential because the ec provider might fail with an unknown version if the hash is included. Additionally, we need to override docker_image_tag. Instead of creating complex logic to handle different scenarios, we always provide docker_image_override in all cases to ensure that our deployment deploys the pinned version.

amirbenun · 2024-01-04T10:30:57Z

.github/workflows/test-environment.yml

+          stack_version="${{ inputs.elk-stack-version }}"
+
+          # Check if the version contains a commit hash, remove it
+          if [[ $stack_version =~ -[a-f0-9]+ ]]; then


I see that we also clean that on the terraform? Where do we use the full version?

For Terraform, we are utilizing the TF_VAR_stack_version: ${{ inputs.elk-stack-version }} environment variable, which is passed as is. Full version is used to deploy pinned BC version.

amirbenun

A few more comments to your consideration

amirbenun · 2024-01-08T12:50:09Z

.github/workflows/test-environment.yml

@@ -152,6 +152,18 @@ jobs:
          echo "::add-mask::$ec_api_key"
          echo "TF_VAR_ec_api_key=$ec_api_key" >> $GITHUB_ENV

+      - name: Process BC version


I would consider getting the hash on a different optional parameter, PIN_VERSION for example.
This way we will avoid the cleaning code implemented both in bash and terraform, and the flow of using the pinned version will be more straight forward PIN_VERSION != "".

@amirbenun I have refactored the code to support a separate pin_version Terraform variable. This eliminates the need for cleanup version in Terraform code. I've updated the step action responsible for initializing and updating TF_vars and provided comments for clarity. Additionally, I have improved the documentation to make it more clear.

amirbenun · 2024-01-08T12:52:09Z

deploy/cloud/modules/ec/variables.tf

+
+# Docker image overrides
+
+variable "docker_image_tag_override" {


Please add comments on why those are needed especially when deploying BC versions.

oren-zohar · 2024-01-08T13:00:30Z

.github/workflows/test-environment.yml

+            -var="project=${{ github.actor }}" \
+            -var="owner=${{ github.actor }}"


oren-zohar · 2024-01-08T13:01:37Z

.github/workflows/upgrade-environment.yml

+      - name: Process BC version
+        id: clean-version
+        run: |
+          # Extract the stack version
+          stack_version="${{ inputs.target-elk-stack-version }}"
+
+          # Check if the version contains a commit hash, remove it
+          if [[ $stack_version =~ -[a-f0-9]+ ]]; then
+            cleaned_version=$(echo $stack_version | awk -F"-" '{print $1}')
+            echo "AGENT_VERSION=$cleaned_version" >> $GITHUB_OUTPUT
+          else
+            echo "AGENT_VERSION=$stack_version" >> $GITHUB_OUTPUT


if we reuse this part maybe we can export it to a script or to an action

The common part is just to retrieve cleaned_version. Afterwards, the handling is different. In this action, we are adding AGENT_VERSION to GITHUB_OUTPUT (different jobs), and in the test-environments step, we are updating env vars of the same job.

gurevichdmitry added 2 commits December 26, 2023 08:41

add ec module

4c75f1f

Merge branch 'main' into dg-upgrade-ec-dep

26e34e0

mergify bot assigned gurevichdmitry Dec 26, 2023

mergify bot added the backport-skip label Dec 26, 2023

gurevichdmitry added 4 commits December 26, 2023 08:54

update terraform format

c541fae

Merge branch 'dg-upgrade-ec-dep' of github.com:elastic/cloudbeat into…

771638a

… dg-upgrade-ec-dep

update ec module format

f54a9c2

update workflow

eb9cdd5

oren-zohar added the 8.13 candidate label Dec 26, 2023

gurevichdmitry added 8 commits December 26, 2023 10:57

updat tf fmt

f444990

update workflow and tf

0e71397

add tags

056d798

add ec api key input support

558e2c3

remove duplicate key

62a121e

update tf and workflow

0eea065

update workflow and docs

738d28d

Merge branch 'main' into dg-upgrade-ec-dep

7df2760

gurevichdmitry marked this pull request as ready for review December 28, 2023 07:58

gurevichdmitry requested a review from a team as a code owner December 28, 2023 07:58

gurevichdmitry added 4 commits January 3, 2024 17:37

Merge branch 'main' into dg-upgrade-ec-dep

cb66ca9

Merge branch 'main' into dg-upgrade-ec-dep

a0b08c5

update workflows

1a1e991

Merge branch 'main' into dg-upgrade-ec-dep

c19101b

oren-zohar requested a review from amirbenun January 4, 2024 08:51

amirbenun reviewed Jan 4, 2024

View reviewed changes

gurevichdmitry added 2 commits January 4, 2024 14:35

Merge branch 'main' into dg-upgrade-ec-dep

91ed41f

Merge branch 'main' into dg-upgrade-ec-dep

d4f0b38

amirbenun approved these changes Jan 8, 2024

View reviewed changes

oren-zohar reviewed Jan 8, 2024

View reviewed changes

gurevichdmitry added 5 commits January 8, 2024 17:28

updated documentation

63bdac7

Merge branch 'main' into dg-upgrade-ec-dep

b8556f4

fix review comments

a71888e

Merge branch 'main' into dg-upgrade-ec-dep

7de6cb4

update format

09f08a0

gurevichdmitry merged commit f3ad55e into main Jan 10, 2024
22 of 23 checks passed

gurevichdmitry deleted the dg-upgrade-ec-dep branch January 10, 2024 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Create Environment] Adding ec module #1742

[Create Environment] Adding ec module #1742

gurevichdmitry commented Dec 26, 2023

mergify bot commented Dec 26, 2023

amirbenun left a comment

amirbenun Jan 4, 2024

gurevichdmitry Jan 4, 2024

amirbenun Jan 4, 2024

gurevichdmitry Jan 4, 2024

oren-zohar Jan 8, 2024

gurevichdmitry Jan 8, 2024

amirbenun Jan 4, 2024

amirbenun Jan 4, 2024

gurevichdmitry Jan 4, 2024

amirbenun Jan 4, 2024

gurevichdmitry Jan 4, 2024

amirbenun Jan 4, 2024

gurevichdmitry Jan 4, 2024 •

edited

Loading

amirbenun left a comment

amirbenun Jan 8, 2024

gurevichdmitry Jan 9, 2024

amirbenun Jan 8, 2024

oren-zohar Jan 8, 2024

oren-zohar Jan 8, 2024

gurevichdmitry Jan 8, 2024


		# Docker image overrides

		variable "docker_image_tag_override" {

		-var="project=${{ github.actor }}" \
		-var="owner=${{ github.actor }}"

[Create Environment] Adding ec module #1742

[Create Environment] Adding ec module #1742

Conversation

gurevichdmitry commented Dec 26, 2023

Summary of your changes

Screenshot/Data

Related Issues

Checklist

Introducing a new rule?

mergify bot commented Dec 26, 2023

amirbenun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gurevichdmitry Jan 4, 2024 • edited Loading

Choose a reason for hiding this comment

amirbenun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gurevichdmitry Jan 4, 2024 •

edited

Loading