Refactor CDR and CIS workflows

[gurevichdmitry]

Summary of your changes

This PR introduces the capability to install the CDR infrastructure with or without the CIS infrastructure. This enhancement allows for more flexibility in setting up the infrastructure based on the specific needs of the deployment.

Key Changes:

• Added support for installing CDR independently or in combination with CIS.
• Adjusted configuration options to ensure seamless integration between CDR and CIS when both are enabled.

Screenshot/Data

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added the necessary README/documentation (if appropriate)

Successful Workflow Test Runs

• CDR integrations run
• CIS integration runs
  • 9.0.0-SNAPSHOT
  • 8.17.0-SNAPSHOT
  • 8.16.0 Release
• CIS + CDR integrations run
• Destroy environment run

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b dg-refactor-cloud-logs-wf upstream/dg-refactor-cloud-logs-wf
git merge upstream/main
git push upstream dg-refactor-cloud-logs-wf

[mergify]

This pull request does not have a backport label. Could you fix it @gurevichdmitry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
  NOTE: backport-v8.x has been added to help with the transition to the new branch 8.x.

[gurevichdmitry]

Workflow runs:

• CDR successful execution
• CIS successful execution
• CDR and CIS successful execution
• Destroy environment execution

[orouz]

it looks like each workflow - cdr infra or create environment can run either infra or both. personally i think a single workflow with 2 checkboxes would be simpler, but maybe i'm missing smth

[gurevichdmitry]

After refactoring the Create Environment workflow, we can provide a choice, such as a combo box with options to select CIS, CDR, or All. From the CDR perspective, we only need to specify whether we want to install CIS as well. In this case, I believe a checkbox would be preferable.

[orouz]

maybe a boolean for CDR and CIS is better? otherwise the string needs to be known to whoever runs this workflow

[gurevichdmitry]

This will be resolved by refactoring the test-environment workflow in this task.

[orouz]

consider splitting this workflow to multiple actions, like deploy-elk, deploy-cdr, deploy-cis and using whichever is required based on inputs.

i think doing so will make the workflow more approachable for future updates as it is getting a bit complex. it'll also remove the need of various wrapper script like manage_infrastructure.sh and set_cloud_env_params.sh

[gurevichdmitry]

This will be resolved by refactoring the test-environment workflow in this task. Regarding wrapping scripts around the Terraform code, I'm not sure I agree with that approach, as it would centralize all logic related to deployment. However, we can reconsider this when the new actions for CIS and CDR are created.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b dg-refactor-cloud-logs-wf upstream/dg-refactor-cloud-logs-wf
git merge upstream/main
git push upstream dg-refactor-cloud-logs-wf