-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Health report integration tests bootstrapper and initial tests implementation #16467
base: feature/health-report-api
Are you sure you want to change the base?
Health report integration tests bootstrapper and initial tests implementation #16467
Conversation
…cenario implementation.
5f7dd0d
to
ffaac9a
Compare
Quality Gate passedIssues Measures |
💚 Build Succeeded
cc @mashhurs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have tried to run a happy case and a sad case that changed expectation.status
from yellow to red. However, both give me YAML 'expectation' section matches the stats.
[14:05:35] logstash git:(pr-16467) ✗ python3 .buildkite/scripts/health-report-tests/main.py
Starting Logstash Health Report Integration test.
LS_VERSION is not specified, using main branch.
logstash-integration-failure_injector successfully installed.
Validating /Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/tests/slow-start.yaml scenario file.
YAML file validation successful!
Logstash is running with PID: 55550.
Testing the scenario: [{'status': 'red'}, {'symptom': 'The pipeline is degraded or at risk of becoming unhealthy; 1 area is impacted and 1 diagnosis is available.'}, {'diagnosis': [{'cause': 'pipeline is loading'}]}, {'impacts': [{'description': 'pipeline is loading'}, {'impact_areas': 'pipeline_execution'}]}, {'details': [{'run_state': 'LOADING'}]}]
Logstash health report: {'path': '/_health_report', 'status': 404, 'error': {'message': 'Not Found'}}
YAML 'expectation' section matches the stats.
[14:46:10] logstash git:(pr-16467) ✗ python3 .buildkite/scripts/health-report-tests/main.py
Starting Logstash Health Report Integration test.
LS_VERSION is not specified, using main branch.
logstash-integration-failure_injector successfully installed.
Validating /Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/tests/slow-start.yaml scenario file.
YAML file validation successful!
Logstash is running with PID: 57476.
Testing the scenario: [{'status': 'yellow'}, {'symptom': 'The pipeline is degraded or at risk of becoming unhealthy; 1 area is impacted and 1 diagnosis is available.'}, {'diagnosis': [{'cause': 'pipeline is loading'}]}, {'impacts': [{'description': 'pipeline is loading'}, {'impact_areas': 'pipeline_execution'}]}, {'details': [{'run_state': 'LOADING'}]}]
Logstash health report: {'path': '/_health_report', 'status': 404, 'error': {'message': 'Not Found'}}
YAML 'expectation' section matches the stats.
def __init__(self): | ||
pass | ||
|
||
def __meets_expectation(self, scenario_content: list) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def __meets_expectation(self, scenario_content: list) -> None: | |
def __is_expected(self, scenario_content: list) -> None: |
|
||
def on(self, scenario_name: str, scenario_content: list) -> None: | ||
print(f"Testing the scenario: {scenario_content}") | ||
if self.__meets_expectation(scenario_content) is False: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if self.__meets_expectation(scenario_content) is False: | |
if self.__is_expected(scenario_content) is False: |
- name: "Slow start pipeline" | ||
- config: | ||
- pipeline.id: slow-start-pp | ||
config.string: | | ||
input { heartbeat {} } | ||
filter { failure_injector { degrade_at => [register] } } | ||
output { stdout {} } | ||
- expectation: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an array containing three items, each of which is a map.
I think you need a map rather than an array of map.
name: "Slow start pipeline"
config:
- pipeline.id: slow-start-pp
config.string: |
input { heartbeat {} }
filter { failure_injector { degrade_at => [register] } }
output { stdout {} }
expectation:
- status: yellow
- symptom: "The pipeline is degraded or at risk of becoming unhealthy; 1 area is impacted and 1 diagnosis is available."
- diagnosis:
- cause: "pipeline is loading"
- impacts:
- description: "pipeline is loading"
- impact_areas: "pipeline_execution"
- details:
- run_state: "LOADING"
logstash_version = os.environ.get("LS_VERSION") | ||
if logstash_version is None: | ||
# version is not specified, use the main branch, no need to git checkout | ||
print(f"LS_VERSION is not specified, using main branch.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggest to rename LS_VERSION to LS_BRANCH. It can't handle version, eg 8.13.4.
$ LS_VERSION="8.13.4" python3 .buildkite/scripts/health-report-tests/main.py
Starting Logstash Health Report Integration test.
Using specified branch: 8.13.4
error: pathspec '8.13.4' did not match any file(s) known to git
Traceback (most recent call last):
File "/Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/main.py", line 74, in <module>
main()
File "/Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/main.py", line 42, in main
with BootstrapContextManager() as bootstrap:
File "/Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/main.py", line 20, in __enter__
self.bootstrap = Bootstrap()
^^^^^^^^^^^
File "/Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/bootstrap.py", line 31, in __init__
util.git_check_out_branch(logstash_version)
File "/Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/util.py", line 17, in git_check_out_branch
run_or_raise_error(["git", "checkout", branch_name],
File "/Users/kcheng/GitHub/kaisecheng/logstash/.buildkite/scripts/health-report-tests/util.py", line 26, in run_or_raise_error
raise Exception(f"{full_error_message}")
Exception: Error occurred while checking out the 8.13.4 branch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have tried LS_VERSION=8.15. It doesn't run the test. Maybe that branch has no python script
$ LS_VERSION="8.15" python3 .buildkite/scripts/health-report-tests/main.py
Starting Logstash Health Report Integration test.
Using specified branch: 8.15
Switched to branch '8.15'
logstash-integration-failure_injector successfully installed.
|
||
def load(self, file_path: str) -> None: | ||
"""Load the YAML file content into self.yaml_content.""" | ||
self.yaml_content: Union[List[Dict[str, Any]], None] = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have another comment related to the structure of yaml file. I think it can be simplified to Union[Dict[str, Any], None]
print(f"Logstash has successfully built.") | ||
|
||
def apply_config(self, config: dict) -> None: | ||
with open(os.getcwd() + "/config/pipelines.yml", 'w') as pipelines_file: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nitpick: we need to be careful not to commit the test config/pipelines.yml, or use --path.settings
to point the config files to other directory
Release notes
[rn:skip]
What does this PR do?
Why is it important/What is the impact to the user?
N.A
Checklist
Author's Checklist
How to test this PR locally
./gradlew clean bootstrap assemble installDefaultGems
logstash-integration-failure_injector
logstash-integration-failure_injector
folder underqa/support
logstash-integration-failure_injector-0.0.1.gem
of Failure injector plugin implementation. #16466 toqa/support/logstash-integration-failure_injector
folderbin/logstash-plugin install qa/support/logstash-integration-failure_injector/logstash-integration-failure_injector-0.0.1.gem
README.md
of this PR to run the integration test.Related issues
Use cases
Screenshots
Logs