Merge main into feature/health-report-api branch #16452
Open
+2,082
−4,365
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320
…#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332
make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <[email protected]>
…alues of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <[email protected]>
---------
Co-authored-by: João Duarte <[email protected]>
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV .
This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields. Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`. Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`. Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields.
…ror without killing the pipeline (#16322) This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it. This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch. Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node. Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event. --------- Co-authored-by: Karen Metts <[email protected]>
* Update release notes for 8.15.1 * update release note --------- Co-authored-by: logstashmachine <[email protected]> Co-authored-by: Kaise Cheng <[email protected]> (cherry picked from commit 2fca7e3) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ed in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) Make inner field of ConditionalEvaluationError transient to be avoided during serialization.
Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters. For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem.
the current DRA publishing task computes the branch from the version contained in the version.yml This is done by taking the major.minor and confirming that a branch exists with that name. However this pattern won't be applicable for 8.x, as that branch currently points to 8.16.0 and there is no 8.16 branch. This commit falls back to reading the buildkite injected BUILDKITE_BRANCH variable.
…rsions (#16421) - add becnhmark dashboard and related save objects - add one buildkite pipeline to test against multiple versions - remove null field in json - add `FLOG_FILE_CNT`, `VAULT_PATH`, `TAGS`
If a user runs Logstash with a hosted JDK and not the one bundled with Logstash distribution, like setting a specific LS_JAVA_HOME, which is minor than JDK 17 then Logstash refuses to start. Has to provide at least a JDK 17 or unset the LS_JAVA_HOME and let Logstash uses the bundled JDK. Updates the jvm.options and JvmOptionsParser to remove support for JDK 11. If the options parser identifies that the running JVM is less than 17, it refuses to start. --------- Co-authored-by: João Duarte <[email protected]>
Removed the tool Ingest Converter
…es. (#16456) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test.
- split main.sh to core.sh and main.sh - rename all.sh to marathon.sh - fix vault expiry issue for long running task - fix unable to call main function - update save object runtime field `release` to return true when `version` contains "SNAPSHOT"
* remove docker image scanning as that's handled by infosec * run buildkite job on a docker image instead of vm (no need to test docker any more)
|
💚 Build Succeeded
History
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release notes
[rn:skip]
What does this PR do?
feature/health-report-apibranch to get the relative changes (integration tests BK pipeline) to health report.Why is it important/What is the impact to the user?
N.A
Checklist
My code follows the style guidelines of this projectI have commented my code, particularly in hard-to-understand areasI have made corresponding changes to the documentationI have made corresponding change to the default configuration files (and/or docker env variables)I have added tests that prove my fix is effective or that my feature worksAuthor's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs