... s/version.asciidoc"

Made with ❤️️ by updatecli

Made with ❤️️ by updatecli

Made with ❤️️ by updatecli

... /Dockerfile"

Updated to content "FROM golang:1.19.10" in file "./packetbeat/Dockerfile"

Updated to content "FROM golang:1.19.10" in file "./x-pack/functionbeat/Dockerfile"

Updated to content "FROM golang:1.19.10" in file "./auditbeat/Dockerfile"

Updated to content "FROM golang:1.19.10" in file "./heartbeat/Dockerfile"

Made with ❤️️ by updatecli

…evice handling (#35807)

* [Release] update version

* add `allow_older_versions` to test output configuration

Some integration tests connect to ES, however not all of them had the
flag `allow_older_versions` set in the output configuration. There are
a few cases where this becomes an issue, specially when upgrading the
Beats version.

---------

Co-authored-by: Tiago Queiroz <[email protected]>

* [Release] update test environment

* Update docker-compose.yml

* Update docker-compose.yml

* Update latest.yml

* Update docker-compose.yml

---------

Co-authored-by: Pierre HILBERT <[email protected]>

Fixes #35646 by only unpacking project monitors once.

This fixes the ever growing temp folder issue and is more efficient to boot. Previously we would call fetch on a monitor source every time it was run, but only cleanup the fetched resource once, when the monitor was unloaded. We now fetch once and cleanup once.

This project also fixes the very confusing issue of two files browser/project.go and browser/source/project.go, we have renamed browser/project.go to browser/sourcejob.go which makes reasoning about this change simpler.

* removing obsolete package from metricbeat

* 6

* 6

* 6

* 6

* 6

* 6

* 6

* 6

* removing log line

* dev flag blocklist

* [Heartbeat] Filter synthetics dev flags if Ui monitor

* Add changelog

* Fix unit tests

* Add missing flags

…st (#35892)

The cursor_seek_fallback option was missing from the documentation. And there was no
test associated with the seek mode so I added it.

Through the test I discovered that the documented behavior of the tail seek mode was
wrong so I corrected the docs.

…led (#35658)

* filebeat/input/filestream - Fix registry cleaner log message

This fixes two issues. (1) The wrong printf formatting was used. (2) The error was being
logged on every shutdown.

    "Registrar stopped"
    "failed to start the registry cleaning routine: %!w(*errors.errorString=&{context canceled})"

* Prevent error on normal input shutdown

Filebeat was logging that the input failed, but the stop was triggered by a normal context cancellation.

    "Input 'tcp' failed with: context canceled"
    "Input 'tcp' stopped (runner)"
    "Crawler stopped"
    "Stopping filebeat"

…server closure (#35903)

Previously the server address registry was not updated when the server was cancelled,
leaving invalid state in the registry and preventing reconfiguration of servers.
Ensure that the server address entry is deleted from the pool when the server is
cancelled.

In addition the pool lock was not being released in the case of a failed
TLS consistency check. The change here without fixing that resulted in a
deadlock. This is also fixed.

…35835 (#35886)

This commit implements integration tests for filestream metrics.

It also moves the "integration test framework" from `x-pack/filebeat`
to `libbeat` and implements some improvements on it.

We have update CLI that takes care of Docker image updates. In order
to cover all the images we need to make them consistent first.

If we keep these images at older versions, we'll be constantly
receiving CVE notifications, so it's easier to automate these updates
even though these images are internal and are never released to the public.

So, we can avoid vulnerability reports from Snyk for the outdated images.

Due to some differences between main and 7.17 we need this second job
to account for it and have better automation.

…35915)

This replaces the conversion to a structpb.Struct-serialisation-deserialisation
path that was being used to a direct conversion to a structpb.Struct type which
can then be used to obtain a map[string]any directly.

Using the test suite as a set of benchmarks shows good performance improvements
almost across the board, with no regression. See #35139 for details
of the benchmarks. Note that the benchmark results shown here are at that commit
or that commit plus the change here. No substantive change has been made to the
package to invalidate the comparison.

goos: darwin
goarch: amd64
pkg: github.com/elastic/beats/v7/x-pack/filebeat/input/cel
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
                                                                        │   cel.bench   │             cel-new.bench             │
                                                                        │    sec/op     │    sec/op      vs base                │
Input/hello_world-16                                                       54.08µ ±  2%   37.92µ ±   3%  -29.88% (p=0.000 n=10)
Input/bad_events_type-16                                                   51.45µ ±  3%   37.13µ ±   2%  -27.84% (p=0.000 n=10)
Input/hello_world_non_nil_state-16                                         53.21µ ±  1%   37.11µ ±   3%  -30.27% (p=0.000 n=10)
Input/what_is_next-16                                                      62.86µ ± 14%   38.48µ ±   3%  -38.79% (p=0.000 n=10)
Input/bad_cursor_type-16                                                   59.79µ ±  1%   40.40µ ±   3%  -32.42% (p=0.000 n=10)
Input/show_state-16                                                        53.69µ ±  3%   38.87µ ±   3%  -27.60% (p=0.000 n=10)
Input/show_provided_state-16                                               65.13µ ±  3%   43.54µ ±   2%  -33.15% (p=0.000 n=10)
Input/iterative_state-16                                                    2.000 ±  0%    2.001 ±   0%        ~ (p=0.247 n=10)
Input/iterative_state_implicit_initial_cursor-16                            2.001 ±  0%    2.000 ±   0%        ~ (p=0.063 n=10)
Input/iterative_state_provided_stored_cursor-16                             1.001 ±  0%    1.000 ±   0%   -0.09% (p=0.029 n=10)
Input/iterative_state_implicit_initial_cursor_provided_stored_cursor-16     1.001 ±  0%    1.001 ±   0%        ~ (p=0.912 n=10)
Input/strings_split-16                                                     69.23µ ±  3%   48.67µ ±   4%  -29.70% (p=0.000 n=10)
Input/ndjson_log_file_simple-16                                            164.7µ ±  3%   147.1µ ±   3%  -10.70% (p=0.000 n=10)
Input/ndjson_log_file_simple_file_scheme-16                                167.0µ ±  6%   152.4µ ±   6%   -8.77% (p=0.000 n=10)
Input/ndjson_log_file_corrupted-16                                         179.3µ ±  6%   153.2µ ±   5%  -14.58% (p=0.000 n=10)
Input/missing_file-16                                                      67.98µ ±  2%   52.31µ ±   3%  -23.05% (p=0.000 n=10)
Input/decode_xml-16                                                        871.6µ ± 28%   780.2µ ±  25%        ~ (p=0.143 n=10)
Input/GET_request-16                                                       728.8µ ± 73%   690.1µ ±  70%        ~ (p=0.579 n=10)
Input/retry_after_request-16                                               766.5µ ± 58%   623.6µ ± 134%        ~ (p=0.393 n=10)
Input/retry_after_request_time-16                                          1.277m ± 31%   1.044m ±  14%        ~ (p=0.247 n=10)
Input/rate_limit_request_0-16                                              1.143m ± 45%   1.059m ±   6%        ~ (p=0.123 n=10)
Input/rate_limit_request_10-16                                             1.089m ± 35%   1.067m ±   7%        ~ (p=0.631 n=10)
Input/rate_limit_request_10_too_slow-16                                    994.7µ ± 29%   930.8µ ±  36%        ~ (p=0.481 n=10)
Input/retry_failure-16                                                    1184.4µ ± 51%   959.6µ ±  12%        ~ (p=0.143 n=10)
Input/POST_request-16                                                      1.376m ± 39%   1.125m ±  21%        ~ (p=0.393 n=10)
Input/repeated_POST_request-16                                             110.1m ±  4%   117.2m ±   2%   +6.39% (p=0.002 n=10)
Input/split_events-16                                                      877.6µ ± 30%   793.4µ ±  35%        ~ (p=0.631 n=10)
Input/split_events_keep_parent-16                                          1.148m ± 54%   1.048m ±  21%        ~ (p=0.529 n=10)
Input/nested_split_events-16                                               1.124m ±  6%   1.038m ±   7%   -7.68% (p=0.009 n=10)
Input/absent_split-16                                                       4.999 ±  0%    4.999 ±   0%        ~ (p=0.684 n=10)
Input/date_cursor-16                                                        2.002 ±  0%    2.002 ±   0%        ~ (p=0.912 n=10)
Input/tracer_filename_sanitization-16                                       2.002 ±  0%    2.002 ±   0%        ~ (p=0.218 n=10)
Input/pagination_cursor_object-16                                           1.001 ±  0%    1.002 ±   0%   +0.09% (p=0.000 n=10)
Input/pagination_cursor_array-16                                            1.002 ±  0%    1.002 ±   0%   +0.08% (p=0.023 n=10)
Input/first_event_cursor-16                                                 3.002 ±  0%    3.001 ±   0%        ~ (p=0.529 n=10)
Input/OAuth2-16                                                            1.243m ± 28%   1.082m ±  23%        ~ (p=0.165 n=10)
Input/simple_multistep_GET_request-16                                      2.821m ± 57%   2.291m ± 107%        ~ (p=0.529 n=10)
Input/three_step_GET_request-16                                            2.821m ± 23%   2.996m ±  32%        ~ (p=0.853 n=10)
Input/type_error_message-16                                               1463.4µ ± 61%   969.2µ ±  89%        ~ (p=0.280 n=10)
geomean                                                                    3.751m         3.243m         -13.56%

                                                                        │   cel.bench    │             cel-new.bench              │
                                                                        │      B/op      │      B/op       vs base                │
Input/hello_world-16                                                       27.76Ki ±  0%    25.97Ki ±  0%   -6.46% (p=0.000 n=10)
Input/bad_events_type-16                                                   26.01Ki ±  0%    24.74Ki ±  0%   -4.91% (p=0.000 n=10)
Input/hello_world_non_nil_state-16                                         27.47Ki ±  0%    25.67Ki ±  0%   -6.54% (p=0.000 n=10)
Input/what_is_next-16                                                      30.49Ki ±  0%    27.89Ki ±  0%   -8.53% (p=0.000 n=10)
Input/bad_cursor_type-16                                                   28.84Ki ±  0%    26.65Ki ±  0%   -7.58% (p=0.000 n=10)
Input/show_state-16                                                        27.24Ki ±  0%    25.53Ki ±  0%   -6.30% (p=0.000 n=10)
Input/show_provided_state-16                                               28.76Ki ±  0%    26.34Ki ±  0%   -8.41% (p=0.000 n=10)
Input/iterative_state-16                                                   55.05Ki ±  1%    45.29Ki ±  1%  -17.74% (p=0.000 n=10)
Input/iterative_state_implicit_initial_cursor-16                           59.17Ki ±  0%    49.39Ki ±  0%  -16.53% (p=0.000 n=10)
Input/iterative_state_provided_stored_cursor-16                            45.98Ki ±  1%    38.63Ki ±  0%  -15.97% (p=0.000 n=10)
Input/iterative_state_implicit_initial_cursor_provided_stored_cursor-16    49.48Ki ±  1%    42.13Ki ±  0%  -14.86% (p=0.000 n=10)
Input/strings_split-16                                                     33.30Ki ±  0%    30.48Ki ±  0%   -8.46% (p=0.000 n=10)
Input/ndjson_log_file_simple-16                                            36.40Ki ±  0%    33.69Ki ±  0%   -7.45% (p=0.000 n=10)
Input/ndjson_log_file_simple_file_scheme-16                                36.54Ki ±  0%    33.83Ki ±  0%   -7.42% (p=0.000 n=10)
Input/ndjson_log_file_corrupted-16                                         39.92Ki ±  0%    36.24Ki ±  0%   -9.20% (p=0.000 n=10)
Input/missing_file-16                                                      27.92Ki ±  0%    26.10Ki ±  0%   -6.52% (p=0.000 n=10)
Input/decode_xml-16                                                        89.81Ki ±  0%    82.66Ki ±  0%   -7.96% (p=0.000 n=10)
Input/GET_request-16                                                       60.89Ki ±  0%    57.45Ki ±  0%   -5.66% (p=0.000 n=10)
Input/retry_after_request-16                                               59.99Ki ±  0%    56.13Ki ±  0%   -6.43% (p=0.001 n=10)
Input/retry_after_request_time-16                                          60.00Ki ±  0%    56.19Ki ±  0%   -6.35% (p=0.000 n=10)
Input/rate_limit_request_0-16                                              63.69Ki ±  0%    58.42Ki ±  0%   -8.27% (p=0.000 n=10)
Input/rate_limit_request_10-16                                             63.64Ki ±  0%    58.54Ki ±  0%   -8.02% (p=0.000 n=10)
Input/rate_limit_request_10_too_slow-16                                    63.77Ki ±  0%    58.45Ki ±  0%   -8.34% (p=0.001 n=10)
Input/retry_failure-16                                                     60.12Ki ±  0%    56.31Ki ±  0%   -6.34% (p=0.001 n=10)
Input/POST_request-16                                                      64.29Ki ±  0%    60.64Ki ±  0%   -5.67% (p=0.000 n=10)
Input/repeated_POST_request-16                                            103.90Ki ±  2%    97.28Ki ±  3%   -6.37% (p=0.000 n=10)
Input/split_events-16                                                      59.25Ki ±  0%    56.50Ki ±  0%   -4.63% (p=0.000 n=10)
Input/split_events_keep_parent-16                                          64.27Ki ±  0%    60.44Ki ±  0%   -5.96% (p=0.000 n=10)
Input/nested_split_events-16                                               62.33Ki ±  0%    59.42Ki ±  0%   -4.68% (p=0.000 n=10)
Input/absent_split-16                                                      175.5Ki ±  3%    172.8Ki ±  4%        ~ (p=0.089 n=10)
Input/date_cursor-16                                                       134.7Ki ±  6%    136.6Ki ±  7%        ~ (p=0.393 n=10)
Input/tracer_filename_sanitization-16                                      209.5Ki ±  5%    212.0Ki ±  4%        ~ (p=0.684 n=10)
Input/pagination_cursor_object-16                                          99.86Ki ± 12%   100.96Ki ±  8%        ~ (p=0.739 n=10)
Input/pagination_cursor_array-16                                           98.43Ki ± 16%   103.33Ki ± 12%        ~ (p=0.912 n=10)
Input/first_event_cursor-16                                                155.1Ki ± 14%    151.9Ki ± 19%        ~ (p=0.353 n=10)
Input/OAuth2-16                                                            81.98Ki ±  0%    79.54Ki ±  0%   -2.97% (p=0.000 n=10)
Input/simple_multistep_GET_request-16                                      83.07Ki ±  0%    78.70Ki ±  1%   -5.26% (p=0.000 n=10)
Input/three_step_GET_request-16                                            105.3Ki ±  0%    100.7Ki ±  1%   -4.35% (p=0.000 n=10)
Input/type_error_message-16                                                50.86Ki ±  0%    50.24Ki ±  0%   -1.24% (p=0.000 n=10)
geomean                                                                    58.17Ki          54.43Ki         -6.43%

                                                                        │  cel.bench   │            cel-new.bench             │
                                                                        │  allocs/op   │  allocs/op    vs base                │
Input/hello_world-16                                                       204.0 ±  0%    166.0 ±  1%  -18.63% (p=0.000 n=10)
Input/bad_events_type-16                                                   177.0 ±  0%    151.0 ±  0%  -14.69% (p=0.000 n=10)
Input/hello_world_non_nil_state-16                                         202.0 ±  0%    164.0 ±  0%  -18.81% (p=0.000 n=10)
Input/what_is_next-16                                                      261.0 ±  0%    203.0 ±  0%  -22.22% (p=0.000 n=10)
Input/bad_cursor_type-16                                                   235.0 ±  0%    187.0 ±  0%  -20.43% (p=0.000 n=10)
Input/show_state-16                                                        199.0 ±  0%    164.0 ±  0%  -17.59% (p=0.000 n=10)
Input/show_provided_state-16                                               269.0 ±  0%    204.0 ±  0%  -24.16% (p=0.000 n=10)
Input/iterative_state-16                                                   702.0 ±  1%    485.0 ±  1%  -30.91% (p=0.000 n=10)
Input/iterative_state_implicit_initial_cursor-16                           766.0 ±  0%    549.0 ±  0%  -28.33% (p=0.000 n=10)
Input/iterative_state_provided_stored_cursor-16                            507.5 ±  1%    361.0 ±  1%  -28.87% (p=0.000 n=10)
Input/iterative_state_implicit_initial_cursor_provided_stored_cursor-16    553.0 ±  1%    406.0 ±  1%  -26.58% (p=0.000 n=10)
Input/strings_split-16                                                     306.0 ±  0%    243.0 ±  0%  -20.59% (p=0.000 n=10)
Input/ndjson_log_file_simple-16                                            319.0 ±  0%    254.0 ±  0%  -20.38% (p=0.000 n=10)
Input/ndjson_log_file_simple_file_scheme-16                                319.0 ±  0%    254.0 ±  0%  -20.38% (p=0.000 n=10)
Input/ndjson_log_file_corrupted-16                                         373.0 ±  0%    294.0 ±  0%  -21.18% (p=0.000 n=10)
Input/missing_file-16                                                      204.0 ±  0%    168.0 ±  0%  -17.65% (p=0.000 n=10)
Input/decode_xml-16                                                        903.0 ±  0%    749.0 ±  0%  -17.05% (p=0.000 n=10)
Input/GET_request-16                                                       540.0 ±  0%    467.0 ±  0%  -13.52% (p=0.000 n=10)
Input/retry_after_request-16                                               570.0 ±  0%    479.0 ±  0%  -15.96% (p=0.000 n=10)
Input/retry_after_request_time-16                                          570.0 ±  0%    479.0 ±  0%  -15.96% (p=0.000 n=10)
Input/rate_limit_request_0-16                                              623.0 ±  0%    511.0 ±  0%  -17.98% (p=0.000 n=10)
Input/rate_limit_request_10-16                                             623.0 ±  0%    512.0 ±  0%  -17.82% (p=0.000 n=10)
Input/rate_limit_request_10_too_slow-16                                    623.0 ±  0%    512.0 ±  0%  -17.82% (p=0.000 n=10)
Input/retry_failure-16                                                     571.0 ±  0%    479.0 ±  0%  -16.11% (p=0.000 n=10)
Input/POST_request-16                                                      579.0 ±  0%    497.0 ±  0%  -14.16% (p=0.000 n=10)
Input/repeated_POST_request-16                                            1045.5 ±  1%    879.5 ±  0%  -15.88% (p=0.000 n=10)
Input/split_events-16                                                      504.0 ±  0%    442.0 ±  0%  -12.30% (p=0.000 n=10)
Input/split_events_keep_parent-16                                          581.0 ±  0%    500.0 ±  0%  -13.94% (p=0.000 n=10)
Input/nested_split_events-16                                               559.0 ±  0%    498.0 ±  0%  -10.91% (p=0.000 n=10)
Input/absent_split-16                                                     1.625k ±  1%   1.448k ±  1%  -10.92% (p=0.000 n=10)
Input/date_cursor-16                                                      1.353k ±  1%   1.358k ±  1%        ~ (p=0.566 n=10)
Input/tracer_filename_sanitization-16                                     1.697k ±  1%   1.690k ±  1%        ~ (p=0.566 n=10)
Input/pagination_cursor_object-16                                          869.0 ± 44%    874.5 ± 30%        ~ (p=0.493 n=10)
Input/pagination_cursor_array-16                                           870.5 ± 50%    869.5 ± 37%        ~ (p=1.000 n=10)
Input/first_event_cursor-16                                               1.603k ±  1%   1.596k ±  2%        ~ (p=0.541 n=10)
Input/OAuth2-16                                                            658.5 ±  0%    621.0 ±  0%   -5.69% (p=0.000 n=10)
Input/simple_multistep_GET_request-16                                      740.0 ±  0%    665.0 ±  0%  -10.14% (p=0.000 n=10)
Input/three_step_GET_request-16                                            939.0 ±  0%    864.0 ±  0%   -7.99% (p=0.000 n=10)
Input/type_error_message-16                                                342.0 ±  0%    341.0 ±  0%   -0.29% (p=0.000 n=10)
geomean                                                                    526.0          444.9        -15.42%

* docs: Close changelog for 8.8.2

* Update CHANGELOG.asciidoc

* Update CHANGELOG.asciidoc

Co-authored-by: Craig MacKenzie <[email protected]>

* Update CHANGELOG.asciidoc

* Update CHANGELOG.asciidoc

Co-authored-by: David Kilfoyle <[email protected]>

* Update CHANGELOG.asciidoc

Co-authored-by: David Kilfoyle <[email protected]>

* Update CHANGELOG.asciidoc

Co-authored-by: David Kilfoyle <[email protected]>

* Update CHANGELOG.asciidoc

Co-authored-by: David Kilfoyle <[email protected]>

* Update CHANGELOG.asciidoc

Co-authored-by: David Kilfoyle <[email protected]>

* Update CHANGELOG.asciidoc

Co-authored-by: David Kilfoyle <[email protected]>

---------

Co-authored-by: Pierre HILBERT <[email protected]>
Co-authored-by: Craig MacKenzie <[email protected]>
Co-authored-by: David Kilfoyle <[email protected]>
(cherry picked from commit 15772de)

Co-authored-by: Elastic Machine <[email protected]>

Remove a NOTE section telling the `cloud.id` does not explicitly specify a port because it does specify a port nowadays.

…ok (#35798)

Register an elastic-agent diagnostics hook to return the input metrics (encoded to JSON)
in agent diagnostic dumps.

- If a duplicate ID was used for filestream, set the ID for metrics
to an empty string to prevent a panic.
- A valid metrics instance will still be generated, but collection of
metrics will not occur.

…35829)

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli

---------

Co-authored-by: apmmachine <[email protected]>

…35999)

* adding deployment condition status

* Update CHANGELOG.next.asciidoc

* fixing unittests

---------

Co-authored-by: Chris Mark <[email protected]>

* Bump github.com/elastic/go-elasticsearch/v8 from 8.8.1 to 8.8.2

Bumps [github.com/elastic/go-elasticsearch/v8](https://github.com/elastic/go-elasticsearch) from 8.8.1 to 8.8.2.
- [Release notes](https://github.com/elastic/go-elasticsearch/releases)
- [Changelog](https://github.com/elastic/go-elasticsearch/blob/v8.8.2/CHANGELOG.md)
- [Commits](elastic/go-elasticsearch@v8.8.1...v8.8.2)

---
updated-dependencies:
- dependency-name: github.com/elastic/go-elasticsearch/v8
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update NOTICE.txt

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* do not print context canceled errors

* add comments

* add changelog

* update changelog

…36016)

* Fix recovering from invalid output configuration under Elastic-Agent

This commit fixes two problems in the `ManagerV2`:
- 1. If the first output configuration was invalid, it would never be
"saved", so if a new output configuration was received, the ManagerV2
would try to reload it even with `stopOnOutputReload` enabled.

2. When `stopOnOutputReload` was set the output reload was skipped,
but the ManagerV2 would still proceed with reloading inputs. This
created a race condition where some inputs/runners would receive their
stop signal before even start, effectively locking the shutdown
process.

* Add and refactor tests

This commit add tests to ensure Beats can recover from an invalid
output configuration when running under Elastic-Agent.

It also refactors some of the code used to mock the Elastic-Agent and
aggregates some shared code into a single package.

* PR improvements

* Add changelog

* PR improvements

…ml" (#36003)

Made with ❤️️ by updatecli

Co-authored-by: apmmachine <[email protected]>

Also add a check in the common path to check that the addr slice is the
same length as the undefined addr slice, returning an error on mismatch
rather than allowing the panic.

)

* fix rollovers in system/network host data

* fix tests

* fix tests, add log statements, comments

* change logging, logic

* fix off-by-one

)

Ideally this would be a hard requirement, but that would be a breaking
change, so just log at WARN if the configuration is missing.

This commit removes the logger from the WriteJSONKeys which is executed as part of the hot path of the Event and was allocating a new logger for each event.
This has an impact on the memory allocation of the function and because it's executed for each event, we see a direct benefit in the overall memory allocations.

* Improve streambuf append

* Adding changelog comment

* Update CHANGELOG.next.asciidoc

Co-authored-by: Craig MacKenzie <[email protected]>

---------

Co-authored-by: Craig MacKenzie <[email protected]>

* remove deprecate flag that wasn't deprecated

* fix error messages in linter

* fix errors

* Decrease Clones

* Adding changelog

* CodeReview changes

* add CleanSession parameter

* Cleansession parameter add

* correct cleansession parameter name

* Defaut clean session to True

* formating file

* Add missing documentation

* Update filebeat/docs/inputs/input-mqtt.asciidoc

Co-authored-by: Shaunak Kashyap <[email protected]>

* Update filebeat/docs/inputs/input-mqtt.asciidoc

Co-authored-by: Shaunak Kashyap <[email protected]>

* doc correction

* Update CHANGELOG.next.asciidoc

Co-authored-by: Shaunak Kashyap <[email protected]>

* Update filebeat/docs/inputs/input-mqtt.asciidoc

Co-authored-by: Shaunak Kashyap <[email protected]>

* correct formatting

---------

Co-authored-by: Shaunak Kashyap <[email protected]>

This is a new alternative to existing options like `native`, `path`
and `inode_marker`.

Unlike the existing options, this file identity does not rely on any
file system metadata and uses only the file size and its content.

Users can specify what amount of bytes is used to fingerprint the
beginning of each file, optionally it's possible to set an offset from the beginning.

This identity is supposed to be more stable and less affected by the
environment/setup of the users.

This change also contains a few performance optimisations of how we work with the filesystem and watch for file changes.

... s/version.asciidoc"

Made with ❤️️ by updatecli

Made with ❤️️ by updatecli

Made with ❤️️ by updatecli

…36041)

* chore: Updated to content "" in file "testing/environments/snapshot.yml"

Made with ❤️️ by updatecli
---------

Co-authored-by: apmmachine <[email protected]>

By using a buffer of the fingerprint's size. This descreases the
amount of syscalls for reading the file.

Also added the benchmark for tracking the function's performance.

This includes:

* Unix-like:
  * device
	* inode
* Windows:
  * idxlo
	* idxhi
	* vol
* All:
  * Fingerprint (if the fingerprint mode is enabled)

* Unstash terraform config into correct directory

* Fix Jenkinsfile

* fix failing test

* Mention issue for commented code

So, we can compare how much slower the fingerprinting mode against using
`device` and `inode` in the scanner.

CPython 3.0 was released recently which appears to have introduced a regression that leads to failures when installing PyYAML (and perhaps other packages too). This is a temporary fix and this commits needs to be reverted when a proper fix is available.

It's possible that file scanning happens when the file was created but
was not yet written to. In this case the size is 0.

We should not spawn any resources (e.g. harvesters) for such files until they actually
have some content. We create events only when read something from a
file, so having a harvester on an empty file is not useful.

Not handling this edge case also causes our tests to be flaky,
sometimes an expected size does not match an actual size (0).

…ns (#36034)

The code currently prioritises the region_name configuration, even when
it is not provided, against the claims of the documentation. Make a
check whether it is empty before claiming a conflict and using it.

It's supposed to emit a `create` event, not a `write` event when a
previously ignored empty file gets written to.

* update docker documentation example for system metrics

* adjust language

)

…etwork bytes (#36108)

Relax the width of in and out fields from integer to long to allow
values greater beyond 2GiB.

* Add project dir

* Remove comments

* Add changelog

* Fix ndjson parser to store JSON fields correctly under target

* Update CHANGELOG.next.asciidoc

* Move the change log entry to the right section

---------

Co-authored-by: Shaunak Kashyap <[email protected]>

)

Previously now was static for the life of the program, which
corresponded to the life of the input. This could lead to incorrect and
surprising times being provided when the global was used. Obtain a now
value before starting each evaluation and use it to shadow the CEL now
global (and share it with the Go logging to allow correlation between
these).

#36036)

* added min/max functions

* updated methods to accept integer list

* updated the methods to accept atleast 1 arg

* extended the min/max functions to be more generic

* reworked the min/max implementation

* made the min max funcs generic with a custom number ingterface

* Implement min/max using text/template lt func

---------

Co-authored-by: Andrew Kroh <[email protected]>

…ked on input metric (#36101)

…d owner/user handling (#36092)

…36139)

* docs: update docs

* make update

---------

Co-authored-by: Pierre HILBERT <[email protected]>

…n localstack (#35727)

* Test integration tests on localstack

* Add localstack container in CI

* fix panic in SQS input metrics

* Add documentation

* Fix PR comments

* Update forecast API metrics interval

* docs: Prepare Changelog for 8.9.0 (#36128)

Co-authored-by: David Kilfoyle <[email protected]>
Co-authored-by: Pierre HILBERT <[email protected]>
Co-authored-by: James Rodewig <[email protected]>
(cherry picked from commit b1b4d8b)

# Conflicts:
#	CHANGELOG.asciidoc
#	libbeat/docs/release.asciidoc

* Fixing conflicts

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: Pierre HILBERT <[email protected]>

…d entries (#35408)

Co-authored-by: Andrew Kroh <[email protected]>

* upgrade Go to 1.20.5

* ignore known issue with go tool cover

* capture cover output and check it for errors

* debug

* wip

* move ragel files to root directory

the //line directives used to point to the original "ragel" files were creating problems for the coverage tool. They were being reported without the 'parser/' directory. Causing the go tool cover to look for the wrong files and therefore failing.
Moving it all to the root directory fixes the problem.

* upgrade to go1.20.6

* .

* Revert "move ragel files to root directory"

This reverts commit 2b6a159.

* remove debug

* disable //line directives

* goimport

Add a new item in the checklist about commit title and message that should be explanatory.

Decouples run_once from sync pipeline and adds a timeout before exiting for emitting pending events.

* Add publish pipeline timeout to run_once

* Clean up ISyncClient

* Nit wait for run_once

* Rename pipeline

* Add signal tests

* Add pipeline sync tests

* Disable linter false positive

* golint

* Apply suggestions from code review

* Update heartbeat/monitors/pipeline.go

* Apply suggestions from code review

* Add changelog and docs

Migrates some of the libbeat python integration tests to go

---------

Co-authored-by: Amit Kanfer <[email protected]>

* fixed httpjson input metric naming convention to solve parsing error in fleet

* fixed httpjson input metric naming convention to solve parsing error in fleet

* updated asciidoc with issue and pr description

* Update CHANGELOG-developer.next.asciidoc

Co-authored-by: Andrew Kroh <[email protected]>

---------

Co-authored-by: Andrew Kroh <[email protected]>

…erformance (#36178)

The details regexp pattern for the powershell data stream was improved
in the windows integration in elastic/integrations#6154 to reduce
backtracking costs that caused regexp costs to exceed the set runtime
limits. The same pattern on other data streams and in winlogbeat was not
updated. This change brings winlogbeat up to date with the change.

Defer installation until we have a valid config with streams configured
so that we can determine whether the user has blocked install and
whether we need the DLL.

In the fleet-managed case New is called without a complete config, which
means that we have no stream to tell us not to install the Npcap DLL.

## What does this PR do?

Adds a note to
[Modules](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html)
to advise users to migrate to Elastic Agent.

Preview:
[Modules](https://beats_36157.docs-preview.app.elstc.co/guide/en/beats/filebeat/master/filebeat-modules.html)

## Why is it important?

While Filebeat modules are still supported, we do not typically apply
bug fixes or enhancements to these modules, or ship any new modules. Our
focus is now entirely on Elastic Agent integrations.

## Checklist

<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to
have the PR approved

List here all the items you have verified BEFORE sending this PR. Please
DO NOT remove any item, striking through those that do not apply. (Just
in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->

- [x] My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand
areas~~ N/A
- [x] I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration
files~~ N/A
~~- [ ] I have added tests that prove my fix is effective or that my
feature works~~ N/A
~~- [ ] I have added an entry in `CHANGELOG.next.asciidoc` or
`CHANGELOG-developer.next.asciidoc`.~~ N/A

## Related issues

* Resolves elastic/security-docs#3576

---------

Co-authored-by: Janeen Mikell Roberts <[email protected]>
Co-authored-by: David Kilfoyle <[email protected]>

Bumps
[github.com/elastic/go-elasticsearch/v8](https://github.com/elastic/go-elasticsearch)
from 8.8.2 to 8.9.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/elastic/go-elasticsearch/releases">github.com/elastic/go-elasticsearch/v8's
releases</a>.</em></p>
<blockquote>
<h2>8.9.0</h2>
<h2>API</h2>
<p><strong>New API</strong></p>
<ul>
<li><code>Cluster.Info</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/cluster-info.html">Documentation</a></li>
</ul>
<p><strong>Experimental APIs</strong></p>
<ul>
<li><code>QueryRulesetGet</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/get-query-ruleset.html">Documentation</a></li>
<li><code>QueryRulesetDelete</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/delete-query-ruleset.html">Documentation</a></li>
<li><code>QueryRulesetPut</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/put-query-ruleset.html">Documentation</a></li>
<li><code>SearchApplicationRenderQuery</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-application-render-query.html">Documentation</a></li>
<li><code>Security.CreateCrossClusterAPIKey</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html">Documentation</a></li>
<li><code>Security.UpdateCrossClusterAPIKey</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-update-cross-cluster-api-key.html">Documentation</a></li>
</ul>
<h2>Typed API</h2>
<ul>
<li>Propagated request fields towards the endpoint for ease of access,
taking priority over same-name query string fields.</li>
<li>Added a stub for Do methods on endpoints that only support a boolean
response such as <code>core.exists</code>.</li>
<li>NDJSON endpoints support with custom serialization like
<code>core.bulk</code>.</li>
<li>Link to endpoints documentation in API index to better display and
ease of use.</li>
</ul>
<p><strong>fixes</strong></p>
<ul>
<li>Fixed a deserialization issue for <code>Property</code> &amp;
<code>Analyzer</code> <a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/696">#696</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/elastic/go-elasticsearch/blob/main/CHANGELOG.md">github.com/elastic/go-elasticsearch/v8's
changelog</a>.</em></p>
<blockquote>
<h1>8.9.0</h1>
<h2>API</h2>
<p><strong>New API</strong></p>
<ul>
<li><code>Cluster.Info</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/cluster-info.html">Documentation</a></li>
</ul>
<p><strong>Experimental APIs</strong></p>
<ul>
<li><code>QueryRulesetGet</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/get-query-ruleset.html">Documentation</a></li>
<li><code>QueryRulesetDelete</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/delete-query-ruleset.html">Documentation</a></li>
<li><code>QueryRulesetPut</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/put-query-ruleset.html">Documentation</a></li>
<li><code>SearchApplicationRenderQuery</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-application-render-query.html">Documentation</a></li>
<li><code>Security.CreateCrossClusterAPIKey</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html">Documentation</a></li>
<li><code>Security.UpdateCrossClusterAPIKey</code> <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-update-cross-cluster-api-key.html">Documentation</a></li>
</ul>
<h2>Typed API</h2>
<ul>
<li>Propagated request fields towards the endpoint for ease of access,
taking priority over same-name query string fields.</li>
<li>Added a stub for Do methods on endpoints that only support a boolean
response such as <code>core.exists</code>.</li>
<li>NDJSON endpoints support with custom serialization like
<code>core.bulk</code>.</li>
<li>Link to endpoints documentation in API index to better display and
ease of use.</li>
</ul>
<p><strong>fixes</strong></p>
<ul>
<li>Fixed a deserialization issue for <code>Property</code> &amp;
<code>Analyzer</code> <a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/696">#696</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/a3a9de75e79b25cfb271d1bf4c0365a77e20e2d5"><code>a3a9de7</code></a>
Release 8.9.0</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/e8ab3d1cbc3e3c3eb7be4ac088fe4a4bc3d458d4"><code>e8ab3d1</code></a>
Changelog: Add 8.9 release changelog (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/707">#707</a>)
(<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/708">#708</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/b8769ed392e6b8052bad881bdca5c7173413250c"><code>b8769ed</code></a>
[APIs] Bump api 8.9 (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/706">#706</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/0ccfa7b704c3c75df826c3a71fc2f97aa0ba7bab"><code>0ccfa7b</code></a>
API: Update typed API for Elasticsearch v8.9.0 (656080d) (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/704">#704</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/cd52216788490a72e7e9f4bffa15f48fb418935a"><code>cd52216</code></a>
[CI] Update skip list with incompatible tests (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/702">#702</a>)
(<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/703">#703</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/0d8b3966f48b647a8984618a5db570f511e8521c"><code>0d8b396</code></a>
[DOCS] Restructures the repo README (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/697">#697</a>)
(<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/699">#699</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/d27980860b5e5403cbde72f2e9505e75ed8c59b3"><code>d279808</code></a>
[8.9] Bump Elasticsearch apis (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/693">#693</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/d4f29a064888031e4624a82f69fc079ab6462ff3"><code>d4f29a0</code></a>
[DOCS] Changes getting started button link on landing page. (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/689">#689</a>)
(<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/691">#691</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/824db68bb7274f653e1daec885bfb37b91578dbb"><code>824db68</code></a>
[DOCS] Adds getting started content based on the template (<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/686">#686</a>)
(<a
href="https://redirect.github.com/elastic/go-elasticsearch/issues/688">#688</a>)</li>
<li><a
href="https://github.com/elastic/go-elasticsearch/commit/4ea9db9cf63079fd1dbdc92e44737869f40589d9"><code>4ea9db9</code></a>
DX: Only add require_alias parameter if its true, prevents requirement
for a ...</li>
<li>Additional commits viewable in <a
href="https://github.com/elastic/go-elasticsearch/compare/v8.8.2...v8.9.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/elastic/go-elasticsearch/v8&package-manager=go_modules&previous-version=8.8.2&new-version=8.9.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

## What does this PR do?

This PR documents the metrics that are periodically emitted by Beats in
their logs.

<!-- Mandatory
Explain here the changes you made on the PR. Please explain the WHAT:
patterns used, algorithms implemented, design architecture, message
processing, etc.
-->

## Why is it important?

To understand what the metrics mean and how they could be used for
troubleshooting.
<!-- Mandatory
Explain here the WHY, or the rationale/motivation for the changes.
-->

---------

Co-authored-by: David Kilfoyle <[email protected]>
Co-authored-by: David Kilfoyle <[email protected]>
Co-authored-by: Craig MacKenzie <[email protected]>