Skip to content

Commit

Permalink
chore: change base version to 4.27.0 (#26)
Browse files Browse the repository at this point in the history
* Fix small mistakes in function doc comments

Signed-off-by: Peter Elmers <[email protected]>

* Add `redis_scan` input.

Signed-off-by: Marco Amador <[email protected]>

* Add `redis_scan` input integration test.

Signed-off-by: Marco Amador <[email protected]>

* Update using_rest_api.md

Fix typo

Signed-off-by: Geert G <[email protected]>

* Update using_config_files.md

Fix typo

Signed-off-by: Geert G <[email protected]>

* Add NATS request/reply pattern processor

Signed-off-by: Marcel Hauf <[email protected]>

* Add NATS KV Cache

Signed-off-by: Marcel Hauf <[email protected]>

* Update README

* fix: always bind a given stream when SubscribeSync()

Signed-off-by: Anton Frank <[email protected]>

* Update docs

* Ensure we set the stream for jetstream

* Use newer scram package

* Fix batch timer panic

* Accept 'null' as 'false' in if statements (until v5)

* Add CRDB changefeed input

* Rework crdb changefeed input

* Update CHANGELOG

* Punt unused field

Also clean up .golangci.yml a bit.

Signed-off-by: Mihai Todor <[email protected]>

* Fix jsonschema root path resolution

* Fix metrics mapping config parsing

* Bump golangci/golangci-lint-action from 3 to 4

Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3 to 4.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](golangci/golangci-lint-action@v3...v4)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump the production-dependencies group with 24 updates

Bumps the production-dependencies group with 24 updates:

| Package | From | To |
| --- | --- | --- |
| [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) | `1.58.0` | `1.59.0` |
| [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) | `1.34.0` | `1.36.1` |
| [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) | `1.36.0` | `1.37.0` |
| [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) | `1.9.1` | `1.9.2` |
| [github.com/ClickHouse/clickhouse-go/v2](https://github.com/ClickHouse/clickhouse-go) | `2.17.1` | `2.18.0` |
| [github.com/IBM/sarama](https://github.com/IBM/sarama) | `1.42.1` | `1.42.2` |
| [github.com/apache/pulsar-client-go](https://github.com/apache/pulsar-client-go) | `0.11.1` | `0.12.0` |
| [github.com/aws/aws-lambda-go](https://github.com/aws/aws-lambda-go) | `1.45.0` | `1.46.0` |
| [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) | `1.50.0` | `1.50.15` |
| [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) | `0.26.0` | `0.27.0` |
| [github.com/go-faker/faker/v4](https://github.com/go-faker/faker) | `4.2.0` | `4.3.0` |
| [github.com/jhump/protoreflect](https://github.com/jhump/protoreflect) | `1.15.4` | `1.15.6` |
| [github.com/klauspost/compress](https://github.com/klauspost/compress) | `1.17.4` | `1.17.6` |
| [github.com/sijms/go-ora/v2](https://github.com/sijms/go-ora) | `2.8.6` | `2.8.7` |
| [github.com/twmb/franz-go](https://github.com/twmb/franz-go) | `1.16.0` | `1.16.1` |
| [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) | `1.22.0` | `1.23.1` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) | `1.22.0` | `1.23.1` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) | `1.22.0` | `1.23.1` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp](https://github.com/open-telemetry/opentelemetry-go) | `1.22.0` | `1.23.1` |
| [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) | `1.22.0` | `1.23.1` |
| [go.opentelemetry.io/otel/trace](https://github.com/open-telemetry/opentelemetry-go) | `1.22.0` | `1.23.1` |
| [golang.org/x/net](https://github.com/golang/net) | `0.20.0` | `0.21.0` |
| [golang.org/x/oauth2](https://github.com/golang/oauth2) | `0.16.0` | `0.17.0` |
| [google.golang.org/api](https://github.com/googleapis/google-api-go-client) | `0.157.0` | `0.162.0` |


Updates `cloud.google.com/go/bigquery` from 1.58.0 to 1.59.0
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@bigquery/v1.58.0...bigquery/v1.59.0)

Updates `cloud.google.com/go/pubsub` from 1.34.0 to 1.36.1
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@pubsub/v1.34.0...pubsub/v1.36.1)

Updates `cloud.google.com/go/storage` from 1.36.0 to 1.37.0
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@pubsub/v1.36.0...spanner/v1.37.0)

Updates `github.com/Azure/azure-sdk-for-go/sdk/azcore` from 1.9.1 to 1.9.2
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](Azure/azure-sdk-for-go@sdk/azcore/v1.9.1...sdk/azcore/v1.9.2)

Updates `github.com/ClickHouse/clickhouse-go/v2` from 2.17.1 to 2.18.0
- [Release notes](https://github.com/ClickHouse/clickhouse-go/releases)
- [Changelog](https://github.com/ClickHouse/clickhouse-go/blob/main/CHANGELOG.md)
- [Commits](ClickHouse/clickhouse-go@v2.17.1...v2.18.0)

Updates `github.com/IBM/sarama` from 1.42.1 to 1.42.2
- [Release notes](https://github.com/IBM/sarama/releases)
- [Changelog](https://github.com/IBM/sarama/blob/main/CHANGELOG.md)
- [Commits](IBM/sarama@v1.42.1...v1.42.2)

Updates `github.com/apache/pulsar-client-go` from 0.11.1 to 0.12.0
- [Release notes](https://github.com/apache/pulsar-client-go/releases)
- [Changelog](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG.md)
- [Commits](apache/pulsar-client-go@v0.11.1...v0.12.0)

Updates `github.com/aws/aws-lambda-go` from 1.45.0 to 1.46.0
- [Release notes](https://github.com/aws/aws-lambda-go/releases)
- [Commits](aws/aws-lambda-go@v1.45.0...v1.46.0)

Updates `github.com/aws/aws-sdk-go` from 1.50.0 to 1.50.15
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](aws/aws-sdk-go@v1.50.0...v1.50.15)

Updates `github.com/getsentry/sentry-go` from 0.26.0 to 0.27.0
- [Release notes](https://github.com/getsentry/sentry-go/releases)
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md)
- [Commits](getsentry/sentry-go@v0.26.0...v0.27.0)

Updates `github.com/go-faker/faker/v4` from 4.2.0 to 4.3.0
- [Release notes](https://github.com/go-faker/faker/releases)
- [Commits](go-faker/faker@v4.2.0...v4.3.0)

Updates `github.com/jhump/protoreflect` from 1.15.4 to 1.15.6
- [Release notes](https://github.com/jhump/protoreflect/releases)
- [Commits](jhump/protoreflect@v1.15.4...v1.15.6)

Updates `github.com/klauspost/compress` from 1.17.4 to 1.17.6
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](klauspost/compress@v1.17.4...v1.17.6)

Updates `github.com/sijms/go-ora/v2` from 2.8.6 to 2.8.7
- [Release notes](https://github.com/sijms/go-ora/releases)
- [Commits](sijms/go-ora@v2.8.6...v2.8.7)

Updates `github.com/twmb/franz-go` from 1.16.0 to 1.16.1
- [Changelog](https://github.com/twmb/franz-go/blob/master/CHANGELOG.md)
- [Commits](twmb/franz-go@v1.16.0...v1.16.1)

Updates `go.opentelemetry.io/otel` from 1.22.0 to 1.23.1
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.23.1)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace` from 1.22.0 to 1.23.1
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.23.1)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` from 1.22.0 to 1.23.1
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.23.1)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` from 1.22.0 to 1.23.1
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.23.1)

Updates `go.opentelemetry.io/otel/sdk` from 1.22.0 to 1.23.1
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.23.1)

Updates `go.opentelemetry.io/otel/trace` from 1.22.0 to 1.23.1
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.23.1)

Updates `golang.org/x/net` from 0.20.0 to 0.21.0
- [Commits](golang/net@v0.20.0...v0.21.0)

Updates `golang.org/x/oauth2` from 0.16.0 to 0.17.0
- [Commits](golang/oauth2@v0.16.0...v0.17.0)

Updates `google.golang.org/api` from 0.157.0 to 0.162.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](googleapis/google-api-go-client@v0.157.0...v0.162.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/bigquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: cloud.google.com/go/pubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: github.com/ClickHouse/clickhouse-go/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/IBM/sarama
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: github.com/apache/pulsar-client-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/aws/aws-lambda-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: github.com/getsentry/sentry-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/go-faker/faker/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/jhump/protoreflect
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: github.com/sijms/go-ora/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: github.com/twmb/franz-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: go.opentelemetry.io/otel
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: go.opentelemetry.io/otel/trace
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <[email protected]>

* Mark version in CHANGELOG for RCs

* Update release notes script with release candidates section

* Migrate all aws components to v2 apis

* Add support for otlp sampling (redpanda-data#2364)

* add otlp sampling option

Signed-off-by: j0shthomas <[email protected]>

* slight docs change

Signed-off-by: j0shthomas <[email protected]>

* gen docs

Signed-off-by: j0shthomas <[email protected]>

* gofmt

Signed-off-by: j0shthomas <[email protected]>

* slight formatting

Signed-off-by: j0shthomas <[email protected]>

---------

Signed-off-by: j0shthomas <[email protected]>
Co-authored-by: joshuathomas97 <[email protected]>

* Tidy up some fields

* Add support for ARNs in kinesis components

* reduce kinesis output wait timer

* Favour faster error return for kinesis output connect

* Fix message structured set to null

* Add CosmosDB components and gocosmos SQL driver

This is a revamp of redpanda-data#1583, where I did the initial exploration to
add these components. Since then, [gocosmos](https://pkg.go.dev/github.com/microsoft/gocosmos)
became an official Microsoft SQL driver for CosmosDB and it is
currently marked as experimental, so I decided to include it as
well because it comes with some extra goodies, such as support for
[hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys)
and [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query),
which are not currently supported by the [`azcosmos`](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos)
driver.

Signed-off-by: Mihai Todor <[email protected]>

* Handle empty pages when filtering on large tables

Signed-off-by: Marco Amador <[email protected]>

* Update some of the docs fields

* Fix output resources blocking unit tests

* WIP: Add opensearch output

* Finish opensearch output, update deps

* Reduce bloblang parser allocations

* Simplify delimited parser

* Bumped up runc version

Signed-off-by: Anitha Gopinath <[email protected]>

* Bumped up runc version

Signed-off-by: Anitha Gopinath <[email protected]>

* Update CHANGELOG

* Add explicit slug to add components

* Fix template config parsing

* Fix non-generalised docs defaults

* Migrate parsers to use generics

* Fix diff and patch docs

* Add credit property to amqp1 input (redpanda-data#2357)

* Add support for max_in_flight messages (aka credit) for amqp_1 input

Signed-off-by: Andreas Habel <[email protected]>

* Update CHANGELOG

Signed-off-by: Andreas Habel <[email protected]>

* Changed name of description var to fix linting

Signed-off-by: Andreas Habel <[email protected]>

* Updated CHANGELOG and default value of credit

Signed-off-by: Andreas Habel <[email protected]>

* Updated docs

Signed-off-by: Andreas Habel <[email protected]>

---------

Signed-off-by: Andreas Habel <[email protected]>
Signed-off-by: Andreas Habel <[email protected]>
Co-authored-by: Andreas Habel <[email protected]>

* Prevent redundant bootstrap reattempts with studio sync

* Fix file input race condition on errored close

* Tidy up cockroachdb changefeed shutdown

* Pre-emptively recover sqlx close panics

* Add some examples to http_server input

* added config param inbox_prefix for the NATS connection

Signed-off-by: Tom Hieß <[email protected]>

* Add root level if statements

* added support for `new` and `last_per_subject` delivery strategies

Signed-off-by: Daan Gerits <[email protected]>

* Add sql cache

* MongoDB input - add batching

Signed-off-by: Brad Anderson <[email protected]>

* Tidy up docs and fields

* Fix double fire on @hourly generate intervals

* Fix fmt interpretted imports of configs

* Support extended JSON in mongodb mappings

* Rename url field on otel tracer targets to address

* Bump clickhouse-go dependecy to v2.21.1

Signed-off-by: Kevin Joiner <[email protected]>

* Add idempotent_write field to kafka output

* Refactor and tidy up nats connections

* Tidy up all AWS integration tests

* Update CHANGELOG

* Move all docker examples to the new repo

* Avoid emitting empty batches from table storage input

* Remove toolchain version pin

* Refactor nats kv cache with new connection stuff

* Update CHANGELOG

* Skip processing SFTP file if the cache.Get returns an unknown error

Signed-off-by: Daniel Mateus Pires <[email protected]>

* Add field error_patterns to the drop_on output

* Tidy up redis scan input tests and docs

* Apply goimports

* Add warpstream as sponsor

* Avoid redirect in curl command

* Fix unarchive processor lint regression

* Loop sftp files and move on when theyre missing

* Add auto_replay_nacks field to auto retry inputs

* Tighten up processor context exits

* fix vulnerability GO-2024-2611 / Infinite loop in JSON unmarshaling in google.golang.org/protobuf

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix GO-2024-2605 / SQL injection in github.com/jackc/pgx/v4

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix GO-2024-2606 / SQL injection in github.com/jackc/pgproto3 and github.com/jackc/pgx

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix dependency github.com/jackc/pgconn that it is incompatible with current method signatures and fix CVE-2024-27304

Signed-off-by: Tiago Peczenyj <[email protected]>

* update dependency github.com/jackc/pgx/v4

Signed-off-by: Tiago Peczenyj <[email protected]>

* Fix DockerHub REST API examples

Fixes redpanda-data#2422.

Looks like DockerHub now rejects the default
`Content-Type:application/octet-stream` header which Benthos sends
by default with HTTP error 406.

Signed-off-by: Mihai Todor <[email protected]>

* Bump github.com/nats-io/nats-server/v2 from 2.8.4 to 2.9.23

Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.8.4 to 2.9.23.
- [Release notes](https://github.com/nats-io/nats-server/releases)
- [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml)
- [Commits](nats-io/nats-server@v2.8.4...v2.9.23)

---
updated-dependencies:
- dependency-name: github.com/nats-io/nats-server/v2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

* apply gofumpt -l -w on each source file

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply gofumpt -l -w -extra on each source file

Signed-off-by: Tiago Peczenyj <[email protected]>

* enable linter nolintlint and apply fix

Signed-off-by: Tiago Peczenyj <[email protected]>

* fun make fmt

Signed-off-by: Tiago Peczenyj <[email protected]>

* add linter bodyclose

Signed-off-by: Tiago Peczenyj <[email protected]>

* add linter rowserrcheck and fix some cases

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix bug "http: read on closed response body"

Signed-off-by: Tiago Peczenyj <[email protected]>

* enable linter mirror and autofix code

Signed-off-by: Tiago Peczenyj <[email protected]>

* add linter predeclared and fix code

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix dupwords -- will not add this linter but fix some notorious issues

Signed-off-by: Tiago Peczenyj <[email protected]>

* enable lint tenv and manually fix issues

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix revive superfluous-else lint issues manually

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix revive indent-error-flow lint issues manually

Signed-off-by: Tiago Peczenyj <[email protected]>

* Add retry processor

* Update docs

* Update docs

* Fix deadlock in nats_kv cache

The process will exit if connect() fails, but it's probably best
to up resources anyway.

Signed-off-by: Mihai Todor <[email protected]>

* Change sqs visibility timeout error logs to debug level

* Remove unused fields from nats_kv processor

Signed-off-by: Mihai Todor <[email protected]>

* fix386build

Signed-off-by: Girish Bhutiya <[email protected]>

* Make input/output init logs for consistent

* Fix schema registry encode race condition

* chore: fix function names in comment

Signed-off-by: mountcount <[email protected]>

* add noop cache (redpanda-data#2453)

Signed-off-by: Tiago Peczenyj <[email protected]>

* Update CHANGELOG

* Prepare for pr 2458: fix misspell (redpanda-data#2467)

* fix misspell

Signed-off-by: Tiago Peczenyj <[email protected]>

* run linter misspell with autofix option

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix doc

Signed-off-by: Tiago Peczenyj <[email protected]>

* Revert "fix doc"

This reverts commit c5370a3.

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix doc

Signed-off-by: Tiago Peczenyj <[email protected]>

---------

Signed-off-by: Tiago Peczenyj <[email protected]>

* Prepare for pr 2458: try fix govet nilness manually (redpanda-data#2480)

* try fix govet nilness manually

Signed-off-by: Tiago Peczenyj <[email protected]>

* Update input.go

* Update processor_log.go

---------

Signed-off-by: Tiago Peczenyj <[email protected]>
Co-authored-by: Ashley Jeffs <[email protected]>

* Prepare for pr 2458: add linter testifylint (redpanda-data#2475)

* apply testfylint nil-compare and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testfylint compares and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testifylint error-is-as and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* bump golangci-lint version

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testifylint bool-compares and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testifylint empty and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testifylint len and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testifylint expected-actual and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply testifylint error-nil and run autofix via cmd

Signed-off-by: Tiago Peczenyj <[email protected]>

---------

Signed-off-by: Tiago Peczenyj <[email protected]>
Co-authored-by: Ashley Jeffs <[email protected]>

* Prepare for pr 2458: fix some gocritic lint issues (redpanda-data#2474)

* run linter gocritic equalFold with autofix

Signed-off-by: Tiago Peczenyj <[email protected]>

* run linter gocritic httpNoBody with autofix

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply gocritic argOrder autofix

Signed-off-by: Tiago Peczenyj <[email protected]>

* apply gocritic sloppylen manual fix

Signed-off-by: Tiago Peczenyj <[email protected]>

* log internal error on rows.Err before closing

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix gocritic emptyStringTest part 1

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix issues realted to gocritic emptyStringTest

Signed-off-by: Tiago Peczenyj <[email protected]>

---------

Signed-off-by: Tiago Peczenyj <[email protected]>
Co-authored-by: Ashley Jeffs <[email protected]>

* More lints

* Prepare for pr 2458: add linter perfsprint (redpanda-data#2470)

* add linter perfsprint

Signed-off-by: Tiago Peczenyj <[email protected]>

* fix .golangci.yaml

Signed-off-by: Tiago Peczenyj <[email protected]>

* try new version

Signed-off-by: Tiago Peczenyj <[email protected]>

---------

Signed-off-by: Tiago Peczenyj <[email protected]>
Co-authored-by: Ashley Jeffs <[email protected]>

* Create new scanner type for JSON documents

Signed-off-by: gregfurman <[email protected]>

* Fix bloblang integer comparisons

* fix GO-2024-2687 HTTP/2 CONTINUATION flood in net/http

Signed-off-by: Tiago Peczenyj <[email protected]>

* Fix `azure_blob_storage` input pagination (redpanda-data#2491)

* Fix azure_blob_storage input pagination

Avoid fetching all pages in `newAzureTargetReader()`.

Signed-off-by: Mihai Todor <[email protected]>

* Rework azure_blob_storage input paging implementation

It looks like we can cache the pager directly.

Signed-off-by: Mihai Todor <[email protected]>

* Remove unused parameter.

Signed-off-by: Mihai Todor <[email protected]>

---------

Signed-off-by: Mihai Todor <[email protected]>

* Add targets_input field to azure_blob_storage input

* Return tracked batch errors on nacked split batches

* Fix docs and handle phantom messages

* Enhance `nats_kv` processor (redpanda-data#2495)

* Add `keys_filter` field to nats_kv processor

This is a slightly-breaking change to the nats_kv processor which
is still beta. It alters the behaviour of the `keys` operation
which now returns an array of keys instead of a batch of messages,
as discussed with @codegangsta.

I'm a bit concerned that the history operation still returns a
batch of messages, which might confuse people. Happy to modify
that one as well to return an array of objects if you'd like. In
the current implementation, some of the fields of each history
item are emitted as message metadata, which is a bit odd anyway,
since they're not identical across the batch...

Signed-off-by: Mihai Todor <[email protected]>

* Log error in debug mode if closing the key watcher fails

Signed-off-by: Mihai Todor <[email protected]>

* Switch the `nats_kv` components to the new jetstream API

Signed-off-by: Mihai Todor <[email protected]>

* Refactor the `nats_kv` `keys` operation

Use the `key` field for filtering keys instead of adding an extra
`keys_filter` field.

Signed-off-by: Mihai Todor <[email protected]>

* Adjust the `nats_kv` `history` operation

Return an array of objects instead of a batch of messages.

Signed-off-by: Mihai Todor <[email protected]>

* Add `timeout` field to the `nats_kv` processor

Signed-off-by: Mihai Todor <[email protected]>

* Update the changelog

Signed-off-by: Mihai Todor <[email protected]>

---------

Signed-off-by: Mihai Todor <[email protected]>
Co-authored-by: Ashley Jeffs <[email protected]>

* Add reject_errored output

* Fix some comments

Signed-off-by: fanqiaojun <[email protected]>

* Fix parse_form_url_encoded bloblang method

Use correct data type when a query parameter is repeated multiple
times.

Signed-off-by: Mihai Todor <[email protected]>

* Add nats_reply_subject to NATS message metadata

Signed-off-by: Robby Zambito <[email protected]>

* Update docs

* Tidy up nats r/r proc

* Whoops, remove duplicate conn tail fields

* Tidy up json_documents scanner

* Cleanup redundant fields in `nats_kv` processor

Signed-off-by: Mihai Todor <[email protected]>

* Remove redundant wasm import from x_benthos_extra.go

I'm not sure why this was added here, but it's also imported via
`public/components/all/package.go`, which imports
`github.com/benthosdev/benthos/v4/public/components/wasm`.

Signed-off-by: Mihai Todor <[email protected]>

* Fix sensitive field scrubber

Don't throw an error when scrubbing empty password fields.

Signed-off-by: Mihai Todor <[email protected]>

* Update docs for the `branch` processor

Clarify how the `@` operator can be used to reference the metadata
of the origin message and replace the deprecated `meta()` function
with `metadata()`.

Signed-off-by: Mihai Todor <[email protected]>

* Add tests and fix return message as messagebatch

Signed-off-by: John Hooks <[email protected]>

* Fix link to NATS JWT docs

Signed-off-by: Florian Schäfer <[email protected]>

* Fix lints

* Move checkpoint package out

* Move integration test package into public

* Use public shutdown package

* Add some TODO V5 markers in prep

* Add batch sqs ack unit test

* Update CHANGELOG

* fix: Exit after process messge batch

* fix: lint

---------

Signed-off-by: Peter Elmers <[email protected]>
Signed-off-by: Marco Amador <[email protected]>
Signed-off-by: Geert G <[email protected]>
Signed-off-by: Marcel Hauf <[email protected]>
Signed-off-by: Anton Frank <[email protected]>
Signed-off-by: Mihai Todor <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: j0shthomas <[email protected]>
Signed-off-by: Anitha Gopinath <[email protected]>
Signed-off-by: Andreas Habel <[email protected]>
Signed-off-by: Andreas Habel <[email protected]>
Signed-off-by: Tom Hieß <[email protected]>
Signed-off-by: Daan Gerits <[email protected]>
Signed-off-by: Brad Anderson <[email protected]>
Signed-off-by: Kevin Joiner <[email protected]>
Signed-off-by: Daniel Mateus Pires <[email protected]>
Signed-off-by: Tiago Peczenyj <[email protected]>
Signed-off-by: Girish Bhutiya <[email protected]>
Signed-off-by: mountcount <[email protected]>
Signed-off-by: gregfurman <[email protected]>
Signed-off-by: fanqiaojun <[email protected]>
Signed-off-by: Robby Zambito <[email protected]>
Signed-off-by: John Hooks <[email protected]>
Signed-off-by: Florian Schäfer <[email protected]>
Co-authored-by: Peter Elmers <[email protected]>
Co-authored-by: Marco Amador <[email protected]>
Co-authored-by: Geert G <[email protected]>
Co-authored-by: Marcel Hauf <[email protected]>
Co-authored-by: Ashley Jeffs <[email protected]>
Co-authored-by: Anton Frank <[email protected]>
Co-authored-by: Dan Goodman <[email protected]>
Co-authored-by: Mihai Todor <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joshua Thomas <[email protected]>
Co-authored-by: joshuathomas97 <[email protected]>
Co-authored-by: Arunachalam Lakshmanan <[email protected]>
Co-authored-by: Anitha Gopinath <[email protected]>
Co-authored-by: andreas-habel <[email protected]>
Co-authored-by: Andreas Habel <[email protected]>
Co-authored-by: Tom Hieß <[email protected]>
Co-authored-by: Daan Gerits <[email protected]>
Co-authored-by: Brad Anderson <[email protected]>
Co-authored-by: Kevin Joiner <[email protected]>
Co-authored-by: Daniel Mateus Pires <[email protected]>
Co-authored-by: Tiago Peczenyj <[email protected]>
Co-authored-by: Girish Bhutiya <[email protected]>
Co-authored-by: mountcount <[email protected]>
Co-authored-by: gregfurman <[email protected]>
Co-authored-by: fanqiaojun <[email protected]>
Co-authored-by: Robby Zambito <[email protected]>
Co-authored-by: John Hooks <[email protected]>
Co-authored-by: Florian Schäfer <[email protected]>
Co-authored-by: junior.lima <[email protected]>
  • Loading branch information
1 parent 0ccfa50 commit 2b6ce17
Show file tree
Hide file tree
Showing 482 changed files with 7,595 additions and 3,820 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
- name: Lint
uses: golangci/golangci-lint-action@v4
with:
version: v1.54.1
version: v1.57.1
args: --timeout 30m cmd/... internal/... public/...

# TODO: Remove this after https://github.com/golangci/golangci-lint/issues/3094 is addressed
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ site
TODO.md
release_notes.md
.idea
.vscode
.vscode
.op
38 changes: 35 additions & 3 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,26 @@ issues:
include:
- EXC0012
- EXC0014

exclude-rules:
- path: _test.go
linters:
- bodyclose
- perfsprint
linters-settings:
revive:
enable-all-rules: false
rules:
- name: superfluous-else
errcheck:
exclude-functions:
- (*github.com/benthosdev/benthos/v4/internal/batch.Error).Failed
- (*github.com/benthosdev/benthos/v4/public/service.BatchError).Failed
govet:
enable-all: true
disable:
- fieldalignment
- deepequalerrors
- shadow
gocritic:
enabled-tags:
- diagnostic
Expand All @@ -32,7 +46,17 @@ linters-settings:
# main:
# deny:
# - pkg: "io/ioutil"

testifylint:
disable-all: true
enable:
- nil-compare
- compares
- error-is-as
- bool-compare
- empty
- len
- expected-actual
- error-nil
linters:
disable-all: true
enable:
Expand All @@ -51,11 +75,19 @@ linters:
- goimports
# gocritic is very slow (golangci-lint v1.52.0)
# - gocritic
# - revive
- revive
- unconvert
- durationcheck
# - depguard
# - bodyclose
# - gosec
# - misspell
# - prealloc
- perfsprint
- testifylint
- tenv
- predeclared
- mirror
- rowserrcheck
- bodyclose
- nolintlint
52 changes: 34 additions & 18 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,40 @@ Changelog

All notable changes to this project will be documented in this file.

## 4.27.0 - 2024-04-23

### Added

- New `nats_kv` cache type.
- The `nats_jetstream` input now supports `last_per_subject` and `new` deliver fallbacks.
- Field `error_patterns` added to the `drop_on` output.
- New `redis_scan` input type.
- Field `auto_replay_nacks` added to all inputs that traditionally automatically retry nacked messages as a toggle for this behaviour.
- New `retry` processor.
- New `noop` cache.
- Field `targets_input` added to the `azure_blob_storage` input.
- New `reject_errored` output.
- New `nats_request_reply` processor.
- New `json_documents` scanner.

### Fixed

- The `unarchive` processor no longer yields linting errors when the format `csv:x` is specified. This is a regression introduced in v4.25.0.
- The `sftp` input will no longer consume files when the watcher cache returns an error. Instead, it will reattempt the file upon the next poll.
- The `aws_sqs` input no longer logs error level logs for visibility timeout refreshing errors.
- The `nats_kv` processor now allows [nats wildcards](https://docs.nats.io/nats-concepts/subjects#wildcards) for the `keys` operation.
- The `nats_kv` processor `keys` operation now returns a single message with an array of found keys instead of a batch of messages.
- The `nats_kv` processor `history` operation now returns a single message with an array of objects containing the record fields instead of a batch of messages.
- Field `timeout` added to the `nats_kv` processor to specify the maximum period to wait on an operation before aborting and returning an error.
- Bloblang comparison operators (`>`, `<`, `<=`, `>=`) now match the precision of the compared integers when applicable.
- The `parse_form_url_encoded` Bloblang method no longer produces results with an unknown data type for repeated query parameters.
- The `echo` CLI command no longer fails to sanitise configs when encountering an empty `password` field.

### Changed

- The log events from all inputs and outputs when they first connect have been made more consistent and no longer contain any information regarding the nature of their connections.
- Splitting message batches with a `split` processor (or custom plugins) no longer results in downstream error handling loops around nacks. This was previously implemented as a feature to ensure unbounded expanded and split batches don't flood downstream services in the event of a minority of errors. However, introducing more clever origin tracking of errored messages has eliminated the need for this undocumented behaviour.

## 4.26.0 - 2024-03-18

### Added
Expand Down Expand Up @@ -34,24 +68,6 @@ All notable changes to this project will be documented in this file.

### Added

- Field `credit` added to the `amqp_1` input to specify the maximum number of unacknowledged messages the sender can transmit.
- Bloblang now supports root-level `if` statements.
- New experimental `sql` cache.

### Changed

- The default value of the `amqp_1.credit` input has changed from `1` to `64`.

## 4.25.1 - 2024-03-01

### Fixed

- Fixed a regression in v4.25.0 where [template based components](https://www.benthos.dev/docs/configuration/templating) were not parsing correctly from configs.

## 4.25.0 - 2024-03-01

### Added

- Field `address_cache` added to the `socket_server` input.
- Field `read_header` added to the `amqp_1` input.
- All inputs with a `codec` field now support a new field `scanner` to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information [check out the scanners page](https://www.benthos.dev/docs/components/scanners/about).
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ PATHINSTBIN = $(DEST_DIR)/bin
PATHINSTTOOLS = $(DEST_DIR)/tools
PATHINSTSERVERLESS = $(DEST_DIR)/serverless
PATHINSTDOCKER = $(DEST_DIR)/docker
DOCKER_IMAGE ?= jeffail/benthos
DOCKER_IMAGE ?= ghcr.io/benthosdev/benthos

VERSION := $(shell git describe --tags || echo "v0.0.0")
VER_CUT := $(shell echo $(VERSION) | cut -c2-)
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@ Do you like looking at stuff? Get angry and smash things when you're forced to r
Grab a binary for your OS from [here.][releases] Or use this script:
```shell
curl -Lsf https://sh.benthos.dev | bash
curl -Lsf https://www.benthos.dev/sh/install | bash
```

Or pull the docker image:

```shell
docker pull jeffail/benthos
docker pull ghcr.io/benthosdev/benthos
```

Benthos can also be installed via Homebrew:
Expand All @@ -87,10 +87,10 @@ Or, with docker:

```shell
# Using a config file
docker run --rm -v /path/to/your/config.yaml:/benthos.yaml jeffail/benthos
docker run --rm -v /path/to/your/config.yaml:/benthos.yaml ghcr.io/benthosdev/benthos

# Using a series of -s flags
docker run --rm -p 4195:4195 jeffail/benthos \
docker run --rm -p 4195:4195 ghcr.io/benthosdev/benthos \
-s "input.type=http_server" \
-s "output.type=kafka" \
-s "output.kafka.addresses=kafka-server:9092" \
Expand Down
10 changes: 2 additions & 8 deletions cmd/tools/benthos_docs_gen/bloblang_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,7 @@ func TestFunctionExamples(t *testing.T) {
require.NoError(t, err)

key := "BENTHOS_TEST_BLOBLANG_FILE"
os.Setenv(key, tmpJSONFile.Name())
t.Cleanup(func() {
os.Unsetenv(key)
})
t.Setenv(key, tmpJSONFile.Name())

for _, spec := range query.FunctionDocs() {
spec := spec
Expand Down Expand Up @@ -89,10 +86,7 @@ func TestMethodExamples(t *testing.T) {
require.NoError(t, err)

key := "BENTHOS_TEST_BLOBLANG_SCHEMA_FILE"
os.Setenv(key, tmpJSONFile.Name())
t.Cleanup(func() {
os.Unsetenv(key)
})
t.Setenv(key, tmpJSONFile.Name())

for _, spec := range query.MethodDocs() {
spec := spec
Expand Down
12 changes: 7 additions & 5 deletions config/examples/track_benthos_downloads.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
pipeline:
threads: 20
processors:
- bloblang: 'root = {}'
- mapping: 'root = {}'
- workflow:
meta_path: results
order: [ [ dockerhub, github, homebrew ] ]
Expand All @@ -17,7 +17,9 @@ processor_resources:
url: https://hub.docker.com/v2/repositories/jeffail/benthos/
verb: GET
retries: 0
- bloblang: |
headers:
Content-Type: application/json
- mapping: |
root.source = "docker"
root.dist = "docker"
root.download_count = this.pull_count
Expand All @@ -34,7 +36,7 @@ processor_resources:
url: https://api.github.com/repos/benthosdev/benthos/releases
verb: GET
retries: 0
- bloblang: |
- mapping: |
root = this.map_each(release -> release.assets.map_each(asset -> {
"source": "github",
"dist": asset.name.re_replace_all("^benthos-?((lambda_)|_)[0-9\\.]+(-rc[0-9]+)?_([^\\.]+).*", "$2$4"),
Expand All @@ -44,7 +46,7 @@ processor_resources:
- unarchive:
format: json_array
- resource: metric_gauge
- bloblang: 'root = if batch_index() != 0 { deleted() }'
- mapping: 'root = if batch_index() != 0 { deleted() }'

- label: homebrew
branch:
Expand All @@ -55,7 +57,7 @@ processor_resources:
url: https://formulae.brew.sh/api/formula/benthos.json
verb: GET
retries: 0
- bloblang: |
- mapping: |
root.source = "homebrew"
root.dist = "brew"
root.download_count = this.analytics.install.30d.benthos
Expand Down
Loading

0 comments on commit 2b6ce17

Please sign in to comment.