Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_kinesis_streams: add custom port support for Kinesis output plugin #9317

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

azalio
Copy link

@azalio azalio commented Aug 30, 2024

This PR enhances the Kinesis output plugin by adding support for custom port configuration. The changes include:

  1. Added a new configuration option 'port' to allow users to specify a custom port for Kinesis connections.

  2. Modified the plugin initialization to use the custom port when specified, defaulting to the standard port (443) when not provided.

  3. Updated the connection logic to utilize the custom port when establishing connections to Kinesis.

  4. Added error handling to validate the port number, ensuring it's within a valid range (1-65535).

  5. Updated documentation to reflect the new 'port' configuration option.

  6. Added new unit tests to verify the functionality of the custom port feature:

    • Test for default port behavior
    • Test for successful custom port configuration
    • Test for handling invalid port numbers

These changes provide users with more flexibility in configuring their Kinesis output, especially in environments where non-standard ports are required.

The implementation maintains backwards compatibility, as the default behavior remains unchanged when no custom port is specified.

Closes #9160

@azalio
Copy link
Author

azalio commented Aug 30, 2024

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[SERVICE]
    daemon       Off
    log_level    info

[INPUT]
    Name              tail
    Path              /var/log/kubernetes/audit/audit.log
    Tag               audit_log

[OUTPUT]
    Name kinesis_streams
    Match           audit_log
    region          pre-prod
    stream          /pre-prod/audit-trails-input/topics/data-plane/k8s
    endpoint        https://hostname.cloud-preprod.example.com
    port            8443

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1450


@azalio
Copy link
Author

azalio commented Aug 30, 2024

@cosmo0920 Please, could you do a review.

tests/runtime/out_kinesis.c Outdated Show resolved Hide resolved
This patch adds the ability to set a custom port for the Kinesis Streams
output plugin. It introduces a new configuration option 'Port' that allows
users to specify a non-standard port when connecting to the Kinesis service.

The implementation includes:
- Using the port number set in the output instance's host structure
- Falling back to the default HTTPS port (443) if no port is specified
- Validating the port number to ensure it's within the valid range (1-65535)
- Adding appropriate debug and error logging

This feature is useful for scenarios involving proxies or custom network
configurations. It has been tested with various port configurations,
including the default, custom ports, and invalid inputs.

Signed-off-by: Mikhail [azalio] Petrov <[email protected]>
This commit introduces three new test cases for the kinesis_streams output plugin:

1. flb_test_kinesis_default_port: Verifies that the plugin works correctly
   with the default port configuration.

2. flb_test_kinesis_custom_port: Ensures that the plugin can be configured
   to use a custom port (8443 in this case) and still function properly.

3. flb_test_kinesis_invalid_port: Checks that the plugin fails to start
   when an invalid port number (99999) is specified.

These tests enhance the coverage of the kinesis_streams plugin by validating
its behavior with different port configurations, including error handling
for invalid inputs.

Signed-off-by: Mikhail [azalio] Petrov <[email protected]>
@azalio azalio changed the title Kinesis custom port option out_kinesis_streams: add custom port support for Kinesis output plugin Sep 2, 2024
@cosmo0920
Copy link
Contributor

I kicked off the testing workflows. Waiting for the results.

@azalio
Copy link
Author

azalio commented Sep 2, 2024

I kicked off the testing workflows. Waiting for the results.

As I see it, all the checks have been successfully completed. Could you merge PR?

@cosmo0920 cosmo0920 added this to the Fluent Bit v3.1-next milestone Sep 2, 2024
@cosmo0920
Copy link
Contributor

cosmo0920 commented Sep 2, 2024

I added 3.1-next v3.2.0 milestone. It will be queued for the reviewing queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow you to specify the port when working with the kinesis plugin.
2 participants