Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add byte-based ingestion limits to the queue and output #39776

Draft
wants to merge 121 commits into
base: main
Choose a base branch
from
Draft

Add byte-based ingestion limits to the queue and output #39776

wants to merge 121 commits into from

Commits on Apr 10, 2024

  1. cleanup

    @faec
    faec committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    a4019ec View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. cleanups

    @faec
    faec committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    a3d3757 View commit details
    Browse the repository at this point in the history

  2. break input sources up into separate helper functions

    @faec
    faec committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    597e0a5 View commit details
    Browse the repository at this point in the history

  3. finish helper function split

    @faec
    faec committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    0df748a View commit details
    Browse the repository at this point in the history

  4. rewrite the sqsReader main loop

    @faec
    faec committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    4b70900 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. simplify sqsReader loop

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    90d9e24 View commit details
    Browse the repository at this point in the history

  2. adjust variable names

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    5f94e9b View commit details
    Browse the repository at this point in the history

  3. remove unused parameter

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    b797261 View commit details
    Browse the repository at this point in the history

  4. createS3Lister -> createS3Poller

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    88f3980 View commit details
    Browse the repository at this point in the history

  5. remove unused error checks

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    9f32df6 View commit details
    Browse the repository at this point in the history

  6. cleanup

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    48ec82a View commit details
    Browse the repository at this point in the history

  7. make a wrapper for v2.Canceler that doesn't use an extra goroutine

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    58e084a View commit details
    Browse the repository at this point in the history

  8. remove unused parameter

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    1974f8f View commit details
    Browse the repository at this point in the history

  9. cleanup

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    646374c View commit details
    Browse the repository at this point in the history

  10. remove redundant helper

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    a43cae6 View commit details
    Browse the repository at this point in the history

  11. adjust variable names

    @faec
    faec committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    f46ef06 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2024

  1. remove extra index indirection in state lookup

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    d9be04b View commit details
    Browse the repository at this point in the history

  2. remove redundant sync.Map

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    5e1fbcc View commit details
    Browse the repository at this point in the history

  3. merge redundant state maps

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    c16a22f View commit details
    Browse the repository at this point in the history

  4. remove redundant state map

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    f07915a View commit details
    Browse the repository at this point in the history

  5. simplify s3Poller worker handling

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    0f483a3 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' of github.com:elastic/beats into awss3-cleanup

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    8916d91 View commit details
    Browse the repository at this point in the history

  7. simplify waitgroup handling / unused errors

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    a8cb6bd View commit details
    Browse the repository at this point in the history

  8. clean up context handling

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    78a7db4 View commit details
    Browse the repository at this point in the history

  9. adjust delay timer

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    edc1bd3 View commit details
    Browse the repository at this point in the history

  10. remove unused struct fields

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    1497be4 View commit details
    Browse the repository at this point in the history

  11. cleanup

    @faec
    faec committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    a3e0dc8 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Refactor cloudwatch worker task allocation

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    219e857 View commit details
    Browse the repository at this point in the history

  2. add unit tests for cloudwatchPoller.receive

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    977a0d3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of github.com:elastic/beats into cloudwatch-fix

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    71134f9 View commit details
    Browse the repository at this point in the history

  4. update changelog

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    6cf5506 View commit details
    Browse the repository at this point in the history

  5. make check

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    ff24571 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'cloudwatch-fix' into awss3-cleanup

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    5ec8a86 View commit details
    Browse the repository at this point in the history

  7. Remove unused custom semaphore helper

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    2a6abb8 View commit details
    Browse the repository at this point in the history

  8. cleanups in input.go

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    12a2a3c View commit details
    Browse the repository at this point in the history

  9. revert unintentional return value change

    @faec
    faec committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    dd29fa0 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Concurrency / error handling fixes in awss3

    @faec
    faec committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    4956db9 View commit details
    Browse the repository at this point in the history

  2. give the registry accessor its own mutex

    @faec
    faec committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    fc641e1 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. update tests

    @faec
    faec committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    4a9cb60 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of github.com:elastic/beats into s3-concurrency-fix

    @faec
    faec committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    959d557 View commit details
    Browse the repository at this point in the history

  3. make check

    @faec
    faec committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    3d93d22 View commit details
    Browse the repository at this point in the history

  4. lint

    @faec
    faec committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    7d6369f View commit details
    Browse the repository at this point in the history

  5. lint

    @faec
    faec committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    b4b5b28 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Merge branch 'main' of github.com:elastic/beats into s3-concurrency-fix

    @faec
    faec committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    45619e3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 's3-concurrency-fix' of github.com:faec/beats into awss3… 

    …-cleanup
    @faec
    faec committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    e88be00 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into s3-concurrency-fix

    @faec
    faec committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    1308a2d View commit details
    Browse the repository at this point in the history

  4. Merge branch 's3-concurrency-fix' into awss3-cleanup

    @faec
    faec committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    0abf663 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. cleaning up context use

    @faec
    faec committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    942ae03 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into s3input-cleanup

    @faec
    faec committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e84471b View commit details
    Browse the repository at this point in the history

  3. Merge branch 's3input-cleanup' into awss3-cleanup

    @faec
    faec committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    0289604 View commit details
    Browse the repository at this point in the history

  4. splitting S3 and SQS into distinct inputs internally

    @faec
    faec committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    2c084bb View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. splitting awss3 into two input objects

    @faec
    faec committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    dbe4691 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Merge branch 'main' of github.com:elastic/beats into s3input-cleanup

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    73d1465 View commit details
    Browse the repository at this point in the history

  2. Merge branch 's3input-cleanup' into awss3-cleanup

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    ad7d342 View commit details
    Browse the repository at this point in the history

  3. reorganize {s3,sqs}.go by adding {s3,sqs}_input.go for the code speci… 

    …fic to the input API
    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    e05c45d View commit details
    Browse the repository at this point in the history

  4. clean up sqs helpers

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    54f0a87 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into awss3-cleanup

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    122ee8c View commit details
    Browse the repository at this point in the history

  6. fix merge

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d396457 View commit details
    Browse the repository at this point in the history

  7. update tests

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    be54ac7 View commit details
    Browse the repository at this point in the history

  8. merge sqsReaderInput and sqsReader

    @faec
    faec committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    568d2b0 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. get tests building again

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    383c111 View commit details
    Browse the repository at this point in the history

  2. remove redundant fields

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    4b2ea11 View commit details
    Browse the repository at this point in the history

  3. more reorganization

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    de36816 View commit details
    Browse the repository at this point in the history

  4. organizing

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    3117334 View commit details
    Browse the repository at this point in the history

  5. reordering code

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    6b43ac5 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' of github.com:elastic/beats into awss3-cleanup

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    d38af54 View commit details
    Browse the repository at this point in the history

  7. clean up states initialization

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    3712af4 View commit details
    Browse the repository at this point in the history

  8. remove unused helper

    @faec
    faec committed May 1, 2024
    Configuration menu
    Copy the full SHA
    894ba4c View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. working on test updates

    @faec
    faec committed May 2, 2024
    Configuration menu
    Copy the full SHA
    31f3b95 View commit details
    Browse the repository at this point in the history

  2. fix benchmark tests

    @faec
    faec committed May 2, 2024
    Configuration menu
    Copy the full SHA
    7d12f0a View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. updating unit tests

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    f1b7761 View commit details
    Browse the repository at this point in the history

  2. fix remaining tests

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    fa22239 View commit details
    Browse the repository at this point in the history

  3. remove unused debug parameter

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    e253681 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' of github.com:elastic/beats into awss3-cleanup

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    5b922df View commit details
    Browse the repository at this point in the history

  5. remove commented code

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    1fef199 View commit details
    Browse the repository at this point in the history

  6. move helper function

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    800fc73 View commit details
    Browse the repository at this point in the history

  7. clean up aws client config modifiers

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    1694f0d View commit details
    Browse the repository at this point in the history

  8. reorder helper functions

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    63be523 View commit details
    Browse the repository at this point in the history

  9. reorder helper functions

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    1bae757 View commit details
    Browse the repository at this point in the history

  10. update comments

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    939c38f View commit details
    Browse the repository at this point in the history

  11. move log creation earlier

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    b032106 View commit details
    Browse the repository at this point in the history

  12. update comments

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    0995921 View commit details
    Browse the repository at this point in the history

  13. make check

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    5d9f731 View commit details
    Browse the repository at this point in the history

  14. Working on queue byte limits

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    a8323a4 View commit details
    Browse the repository at this point in the history

  15. Fill out more of the byte limits API, remove EntryID

    @faec
    faec committed May 3, 2024
    Configuration menu
    Copy the full SHA
    8ab68ed View commit details
    Browse the repository at this point in the history

Commits on May 4, 2024

  1. convert circular buffer indices to a helper type storing the absolute… 

    … event order
    @faec
    faec committed May 4, 2024
    Configuration menu
    Copy the full SHA
    60e1c2c View commit details
    Browse the repository at this point in the history

  2. Finish most byte bounds logic, add bulk_max_bytes to ES output

    @faec
    faec committed May 4, 2024
    Configuration menu
    Copy the full SHA
    812f87d View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. remove drop on cancel option

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    3fcb875 View commit details
    Browse the repository at this point in the history

  2. producer.Cancel -> producer.Close

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    47c07a6 View commit details
    Browse the repository at this point in the history

  3. remove OnDrop callbacks

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    f9d4c39 View commit details
    Browse the repository at this point in the history

  4. remove internal cancellation helpers

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    d398b46 View commit details
    Browse the repository at this point in the history

  5. remove the queue's shipper metrics hook

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    bad2498 View commit details
    Browse the repository at this point in the history

  6. remove unused fields and producer cancel tests

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    eca723b View commit details
    Browse the repository at this point in the history

  7. Merge branch 'remove-producer-cancel' into queue-metrics

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    c79cc3e View commit details
    Browse the repository at this point in the history

  8. fix merge

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    bffd70d View commit details
    Browse the repository at this point in the history

  9. moving metric ownership around

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    7409be1 View commit details
    Browse the repository at this point in the history

  10. plumbing for queue metrics

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    75ac0f4 View commit details
    Browse the repository at this point in the history

  11. flesh out queue observer internals

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    ca949b8 View commit details
    Browse the repository at this point in the history

  12. update queue filled percent

    @faec
    faec committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a279862 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Merge branch 'main' into queue-metrics

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    d0f1939 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into memqueue-byte-limits

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    ee2fa2b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into remove-producer-cancel

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    f512643 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'remove-producer-cancel' into memqueue-byte-limits

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    6d585bf View commit details
    Browse the repository at this point in the history

  5. clean up shipper metric hooks

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    517ffe1 View commit details
    Browse the repository at this point in the history

  6. use the metrics observer from the memqueue

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    4f6d02c View commit details
    Browse the repository at this point in the history

  7. configure gauges

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    fd18f4e View commit details
    Browse the repository at this point in the history

  8. report queue metrics from the disk queue

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    2f6ba9b View commit details
    Browse the repository at this point in the history

  9. fix disk queue initialization

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    ce2c287 View commit details
    Browse the repository at this point in the history

  10. outputObserver -> retryObserver

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    e70c13b View commit details
    Browse the repository at this point in the history

  11. move queue draining logic into the queue

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    e6dbb2d View commit details
    Browse the repository at this point in the history

  12. shadow acked var the simple way

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    afa3793 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'queue-metrics' into memqueue-byte-limits

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    f8157ac View commit details
    Browse the repository at this point in the history

  14. memqueue uses event or byte limits, not both

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    5a158ec View commit details
    Browse the repository at this point in the history

  15. fix byte vs event logic

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    24c5564 View commit details
    Browse the repository at this point in the history

  16. clean up FIFO handling

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    3da6d32 View commit details
    Browse the repository at this point in the history

  17. replace batchList implementation with FIFO helper

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    ac94a2b View commit details
    Browse the repository at this point in the history

  18. remove unrelated test change

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    d1ba6a1 View commit details
    Browse the repository at this point in the history

  19. remove unused error

    @faec
    faec committed May 30, 2024
    Configuration menu
    Copy the full SHA
    91e4534 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge branch 'main' of github.com:elastic/beats into queue-byte-limits

    @faec
    faec committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    5aa42eb View commit details
    Browse the repository at this point in the history

  2. fix merge + tests

    @faec
    faec committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    b585727 View commit details
    Browse the repository at this point in the history

  3. Add docs / parameter checks

    @faec
    faec committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    3643a8f View commit details
    Browse the repository at this point in the history