Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] core: signal blob implementation #9326

Draft
wants to merge 19 commits into
base: master
Choose a base branch
from
Draft

[DRAFT] core: signal blob implementation #9326

wants to merge 19 commits into from

Commits on Sep 16, 2024

  1. sched: add coroutine support to timer callback 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    b7510a0 View commit details
    Browse the repository at this point in the history

  2. input_event: add support for Blob signal 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    1c93041 View commit details
    Browse the repository at this point in the history

  3. input_chunk: register chunks blob type 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    67d3296 View commit details
    Browse the repository at this point in the history

  4. input: use FLB_INLINE macro 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    2404460 View commit details
    Browse the repository at this point in the history

  5. engine_macros: adjust scheduler macros 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    4996a26 View commit details
    Browse the repository at this point in the history

  6. event: map event type blobs as chunk type blobs 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    19618c3 View commit details
    Browse the repository at this point in the history

  7. output_thread: add plugin callback for worker initialization 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    86d325b View commit details
    Browse the repository at this point in the history

  8. utils: add new function to read files by offset 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    bee712b View commit details
    Browse the repository at this point in the history

  9. router: allow routing for blob input types 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    cefd067 View commit details
    Browse the repository at this point in the history

  10. sqldb: commit before closing handler 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    9cdc593 View commit details
    Browse the repository at this point in the history

  11. output: register output blob 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    332988f View commit details
    Browse the repository at this point in the history

  12. input_blob: add new interface to register blobs 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    63c8b8b View commit details
    Browse the repository at this point in the history

  13. input_chunk: event type blob support 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    6b831ae View commit details
    Browse the repository at this point in the history

  14. in_blob: new plugin to support large binary files 

    Recent patch series add support to process/route large binary files through a zero-copy
strategy. This new in_blob plugin allows to scan a path from the file system and register
files that matched the pattern.

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: blob
      path: '~/logs/blob/*'
      database_file: blob.db

  outputs:
    - name: stdout
      match: '*'

    - name:           azure_blob
      match:          '*'
      path:           kubernetes
      container_name: blobs
      auto_create_container: on
      database_file: azure.db
      part_size: 4M
      upload_parts_timeout: 1s
      workers: 10

Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    6901e6f View commit details
    Browse the repository at this point in the history

  15. out_azure_blob: add support to upload large binary files 

    The recent changes in Fluent Bit, allows to process Blob signal types which
represents large binary files. When a blob arrives to the plugin, it's enqueued
and processed through parts and uploaded as a Block Blob. Part sizes are configurable
and survives service restart.

example usage:

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: blob
      path: '~/logs/blob/*'
      database_file: blob.db

  outputs:
    - name: stdout
      match: '*'

    - name:           azure_blob
      match:          '*'
      path:           kubernetes
      container_name: blobs
      auto_create_container: on
      database_file: azure.db
      part_size: 4M
      upload_parts_timeout: 1s
      workers: 10
      account_name: abcdefghijk
      shared_key: asdkljaskldjaskldjaskldjasioduasoudaskldjaskld
      tls: on

Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    4a99c4c View commit details
    Browse the repository at this point in the history

  16. build: register in_blob 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    7bd6bd1 View commit details
    Browse the repository at this point in the history

  17. out_stdout: support blob type (print targe file name) 

    Signed-off-by: Eduardo Silva <[email protected]>
    @edsiper
    edsiper committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    e7defa8 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. scheduler: fixed typo 

    Signed-off-by: leonardo-albertovich <[email protected]>
    @leonardo-albertovich
    leonardo-albertovich committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    84c84a7 View commit details
    Browse the repository at this point in the history

  2. scheduler: fixed typo 

    Signed-off-by: leonardo-albertovich <[email protected]>
    @leonardo-albertovich
    leonardo-albertovich committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    38e7ff0 View commit details
    Browse the repository at this point in the history