Pipeline dispatcher

[sunng87]

What problem does the new feature solve?

Currently we can store all the logs/events from one API call to a single table. However, it's possible that several different types of logs share the same source.

What does the feature do?

So in our pipeline, we need a mechanism to dispatch data into different pipelines, and eventually different tables, based on certain fields like type, or any other data that user can extract from the pipeline.

processors:

dispatcher:
  field: type
  routes:
    - value: http
      pipeline: http_pipeline
      table_name: ${prefix}_http
    - value: database
      pipeline: database_pipeline
      table_name: ${prefix}_database
    - default: yes
      pipeline: default_pipeline
      table_name: ${prefix}_events

flowchart TD
    A[entry_pipeline] --> D(dispatcher)
    D --> |type=http| B[http_pipeline]
    D --> |type=database| C[database_pipeline]
    D --> |type=unknown| E[default_pipeline]

Loading

Implementation challenges

No response

[killme2008]

Should we support regex matching for values? Users may not be able to specify all possible values and instead provide a regex expression.

[sunng87]

I think it's possible with processors. They can process all possible values and generate fixed type