Splitting one input into multiple outputs

[MayurVirkar]

Hello,
Basically, I have one input record which is JSON of array of 1000 records, I need to split them into individual 1000 records and run filters on them and insert them individually into ELK.
How can I do it?

[f4t]

Hi, I'm interested by the same feature.
Based on my understanding of the code unfortunately it looks like Gogstash filters only allow exactly 1 (one) event out (as in one event in -> one event out).

Can anyone confirm this ?

I have several use-cases where it'd be useful for filters to allow either zero, one, or multiple events out:

• "splitting" as requested by Mayur above is one of those where we would let multiple events out of the filter.
• "dropping" events would also be a use-case for a filter that returns zero events (although this can already be done through conditional outputs I feel it might be interesting to remove unnecessary processing of events we want to discard).
• Lastly, I'd like to implement event-pairing filters, similar to Logstash "ellapsed" and "aggregate" filters. Those filters hold events in a buffer before releasing the combined result, therefore such filters might need to return zero event first, then two at once.

[tsaikd]

At the beginning, it's designed to process logs one by one. If you want to handle M inputs and N outputs, M and N are unsigned integers, some interfaces need to be changed.

[helgeolav]

If you need something to split your input have a look at Gollum, also written in go.

Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.

It will not solve all of your issues but see if you can use it for something.