Dynamic Chain of Stateful Functions

[dipanjanb]

@misterbykl @thtmnisamnstr to farther ease daisy chaining of Stateful Functions in RTDL - I'm contemplating an approach that will help each function in the chain dynamially identify the next stage and write to the ingress topic of the next function in the chain

The chain is intended to look like

ingest -> pii-detection -> some-other-function -> some-more-function -> ingester

The beginning and end would be fixed - everything else would be dynamic.

The way this can be handled is as follows

• Add a environment variable MODULES for each of ingest and all in-between services. This would look like MODULES : pii-detection, some-other-function, some-more-function
• The ingress topic name of all in-between functions/services would follow a convention <function name>_in. e.g. pii-detection_in, some-other-function_in and so on. So module.yaml of these functions will have the fixed names
• Ingest and in-between services would load the environment variable and split it based on comma which would yield an array of function names
• For Ingest - it would simply take the 0-th name, append _in to the same and use the resulting string as the topic name to write to. Presently it is read from KAFKA_TOPIC environment variable. This variable would still remain and would be used if MODULES is empty i.e. no daisy-chaining is intended
• Each function - will go through the array elements till it comes across its own name
• At that point it should check if there are any more elements. Basically if its own index = length of array -1
• If yes, it should write out to ingress topic - which is what will remain fixed in module.yaml for ingester
• Else it should use the name of the next function in the array to dynamically construct name of the next topic

This way - if a user wants to use multiple functions and add/remove functions as required - they will just need to change the MODULES variable in the docker-compose.yaml file and not make changes to a number of module.yaml files within each function directory.

Hope I could explain the idea. Please voice queries if any.

@misterbykl I'll let you know what to do to adopt this for pii-detection

[dipanjanb]

In a future iteration - we will move this from docker-compose.yaml to stream configuration so that we will be able to have per-stream chains

[misterbykl]

We might also consider adding KAFKA key to the MODULES variable the same way you explained above. Providing function name as a KAFKA key to each function would work if message ordering OR partitioning is needed. Even if it is not needed now, it may be useful in the future since this is aimed for real-time events, and real-time events may be requested to be in order.

More info about KAFKA key:

Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go to the same partition in a topic. Kafka guarantees order within a partition, but not across partitions in a topic, so alternatively not providing a key - which will result in round-robin distribution across partitions - will not maintain such order.

ref: https://stackoverflow.com/a/29515696
ref: https://stackoverflow.com/a/61912094

[dipanjanb]

Hi Emre, We're already using a key. Partitioning is achieved using the time window configuration that can be specified in the stream configuration itself. We want to consiously stay away from ordering. The reasons are two-fold - Our target is the Data Lake and not cloud-based business or analytic applications, some off which require event ordering - Event ordering introduces lot of infrastructure-related complexities which, in the long run, impact the product negatively Thanks & Regards, Dipanjan

…

On Fri, 18 Mar 2022 at 14:29, Emre Baykal ***@***.***> wrote: We might also consider adding KAFKA *key* to the MODULES variable the same way you explained above. Providing function name as a KAFKA *key* to each function would work if message ordering OR partitioning is needed. Even if it is not needed now, it may be useful in the future since this is aimed for real-time events, and real-time events may be requested to be in order. More info about KAFKA *key*: Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go to the same partition in a topic. Kafka guarantees order within a partition, but not across partitions in a topic, so alternatively not providing a key - which will result in round-robin distribution across partitions - will not maintain such order. ref: https://stackoverflow.com/a/29515696 ref: https://stackoverflow.com/a/61912094 — Reply to this email directly, view it on GitHub <#20 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDSZMVTISY2NQWRILEBTVARAWNANCNFSM5RA653MA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[misterbykl]

I see. Thank you for the clarification.

[dipanjanb]

Welcome :-)

…

On Fri, 18 Mar 2022 at 18:37, Emre Baykal ***@***.***> wrote: I see. Thank you for the clarification. — Reply to this email directly, view it on GitHub <#20 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS5YVKP6XTRKFVZ5LGLVAR5ZNANCNFSM5RA653MA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[misterbykl]

Hi @dipanjanb. Any update regarding this feature? Let me know if anything needed.

[dipanjanb]

Hi Emre, Not yet - hope to start from this week. Thanks & Regards, Dipanjan

…

On Mon, 28 Mar, 2022, 11:28 pm Emre Baykal, ***@***.***> wrote: Hi @dipanjanb <https://github.com/dipanjanb>. Any update regarding this feature? Let me know if anything needed. — Reply to this email directly, view it on GitHub <#20 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDSZ5QPGWJDWA4EGML7TVCHXM5ANCNFSM5RA653MA> . You are receiving this because you were mentioned.Message ID: ***@***.***>