Make sources have view dependent parameters

[goaaron]

Feature request

I am working on a disaster recovery service for Materialized and am realizing along the way that there are a lot of potentially interesting benefits if you can make the parameters of a source dependent on a value that is calculated in a view.

E.g. the starting_offset of the Kafka source could be a static variable calculated at runtime based on a single parameter coming out of a view that is derived from another Kafka topic. That would generate a parameterized source. Or there could be a notion of a source factory that emits new sources every time they receive some event from a creator source (e.g. new s3 buckets)

This kind of way you get sources that are dependent on other sources and can do some type of dynamic source creation. I am approximating this but inside of a service that rewrites code to build the views–and I'm thinking that it may be useful for some kind of data pipelining capabilities.

[ruf-io]

@goaaron I hope you don't mind that I converted this to a discussion... This is a very intriguing idea, I'm not sure I fully understand it yet. Can you outline a scenario where you'd use this?

[benesch]

I don't have an immediately obvious use case for starting_offset to offer—I too would love to hear what you had in mind, @goaaron—but I can definitely see the appeal of an S3 source where the list of buckets to monitor is driven by some other source.

We've talked about this sort of dynamic "source stacking" in the past. I doubt we'd be able to build something general purpose here, but it does feel like there might be a few key places where we'd want to investigate it. E.g., a Kafka source where the list of topics is generated dynamically, an S3 source where the list of buckets/objects to fetch is generated dynamically, and so on.