Create the command driven connectors RFC #255
Conversation
Signed-off-by: Ramona Łuczkiewicz <[email protected]>
✅ Deploy Preview for tremor-www ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
Signed-off-by: Ramona Łuczkiewicz <[email protected]>
Signed-off-by: Ramona Łuczkiewicz <[email protected]>
Signed-off-by: Ramona Łuczkiewicz <[email protected]>
Signed-off-by: Ramona Łuczkiewicz <[email protected]>
Co-authored-by: Heinz N. Gies <[email protected]> Signed-off-by: agares <[email protected]>
| `control` is an input port, through which the commands are sent. | ||
| Currently only reads are supported, so `data` is an output port. | ||
| Each message in the `data` port is a single event, with metadata containing the original command. | ||
| The commands are standardised as far as it is practical, so the connectors can be swapped without adjusting the rest of the system. |
There was a problem hiding this comment.
Do we support this by namespacing commands or is there another plan? What happens if a connector gets an unsupported command? Should we provide convinience functiuons for them such as file::open("some.file") that create the right events for a command?
There was a problem hiding this comment.
I think some sort of namespacing will be required, e.g. "query" will take a different kind of query (and possibly different arguments) based on the database, as even "file_read" can be different (depending on the underlying technologies, the paths might not be compatible, there might be more arguments needed to locate the file, etc.). I like the idea with convenience functions, I think that would give us good UX (while allowing people to still generate the commands manually if they need/wish to).
Signed-off-by: Ramona Łuczkiewicz <[email protected]>
| ## Unresolved Questions | ||
| [unresolved-questions]: #unresolved-questions | ||
|
|
||
| - How do we enforce uniformity of commands across connectors? |
There was a problem hiding this comment.
I really like the concept of traits, behaviors, personas , or whatever it would be called for that. (I'll go with traid in the rest of the text as it's the shortest, which is pointless since I probably wrote more in this note then I save by using trait ...)
The basic idea is that a connector defines a number of traits that specify the kind of commands it interacts with. Which traits exist we got to define possibly not in the RFC as it'll change over time, but we could start with some examples.
lets look at s3 / file / gcs as an example. I suspect it would work something like this:
- s3 offers the
fileio,objectstoreands3traits - gcs offers the
fileio,objectstoreandgcstraits - file offers the
fileio,fstraits
commands such as list files, open file, close file etc would be in fileio
commands like cache id could be in objectstore
and endpoint specific commands could be in gcs and s3 respectively
The reason for this is that with generic traitrs testing / prototyping and migrating become very easy. As long as no implementation specific features are used prototyping could happen with a local only connector and then switched over to a production connector on deployment.
There was a problem hiding this comment.
Yes - reusable traits could be a huge boon for testing. For example, if we added seek or sync to fileio or other commands to fileio then these would be useable in tests for setup and may imply a set of associated assertions.
|
|
||
| - How do we enforce uniformity of commands across connectors? | ||
| - How would writes work? | ||
| - Are multiple events per command allowed in the output? |
There was a problem hiding this comment.
My gut feeling is that batching for commands is possible, we can unbatch them and go about our work, I don't see anything that would prevent this. On the other side I don't see much of a use for this either.
There was a problem hiding this comment.
yea, I think we can get away with not answering that right now, and decide when/if we actually need it
There was a problem hiding this comment.
We probably want commands from a given command source to be applied in arrival order - that would seem to be a strong enough guarantee initially and consistent with what we already have for user or runtime events. Later, if we need stronger or variant guarantees - perhaps these could be controlled via command traits. A future worth iterating in this RFC IMO ...
Signed-off-by: Ramona Łuczkiewicz [email protected]