Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Key-value store backend? #27

Open
ebiiii opened this issue Jun 16, 2014 · 0 comments
Open

Key-value store backend? #27

ebiiii opened this issue Jun 16, 2014 · 0 comments
Labels

Comments

@ebiiii
Copy link
Contributor

ebiiii commented Jun 16, 2014

Currently the interface to the different databases backend is through the StorageManager. It was built assuming SQL databases (mySQL, postgresQL,...), offering functions that takes SQL queries as input and JDBC drivers are doing the connection to the databases.
But if we want to introduce NoSQL backends, like HBASE, or even memcached, the process of understanding the SQL query and performing it is done at the JDBC driver level (like Apache Phoenix project). Having such adapters would work out almost of the box, but when looking at the whole streaming process would introduce a huge and useless overhead.
Indeed, a lot of functions are just building SQL queries to be able to interact with the DB, but if we know that the DB is just a K-V store, and that the query will just need to be parsed in the next function call, why still using SQL internally?
SQL is very handy for using in the VS configuration files and should be kept here, but it should be parsed much sooner and replaced by an abstraction that can be efficiently passed to SQL and NoSQL backends.

This is just a suggestion and it is not clear for me yet at which level this should be implemented (JDBC, StorageManager, SlidingTimeWindows, VSConfig) ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant