Roadmap

We would like to ease the adoption of the Versatile Data Kit project in various use cases and to that end we are going to implement the following features/enhancements:

Ease of use improvements

This project has been internally in production for more than three years, but it has been internally focused on narrow technologies and we want to work with community to improve ease of use and extend to cover multiple use-cases and technologies. We would like to address any community feedback on the ease of use and operation of the project. This includes various usability improvements as well as extending the existing documentation and examples.

Workflow engine integration

In order to provide the ability to execute multiple data jobs sequentially and to provide better retry and notification capabilities we want to provide an integration with an existing workflow engine/management system like Apache Airflow

Data Lineage tracking capabilities

We want to provide the ability to collect data lineage metainformation during the execution of data jobs. This capability will facilitate the debugging and troubleshooting of Data Jobs as well as provide better data governance and change management.

Support for more Data Base engines and connectors.

We would like to extend the range or supported DB engines by providing connectors for the more popular DBs like Postgres. Also, we want to support more connectors so that people can integrate data form different sources to multiple targets with ease.

Better templates support

Expand the template support to make it easier to create data mart fact and dimension tables with almost zero code.