Drill 2.0 Proposal

Drill 2.0 Proposal

This page serves to document a proposal for Drill 2.0. At the time of writing, we are currently gearing up to release Drill 1.20.0 which means that we have released 19 versions of Drill since it was deemed stable enough to warrant a 1.0 label. Since the second phase of this project's life began there have been some things which have been discussed which are breaking changes. This page serves to document proposed breaking changes that could be included in a Drill 2.0.

Please feel free to add your ideas in the knowledge that a subset will be ultimately be selected by the dev team as the basis for a 2.0 release. Changes recorded here need not neccessarily be user breaking. Anything that is a significant change from how Drill 1.x works is welcome.

General

• Remove deprecated code.

APIs and connectors

Config system

• Add a shared component that applies configuration priorities (... session opt > storage/format config opt > system opt ...) and make all plugins use this component for reading options.

Query planner

• Rebase on current Calcite and review our customisations.

Packaging and distribution

• Split Drill installation packages into "core" and "extra".
• Install plugins and UDFs from an online marketplace, a la the Eclipse marketplace.

Storage and format plugins

• Scrap the columns[] array wherever it occurs (only TextReader?) in favour of distinct, numbered fields column1, column2, column3, ...

• Increase consistency of available options across plugins, e.g. column name and type information only allowed in provided schema not format config.

• Use leading underscore for all implicit fields. Some plugins and connectors do this, but some don't. Particularly the file plugin in core Drill. This could lead to strange results if a file has a column called file.

Vector

• Do something about ObjectHolder.
• Drop union type?

Web UI

• Perhaps a refresh might be in order?