Home

This is an informal set of notes describing Drill internals. The purpose is to share with the Drill community knowledge of Drill internals. The material here is informal because it presents information the author was able to reverse-engineer from code, discussions, and the like.

General

• Updates to "Learning Apache Drill"
• Parting Thoughts

Development Topics

General tips and tricks for working with Drill development.

• Development Tips
• Testing Tips
• Test Data
• Git Workflow for Large Drill Projects
• Launch a Drillbit From Eclipse
• UDFs Background Information
• Batch Handling Upgrades
• Developer's Guide to the Enhanced Vector Framework
• Drill Commit Guidelines
• Drill Within the Big Data Ecosystem
• Install Drill from a Local Build
• Build Drill in a Corporate Environment
• Create a Storage Plugin

Development Findings

Results of various informal forays into Drill performance and more.

• Performance Basics
• Code Generation and "Short, Fat" Queries
• Optimization of External Sort Performance
• Optimization of Filter Performance
• Drill Date/Time Types
• Drill Maps
• Revisit Vector Accessors
• Drill Spill File Format
• Query Scheduling

Internals Topics

Discussions of Drill internals and architecture.

• Runtime Model
• Creating a Data Format Plugin
• Runtime Operator Menagerie
• Runtime Operator Protocol
• Value Vector Implementation
• External Sort Operator
• Code Generation
• RPC Protocol
• Storage Plugin Model
• Memory Management
• The Many Types of Vector Bundles
• Memory Allocator
• Hash Agg Spilling (Thanks to Boaz!)

Apache Arrow Topics

• RowSet Abstractions for Arrow