Home

This is an informal set of notes describing Drill internals. The purpose is to share with the Drill community knowledge of Drill internals. The material here is informal because it presents information the author was able to reverse-engineer from code, discussions, and the like.

General

• Drill Within the Big Data Ecosystem

Development Topics

General tips and tricks for working with Drill development.

• Development Tips
• Testing Tips
• Test Data
• Git Workflow for Large Drill Projects

Development Findings

Results of various informal forays into Drill performance and more.

• Performance Basics
• Code Generation and "Short, Fat" Queries
• Optimization of External Sort Performance
• Optimization of Filter Performance
• Drill Date/Time Types
• Drill Maps
• Revisit Vector Accessors
• Drill Spill File Format
• Query Scheduling

Internals Topics

Discussions of Drill internals and architecture.

• Runtime Model
• Creating a Data Format Plugin
• Runtime Operator Menagerie
• Runtime Operator Protocol
• Value Vector Implementation
• External Sort Operator
• Code Generation
• RPC Protocol
• Storage Plugin Model
• Memory Management
• The Many Types of Vector Bundles
• Memory Allocator
• Hash Agg Spilling (Thanks to Boaz!)