Home

This is an informal set of notes describing Drill internals. The purpose is to share with the Drill community knowledge of Drill internals. The material here is informal because it presents information the author was able to reverse-engineer from code, discussions, and the like.

General

• Updates to "Learning Apache Drill"
• Toward a Workable Dynamic Schema Model
• UTC Timestamp Data Type Proposal

Using and Extending Drill

• Test Data
• UDFs Background Information
• Creating a Data Format Plugin
• Create a Storage Plugin

Development Topics

General tips and tricks for working with Drill development.

• Development Tips
• Testing Tips
• Git Workflow for Large Drill Projects
• Launch a Drillbit From Eclipse
• Batch Handling Upgrades
• Developer's Guide to the Enhanced Vector Framework
• Drill Commit Guidelines
• Drill Within the Big Data Ecosystem
• Install Drill from a Local Build
• Build Drill in a Corporate Environment
• Import Drill into Eclipse

Development Findings

Results of various informal forays into Drill performance and more.

• Performance Basics
• Code Generation and "Short, Fat" Queries
• Optimization of External Sort Performance
• Optimization of Filter Performance
• Drill Date/Time Types
• Drill Maps
• Revisit Vector Accessors
• Drill Spill File Format
• Query Scheduling

Internals Topics

Discussions of Drill internals and architecture.

• Runtime Model
• Runtime Operator Menagerie
• Runtime Operator Protocol
• Value Vector Implementation
• External Sort Operator
• Code Generation
• RPC Protocol
• Storage Plugin Model
• Memory Management
• The Many Types of Vector Bundles
• Memory Allocator
• Hash Agg Spilling (Thanks to Boaz!)
• The State of Drill