Code Generation

Code Generation

Drill relies heavily on code generation for the data-specific functionality in each operator. For example, the FilterBatch (implements the SQL WHERE clause) generates code for the actual filter condition. Code generation is necessary because Drill works with a large number of data types (over 120 different value vector implementations) and many kinds of expressions. The alternative, an interpreter, would be expensive to maintain (given the large number of value vector types) and slower to execute.

Drill has two major forms of code generation:

1. The FreeMarker-based code generation done during the build process, and
2. The JCode-based code generation done at execution time.

This writeup focuses on the second form.

(Code generation seems to follow a pattern established by Hive? Need to research.)

Topics

Code generation is a complex topic covered over multiple pages:

• Code Generation Workflow
• Components
• Code Cache
• Code Transformations
• Scalar Replacement
• Compilation Framework

References

Compilers:

• JavaCompiler in the JDK.
• javax.tools explanation.
• Janino Compiler

Byte code manipulation:

• ASM Package
• ASM Tutorial
• Merging Class Methods with ASM
• Scalar replacement research paper