Building Index-Backed Query Plans in DataFusion
Summary
This article explains how to bring PostgreSQL-style index access patterns to DataFusion via datafusion-index-provider, enabling OLTP-like index scans and multi-index queries. It covers architectural choices, including two custom ExecutionPlan nodes (IndexScanExec and RecordFetchExec), and shows how to compose plans for single-index, AND (intersections), OR (unions with deduplication), and mixed expressions. It also discusses limitations and future work for partial index usage and multi-partition execution.