Indexing the Data Lake for Online Point Queries
Summary
Spotify discusses RAP (Random Access Parquet), an external index approach that enables fast online point queries on data lakes by mapping keys to Parquet file locations and precise, targeted reads. The article outlines the architecture, index format, and optimizations such as sorting by key, co-grouping, interleaving columns, and hoisted values, arguing this can make data lakes viable for interactive access and AI prompts without ETL. It concludes that data lakes can serve both analytical and interactive workloads with low latency.