Lessons from BF-Tree: Building a Concurrent Larger-Than-Memory Index in Rust
Summary
The article explains BF-Tree's mini-page approach to decouple in memory buffers from on-disk 4KB pages, enabling smaller, variable-size in-memory buffers that reduce write amplification and memory waste. It details a custom circular-buffer allocator with a six-state mini-page lifecycle, RAII guards, and contention-safe growth, eviction, and copy-on-access, all implemented in Rust. The piece also covers testing strategies and reproducible concurrency testing using Shuttle.