I built a 2x faster lexer, then discovered I/O was the real bottleneck
Summary
The author builds a 2x faster ARM64 assembly-based Dart lexer but discovers I/O, not lexing, is the real bottleneck when processing a large corpus. By switching from 104,000 individual files to 1,351 tar.gz archives, I/O time drops dramatically and total runtime more than doubles in speed, illustrating the impact of archive formats and syscall overhead on performance. The post also discusses why pub.dev uses tar.gz, potential optimizations (zstd, uncompressed tar, io_uring, SQLite), and implications for build systems and tooling.