How shredding JSON is giving Logfire 1000x query speedups
Summary
The article explains dynamic shredding of semi-structured JSON within Logfire to dramatically speed up queries. It discusses the shift from static shredding to dynamic shredding, the resulting data layouts, and the role of DataFusion/Parquet in enabling columnar pruning and faster analytics. It also covers design decisions, potential impacts on users, and future enhancements.