Hand-Crafting Domain-Specific Compression with an LLM
Summary
This Nanit Engineering post documents using a large language model to design a tailor-made compression scheme for temperature/humidity time-series data. It compares standard time-series codecs with a custom approach (NibbleRun), employing zero-run and delta encodings, timestamp quantization, and gap handling to achieve significant compression while preserving appendability, while also highlighting pitfalls and lessons learned when using LLMs for benchmarking and development.