We could save petabytes of cache storage with Zstandard and Pingora
Summary
Cloudflare describes Cache Transcoding, a system that encodes eligible cached responses with Zstandard inside the Pingora proxy to shrink on-disk cache size and reduce inter-data-center bandwidth. The approach selectively transcodes text content (HTML/JSON/CSS/JS) with a 4 KiB minimum, stores zstd-encoded assets in Tiered Cache, and decodes them on the client path, achieving about 2.8x compression in tests while incurring a small CPU cost. Results from testing over a million requests show significant storage savings with potential future improvements like higher zstd levels and broader content coverage.