What we learned from a 22-Day storage bug (and how we fixed it)
Summary
Mux Video shares a postmortem about a 22-day storage bug where roughly 0.33% of VOD segments were corrupted. The piece details how distributed storage, JIT transcoding, and CDN delivery interacted to produce glitches, the contributing factors, and the fixes implemented to stop new corruption, regenerate affected segments, and purge CDN caches. It also highlights observability gaps and steps to improve incident detection and response.