How We Made Payload Search 60x Faster in ClickHouse
Summary
Hookdeck details a real-world optimization that sped up payload search in ClickHouse from tens of seconds to about 400ms on average, with p99 under 1.4 seconds. The team tackled semi-structured webhook payloads by introducing value-based bucketing and a variable-window iterative scan with adaptive granularity, plus data synchronization across resources to enable efficient time-window queries. The article offers practical lessons for teams dealing with large, semi-structured datasets and performance-sensitive analytics.