When "no healthy upstream" isn't about the upstream you think
Summary
The article analyzes a 'no healthy upstream' incident in a distributed search backend, arguing that the failure was not due to CPU but to downstream dependencies and a systemic design issue. It highlights how missed timeouts, unbounded retries, and head-of-line blocking can cause metastable outages, and presents a concrete fix: implement per-call timeouts, bound retries with a budget, and avoid retries on timeouts. The piece emphasizes diagnosing by following evidence rather than sticking to an initial hypothesis.