LRU is harder to beat than the KV-cache papers suggest
Summary
The article presents a cross dataset analysis of cross-request prefix caching in agentic LLM serving, showing that LRU based policies remain strong in capacity bound regimes and that TTL based assumptions may mislead capacity planning. It uses traces from Mooncake and AgentX to demonstrate idle sessions and the dominance of short idle gaps in recomputation, and cautions about harness bugs and the need to validate against published curves.