Pruning RAG context down to what the answer actually needs
Summary
Kapa explains a three-step approach to pruning RAG context by introducing a lightweight, inexpensive LLM that reads the question and all retrieved chunks and discards unnecessary ones before the expensive model. The result: about 68% of the retrieved chunks are dropped while preserving ~96% recall, reducing per-query cost by roughly one-third, with latency around 0.7 seconds per query. The post covers design choices, evaluation, and deployment in their Product Agent SDK and retrieval API.