From zero to a RAG system: successes and failures
Summary
This post documents a practical, step-by-step journey building a local, private RAG system for a large codebase using Ollama for local LLaMA, LlamaIndex as the orchestrator, and ChromaDB as the vector store. It covers the six major challenges (technology choice, data chaos, indexing scale, GPU constraints, UX, and storage) and the engineering decisions, trade-offs, and lessons learned to produce a fast, usable internal tool.