Wiring an LLM to a vector database and getting a good demo takes an afternoon. I’ve watched teams do it and feel finished. They aren’t — they’re at the start of the hard part.
The demo answers the questions you chose. Production answers the questions your users actually ask, at a volume that shows up on a bill. Three things separate a RAG system that ships from one that quietly stalls: retrieval you’ve measured, evals you trust, and cost you modelled before you committed.
Retrieval is the product; the LLM is the easy part
Everyone obsesses over the model. But in a RAG system, the model is mostly downstream of retrieval. If you hand it the wrong chunks, even a great model gives a confident wrong answer. Garbage in, fluent garbage out.
So measure retrieval directly, before you judge the whole system. For a set of real queries, is the right document in the top-k results? If it isn’t, no amount of prompt-tuning saves you. Most “the AI is hallucinating” problems are actually “retrieval surfaced the wrong context” problems.
Evals before opinions
Without evals, every prompt change is a vibe. Someone tweaks the system prompt, it “feels better,” it ships, and a week later something subtle is worse with no way to prove it.
You don’t need a research-grade eval harness. You need a fixed set of representative queries with known-good answers, and a way to score a change against them before it reaches users. The goal isn’t a perfect score; it’s the ability to tell improvement from regression. That single capability changes how a team works — decisions stop being arguments.
Model the cost before you architect
This is the one that bites. Teams architect for capability, ship, and then discover that each query fans out into several embedding calls, a large-context retrieval, and a top-tier model invocation — and they’re doing it a hundred thousand times a day.
Model it first. Roughly: what does one query cost in embedding calls, retrieval, and generation? Multiply by realistic volume. Often the answer is “fine,” and you move on. Sometimes it’s “we cannot ship this as designed,” and you’d much rather learn that on a whiteboard than from an invoice. The fixes — smaller models where they’re enough, caching, retrieval that doesn’t over-fetch — are cheap when they’re design decisions and expensive when they’re emergencies.
Don’t get locked in
The models change every few months. The provider that’s best today may not be best, or cheapest, or available, next year. Put your prompts, retrieval, and evals behind boundaries so you can swap the model without rewriting the product. Lock-in is a cost you pay later for a convenience you enjoy now.
The pattern under all of this
It’s the same discipline as any serious system: design around the failure modes, measure the things that matter, and keep the expensive decisions reversible. AI doesn’t change that. It just raises the stakes, because the failure modes are subtler and the bill is bigger.
If you’re past the demo and into the hard part, that’s exactly the work we do — RAG & AI Architecture, done right. Or just book a call and bring the problem.