Designing Production-Grade RAG Pipelines

When I shipped an LLM-driven recipe feature — type a dish name, get a recipe with every ingredient semantically matched to real supermarket products and dropped straight into the cart — the hard part was never the LLM call. It was everything around it: chunking, retrieval, grounding and keeping the whole pipeline fast enough to feel instant.

Chunking is a product decision, not a preprocessing step

Most RAG write-ups treat chunking as boilerplate — split text into 512-token windows and move on. In production, chunk boundaries directly shape answer quality. For product data, I chunk at the entity level (one product, one chunk, with structured attributes flattened into the text) rather than by token count, so a single retrieval hit is always a complete, self-contained fact the model can ground an answer in.

Embeddings and retrieval

Transformer embeddings go into Elasticsearch's dense vector fields, queried with approximate nearest-neighbor search alongside traditional BM25 signals. Hybrid retrieval — vector similarity plus keyword matching — consistently outperformed either approach alone, especially for short, typo-prone user queries like ingredient names.

• Recall first, precision second — over-fetch candidates (top 20–50), then rerank down to the 3–5 the model actually sees.
• Metadata filters before vector search — category, availability and price filters cut the search space before the expensive similarity comparison runs.
• Cache aggressively — repeated or similar queries (common in a recipe feature) hit a semantic cache before touching the index at all.

Grounding and evaluation

The generation step only ever sees retrieved context — never a free-floating "answer from memory" — and the prompt explicitly instructs the model to say when it can't find a confident match rather than hallucinate a substitute ingredient. I built a small evaluation harness with labeled query/expected-product pairs that runs on every pipeline change, so a retrieval regression shows up before it ships, not after a user complains.

The result is a feature that feels conversational but is, underneath, a fairly disciplined search-and-rerank system with an LLM doing the last-mile synthesis — which is really what most production RAG systems are.