Building RAG Pipelines That Don't Hallucinate

Grounding LLMs in your data is easy. Grounding them accurately is the hard part. Here's my production playbook.

Rufsan

Senior Full-Stack Developer & Agency Founder

blog.rufsan.dev/rag-pipelines-no-hallucination

🧠

Every week I see another "Build RAG in 10 minutes" tutorial. They all follow the same pattern: chunk documents, embed them, stuff them into the prompt, done. And every week, production teams discover that this naive approach hallucinates 15-30% of the time. Here's what actually works after building RAG systems processing 10K+ daily queries with 97%+ accuracy.

The Chunking Problem Nobody Talks About

Most tutorials chunk by token count — 512 tokens with 50-token overlap. This is the single biggest source of RAG hallucinations. When you split a paragraph mid-thought, the embedding captures half an idea. The retriever returns a fragment. The LLM fills in the gap with plausible-sounding fabrication.

The fix is semantic chunking. Parse document structure first — headings, paragraphs, lists, tables. Chunk at semantic boundaries. A paragraph that explains a concept stays together. A table row stays with its headers. This alone dropped our hallucination rate from 23% to 8%.

code

// Semantic chunking pseudocode
const chunks = document.sections.flatMap(section => {
  if (section.tokenCount <= MAX_TOKENS) return [section];
  return section.paragraphs.reduce((acc, para) => {
    const current = acc[acc.length - 1];
    if (current.tokenCount + para.tokenCount <= MAX_TOKENS) {
      current.content += '\n' + para.content;
      current.tokenCount += para.tokenCount;
    } else {
      acc.push({ ...para });
    }
    return acc;
  }, [{ content: '', tokenCount: 0 }]);
});

Hybrid Retrieval: The Secret Weapon

Pure vector search fails on exact matches. Ask "What is error code E-4012?" and semantic search returns results about error handling in general. Pure keyword search fails on conceptual queries. The solution is hybrid retrieval: run both vector search and BM25 in parallel, then use Reciprocal Rank Fusion to merge the results. Hybrid retrieval improved recall@10 from 72% (vector only) to 91%.

The Confidence Scoring Layer

This is the piece that gets the hallucination rate from 8% to under 3%. After retrieval, before generation, run a cross-encoder reranker on the top candidates. Score each chunk's relevance to the query on a 0-1 scale. If no chunk scores above 0.6, don't generate — return 'I don't have enough information to answer that accurately.'

Yes, refusing to answer feels counterintuitive. But users trust a system that says 'I don't know' far more than one that confidently fabricates. Our CSAT scores jumped from 3.8 to 4.5 after adding the confidence gate.

Grounding the Generation

Even with perfect retrieval, the LLM can still hallucinate during generation. Two techniques eliminated this: structured prompting ('Answer ONLY using facts stated in the provided context. Cite the specific section for each claim.') and citation verification — for each claim in the response, tracing it back to a specific chunk.

Production Numbers

The full pipeline processes 10K+ queries daily. Accuracy sits at 97.2%. Average latency is 1.8 seconds end-to-end. The system handles 50K+ knowledge base articles across 3 languages.

The RAG accuracy stack: Semantic chunking (23% → 8% hallucination) + Hybrid retrieval (8% → 5%) + Confidence gating (5% → 3%) + Citation verification (3% → < 2%). Each layer compounds.

Tags:AI / MLProduction

// Related