RAG Pipeline Architecture: Seven Stages and Where Each Breaks

RAG pipeline architecture fails in predictable places, and this guide names all seven of them in the order data flows: ingestion, chunking, embedding, indexing, retrieval, reranking, and generation. If you only fix one stage, fix chunking. Anthropic’s September 2024 testing cut retrieval failures by 67 percent through contextual chunking plus reranking, the single biggest documented win anywhere in the stack. The stage with the most distinct failure modes, though, is the last one: generation owns four of the seven failure points cataloged in production systems. As of August 29, 2026, here’s where each stage breaks and what the evidence says to do about it.

How we ranked the seven stages of RAG pipeline architecture

Documented failure evidence decided everything. Each stage below appears because peer-reviewed research or primary engineering data pins a specific, reproducible failure to it, anchored by Barnett et al.’s “Seven Failure Points When Engineering a Retrieval Augmented Generation System”, the 2024 IEEE/ACM study that traced failures across three production case studies in research, education, and biomedical domains. Its blunt finding: failures cluster at specific pipeline junctures, and reliability evolves in live operation rather than getting designed in upfront. We build RAG pipelines for a living at AlphaCorp AI, and that finding matches every deployment we’ve shipped. The stages run in data-flow order, with the highest-leverage fixes flagged as they come up.

1. Ingestion and Parsing: Answers That Never Make It In

Ingestion is the stage that extracts usable text from PDFs, scans, HTML, and DOCX files, and it’s where answer-bearing content most often disappears before anything downstream ever runs. Barnett et al. call this failure point 1, “Missing Content”: the corpus never captured the answer, so the system either admits uncertainty or, worse, hallucinates over the gap.

Three things break here more than anything else:

• Tables. A 2024 survey of document parsing found dense financial tables, multi-column spans, and merged cells consistently defeat naive extraction. Tables are the hardest element in the entire parsing problem.
• Scanned documents. A 2026 benchmarking study built around InduOCRBench showed OCR errors don’t stay put. They flow through chunking, embedding, and retrieval, and the downstream LLM treats the garbled text as correct source material.
• Anything non-digital-native. Standard text extraction works fine on clean exports and falls apart on real-world documents, which is why 2024 work on advanced ingestion moved to layout-aware, multi-strategy parsing with LLM-powered extraction for messy and multimodal inputs.

Watch this stage hardest if your corpus is financial filings, medical records, or anything that lived on paper first. Clean SaaS docs can get away with simpler parsing. Scanned invoices cannot.

2. Chunking: The Highest-Leverage Fix in the Whole Pipeline

Chunking, the splitting of parsed text into retrievable units, is the stage where a modest engineering effort buys the largest measured improvement in the entire RAG pipeline. It’s also one of the least standardized decisions in the field: a 2026 cross-domain evaluation benchmarked 36 segmentation methods across six domains and five embedding models and found content-aware chunking clearly beats fixed-length splitting.

The core failure is context fragmentation. A chunk that reads “The company’s revenue grew by 3% over the previous quarter” is useless without knowing which company and which quarter, and that ambiguity survives almost every chunking strategy. Anthropic’s Contextual Retrieval work from September 2024 attacks it directly by prepending a 50 to 100 token LLM-generated context header to each chunk before embedding.

Contextual embeddings cut Anthropic’s top-20 retrieval failure rate from 5.7% to 3.7%. Adding contextual BM25 brought it to 2.9%, and reranking on top brought it to 1.9%, a 67% total reduction, per Anthropic’s September 2024 internal testing.

Fair warning: those figures are vendor-run internal tests from September 2024, and no third party has reproduced the reductions at that magnitude since. The direction is well supported. The exact percentages deserve a grain of salt.

Chunk size itself has no universal answer. A 2025 multi-dataset analysis of chunk size found 64 to 128 token chunks win for concise factual answers while 512 to 1024 tokens win when broader context matters, and the optimum shifts per embedding model: Stella prefers larger chunks for long-range retrieval, Snowflake does better with small, entity-focused ones. Tune per corpus and per model. There’s no shortcut.

3. Embedding: A Mathematical Ceiling You Cannot Tune Away

Embedding converts chunks into dense vectors, and its defining failure is theoretical rather than operational: for any fixed embedding dimension, there exist sets of documents that no single-vector model can ever return as a top-k result for certain queries. That’s the finding of a 2025 analysis of the geometric limits of embedding-based retrieval, which built the LIMIT benchmark around trivially simple attribute-matching queries and watched state-of-the-art embedding models score recall@100 below 20 in many settings. No amount of training fixes that. It’s geometry.

Two more documented weaknesses stack on top:

• The “Semantic Illusion,” identified in work published across late 2025 and 2026: embeddings score plausible hallucinated text as indistinguishable from faithful text, while reasoning-capable models catch the divergence.
• Implicit meaning. 2025 research on embedding semantics argues current models don’t capture nuanced, implicit meaning because training data rarely supervises for it and benchmarks don’t reward it.

The partial fix is multi-vector representation. ColBERT-style models assign multiple vectors per document, which loosens the geometric constraint single vectors hit. Hybrid search with BM25 helps for the same reason. If your queries are precise attribute lookups (“contracts signed in Ohio in Q3”), this stage is your bottleneck. Budget for it.

4. Indexing and Vector Storage: Stale Data and a Tunable Accuracy Ceiling

Indexing makes vector search sub-linear at scale, almost always through the HNSW graph structure introduced in Malkov and Yashunin’s 2016 paper on hierarchical navigable small world graphs, and its central failure is a hard tradeoff you must budget explicitly. Raising the max-links-per-node parameter M improves recall while raising memory and indexing cost. Raising efSearch at query time improves recall while slowing every query. “Approximate” is a dial, and teams that never set it consciously ship a silent accuracy ceiling.

The second failure mode is temporal, and it’s the one nobody warns you about. Vector similarity has no time dimension: an embedding of a deprecated document scores exactly as high as a current one, so when source-system updates fail to propagate, the index serves confidently wrong answers. Deletion bugs are the nastiest version. Removed content can stay retrievable and get presented to the model as legitimate evidence.

This stage is also the door adversaries use. A May 2025 benchmark evaluated 13 poisoning attack methods against 7 defense mechanisms across 5 QA datasets and confirmed attackers can inject crafted text to force chosen outputs, and 2026 follow-up work on RAG security found even lightweight retrieval poisoning steers model outputs, with effective defenses still open. If your corpus accepts external or user-submitted content, treat the index as an attack surface.

5. Retrieval: The Right Chunk Exists but Never Surfaces

Retrieval is the query-time search over the index, and its signature failure is Barnett et al.’s failure point 2, “Missed the Top Ranked Documents”: the answer sits in the corpus, but the top-K cutoff excludes it. This follows directly from the recall tradeoffs baked into approximate search two stages earlier. Everything compounds downstream.

Query formulation is its own failure surface. Multi-hop questions routinely defeat single-shot dense retrieval, and 2025 research on multi-hop document retrieval traces it to inaccurate query decomposition with error propagation, where one early bad retrieval corrupts the final synthesis. The best-documented countermeasure is rewriting: the 2024 DMQR-RAG method showed multi-query rewriting beats single-query rewriting across benchmark datasets by widening the retrieval surface. Cheap to add, measurable to verify. Start there before touching anything heavier.

6. Reranking and Context Assembly: Precision at a Latency Price

Reranking adds a second-pass cross-encoder that jointly scores query and candidate, and it exists because first-pass retrieval optimizes recall rather than fine-grained ordering. The gains are real: one 2025 evaluation of cross-encoder rerankers measured +17.2 percentage points in MRR@3 and +12.1 points in Recall@5 over unreranked hybrid retrieval.

The price is real too. Cross-encoders carry quadratic inference cost, and 2025 practitioner benchmarks put typical reranker impact at a 15 to 30 percent lift in precision and recall against 100 to 500 milliseconds of added latency per query. That math shapes architecture decisions daily. Our RustyRAG stack targets sub-200ms end to end, which means a 500ms reranker is simply off the table and the reranking budget gets spent on distilled, lighter scoring instead. Nobody’s marketing page mentions that the reranker alone can cost more time than your entire latency target.

Even a perfect reranking pass can lose at assembly. Failure point 3, “Not in Context,” covers answers that were retrieved and then dropped while consolidating results into the context window. And position matters once text is in the window: the Stanford “Lost in the Middle” study published in TACL in 2024 showed performance peaks when relevant material sits at the start or end of context and sags in the middle, driven by primacy and recency biases in causal attention. One update worth knowing: a November 2025 study found Gemini 2.5 Flash holds retrieval accuracy steady across positions, including near the context boundary. The middle penalty is generation-dependent now. Don’t assume it applies to your model without testing.

7. Generation and Synthesis: Four Failure Points in One Stage

Generation, where the LLM writes an answer from retrieved context, carries the largest and most varied set of documented failures in RAG pipeline architecture, because retrieval mistakes compound with the model’s own habits here. Barnett et al. locate four of their seven failure points at this single stage:

• Not Extracted (point 4): the answer is in context, but noise or contradicting passages keep the model from pulling it out.
• Wrong Format (point 5): the query asked for a table or list and the model ignored the instruction.
• Incorrect Specificity (point 6): the response is too general or too narrow for what the user and domain need.
• Incomplete (point 7): nothing stated is wrong, but available information for a multi-part query got left out.

Faithfulness sits on top of all four. A May 2025 benchmarking effort behind the FaithJudge framework found LLMs still introduce unsupported claims and contradictions even when handed clearly relevant context. 2026 circuit-tracing work goes deeper: correct RAG answers show distributed, question-constrained integration of evidence, while failures show shallow, context-dominated routing where the model leans on retrieved text without discriminating. Hallucination here is an information-routing problem inside the model.

One trap deserves its own sentence. Because embeddings score plausible hallucinated text the same as faithful text (the 2025 Semantic Illusion result), any evaluation pipeline that checks faithfulness via embedding similarity is blind to exactly the failures it’s supposed to catch. Use a reasoning model as judge, or measure nothing.

Which stage of your RAG pipeline is actually failing?

Decompose the evaluation by stage, because “the RAG system was wrong” diagnoses nothing. The RAGAS framework splits this into four metrics built for exactly this attribution: Context Precision and Context Recall isolate retrieval-side problems (stages 1 through 5), while Faithfulness and Answer Relevance isolate generation-side problems (stages 6 and 7).

The shortcut we apply on client systems, AlphaCorp AI’s stage-attribution rule: never debug the answer, debug the stage that produced it. In practice:

• Wrong or missing source documents in the context? Work stages 1 to 5, starting with chunking.
• Right documents retrieved, wrong answer written? Work stages 6 and 7, starting with context assembly and position.
• Answers that were right last month and drift wrong now? Stage 4. Check index staleness before touching prompts.
• Correct but incomplete or misformatted answers? Pure stage 7. Prompting and output validation, no retrieval work needed.

The common mistake is swapping the generation model when Context Recall is the broken metric. That trades money for nothing.

RAG pipeline architecture: frequently asked questions

What is a RAG pipeline?

A RAG pipeline is the end-to-end system that grounds an LLM’s answers in external documents instead of its parametric memory, built to counter hallucination, outdated knowledge, and untraceable reasoning as framed in the foundational 2023 RAG survey by Gao et al. In engineering terms it runs seven stages: ingestion, chunking, embedding, indexing, retrieval, reranking with context assembly, and generation.

What is RAG architecture, and how has it changed by 2026?

RAG architecture is the structural design of retrieval, augmentation, and generation components, which the 2023 survey traced through Naive, Advanced, and Modular generations. A 2026 survey of RAG architectures extends the map to retriever-centric, generator-centric, hybrid, and reliability-oriented designs, reflecting how much the field moved through 2025 and 2026.

What are the seven failure points of a RAG system?

Barnett et al.’s 2024 study names them: Missing Content, Missed the Top Ranked Documents, Not in Context, Not Extracted, Wrong Format, Incorrect Specificity, and Incomplete. The first three are retrieval-side failures. The last four all occur at generation.

Do long-context models make reranking obsolete?

Not yet. A November 2025 study showed Gemini 2.5 Flash resists the middle-of-context degradation that hit earlier models, but reranking still delivered up to 17.2 points of MRR@3 improvement in 2025 benchmarks by improving what enters the window in the first place. Position tolerance helps. Relevance ordering still pays.

Where to start hardening your pipeline

Start at stage 2. Contextual chunking plus reranking produced the largest measured failure reduction on record (67 percent in Anthropic’s September 2024 testing), and it requires no model swap. Then instrument stage 4 for staleness, since a drifting index quietly poisons every good decision made elsewhere. Third, split your evals retrieval-side versus generation-side so every future failure arrives pre-diagnosed.

And run the numbers on latency before committing to a heavy reranker. A 500ms scoring pass can cost more than your whole response budget. If you’d rather have someone who has shipped this stack walk your architecture stage by stage, talk to the AlphaCorp AI team. The people you talk to are the people who build.