RAG Pipeline Architecture: A Complete Guide

Retrieval-Augmented Generation is how every serious production LLM application grounds its responses in real data. This guide covers every component, how to pick tools, what it costs, and the pitfalls that derail 80% of RAG projects.

What is RAG and why does it matter

Retrieval-Augmented Generation means the model answers questions by first retrieving relevant passages from your own data, then generating a response grounded in those passages. Without RAG, an LLM is limited to its training data — it doesn't know your product, your policies, your customer history, or anything that happened after its knowledge cutoff.

RAG is how you go from "cool demo" to "production system that handles customer questions, legal lookups, internal docs, or specialized analysis." Every enterprise AI deployment worth its runtime cost uses RAG in some form.

The 6 components of every RAG pipeline

A production RAG pipeline has six moving parts. You can substitute vendors at each layer, but you cannot skip layers without cutting quality.

1. Ingestion

Pulls source content — web pages, PDFs, Notion, Confluence, S3 buckets, databases — into a queue for processing. Common tools: Firecrawl (web), Airbyte (ETL), custom connectors.

2. Parsing & chunking

Converts raw documents into semantically coherent chunks. The biggest quality wins in RAG come from getting this layer right. Tools: Unstructured.io (general purpose), LlamaParse (layout-aware PDFs, tables), custom recursive splitters for code and markdown.

3. Embedding

Turns each chunk into a high-dimensional vector. Choice here matters for retrieval quality. OpenAI text-embedding-3-small is the pragmatic default ($0.02 per 1M tokens). Cohere embed-v3 and Voyage-3 are strong alternatives. Self-hosted: BGE-M3.

4. Vector storage

Stores vectors with metadata and supports fast approximate nearest-neighbor search. Pinecone is the managed default; Weaviate and Qdrant self-host well. pgvector works if you already run Postgres and don't need to scale past ~10M vectors.

5. Orchestration & retrieval

The glue that takes a user query, embeds it, retrieves top-k chunks, and constructs the prompt. LangChain has the most integrations; LlamaIndex is more opinionated and tends to produce better default retrievers. For multi-step reasoning, LangGraph.

6. Generation

The LLM that produces the final answer. Claude (Anthropic) and GPT-4 (OpenAI) are the premium options. Gemini 1.5 Flash is 10× cheaper for most tasks with ~80% of the quality. Open-weight alternatives: Llama 3.1, Qwen 2.5.

Bonus: Observability

Non-optional in production. Langfuse (free tier, open-source) and LangSmith (managed by LangChain) trace every query — without this, debugging latency spikes or quality regressions is impossible.

Tool selection: picking the right stack

Cost breakdown (with real numbers)

A production RAG serving ~10,000 queries/day with a 100-page document corpus:

Most RAG projects come in between $150 and $500/month at production scale. The dominant line items are always (1) the model and (2) the vector store. Optimizing either has outsized impact.

Full Python code example

The skeleton of a minimal RAG pipeline with LlamaIndex and Pinecone:

import os
from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough

# 1. Embeddings + store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
store = PineconeVectorStore(
    index_name="kb-prod",
    embedding=embeddings,
)

# 2. Retriever
retriever = store.as_retriever(search_kwargs={"k": 5})

# 3. Model
llm = ChatAnthropic(
    model="claude-sonnet-4-5",
    api_key=os.environ["ANTHROPIC_API_KEY"],
)

# 4. Prompt
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the provided context. If unsure, say so."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

# 5. Chain
def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
)

# Run it
result = chain.invoke("What's our refund policy?")
print(result.content)

This is ~40 lines for a working RAG pipeline. Production adds: chunking strategy, hybrid search (BM25 + vectors), reranking, guardrails, and Langfuse tracing — each is a multiplicative quality improvement.

5 pitfalls that kill RAG projects

1. Naive chunking. Splitting on fixed token boundaries destroys semantic units. Use recursive splitters that respect document structure (headings, paragraphs, code blocks).
2. No reranking. Top-5 vector hits are often wrong. A cheap reranker like Cohere Rerank improves recall by 20–40% for ~$10/month.
3. Ignoring observability. Without tracing every query, debugging quality regressions is impossible. Add Langfuse on day one, not month six.
4. Single-model dependence. When OpenAI has an outage, your workflow dies. Add a gateway like Portkey for automatic failover (free tier).
5. No security layer. Prompt injection and PII leakage are real. Lakera Guard or Protect AI between user input and the model is table stakes for anything touching customer data.

Build yours in 2 minutes

TechStackopoly's visual Workflow Planner loads a starter RAG pipeline with one click, shows live monthly costs as you swap tools, and exports the whole thing as runnable LangChain Python. It's free forever and requires no signup.

Other guides you might want:

• Multi-agent workflow builder
• Document processing pipeline
• LLM observability stack