The RAG Pipeline and Agent Loop Mental Model
Every page in this section, whether it covers LangChain, LlamaIndex, vector databases, or the Model Context Protocol, is an implementation detail sitting on top of exactly two mental models.
//
Search across all documentation pages
Every page in this section, whether it covers LangChain, LlamaIndex, vector databases, or the Model Context Protocol, is an implementation detail sitting on top of exactly two mental models.
The first is the retrieval-augmented generation (RAG) pipeline: a fixed sequence of stages that turns a pile of documents into grounded, cited answers.
The second is the agent loop: a cycle that lets a language model decide what to do next, call a tool, observe the result, and decide again.
Understanding these two models first makes every specific library choice - which vector store, which reranker, which orchestration framework - a much smaller decision than it first appears.
Retrieval-augmented generation exists because language models have two hard limits: a fixed context window and a training cutoff.
You cannot paste your entire knowledge base into a prompt, and the model cannot know about documents written after it was trained.
RAG solves both by retrieving only the relevant slice of text at query time and handing it to the model as context.
The pipeline has five conceptual stages, and each one exists to fix a specific failure of the stage before it.
Chunking splits long documents into retrieval-sized pieces, because embedding an entire PDF as one vector loses the fine-grained meaning of any single paragraph inside it.
Embedding converts each chunk into a vector, a list of numbers that places semantically similar text near each other in a high-dimensional space.
Retrieval takes a query, embeds it the same way, and finds the nearest chunk vectors, which is the pipeline's first real point of failure since "nearest" is not the same as "correct."
Reranking takes the top candidates from retrieval and re-scores them with a more expensive, more accurate model, because fast approximate search over millions of vectors necessarily leaves relevant results ranked lower than they should be.
Generation is the final step, where the LLM writes an answer using only the retrieved (and reranked) context, ideally with citations back to source chunks.
The agent loop is a different but complementary model.
Where RAG is a straight pipeline (data flows one direction, once), an agent loop is a cycle: the model receives state, decides on an action, an action executes, and the result feeds back in as new state.
The simplest mental picture is a thermostat: sense the temperature, decide whether to heat or cool, act, sense again.
For an LLM agent, "sense" is the conversation and tool results so far, "decide" is the model choosing a tool call or a final answer, and "act" is actually invoking that tool.
Tool use (sometimes called function calling) is what makes the loop useful: instead of the model only producing prose, it produces a structured request - a function name and arguments - that your code executes and feeds back as an observation.
The two models interact constantly in real systems, because the most common agent tool is a RAG retriever.
An agent loop's "decide" step commonly resolves to "call the search_docs tool," which triggers the entire RAG pipeline, whose output becomes the "observe" step feeding the next decision.
This is why the two mental models belong on the same page: RAG without a loop is a single-shot Q&A system, and an agent loop without RAG has no way to ground its actions in your actual data.
Inside the RAG pipeline, the embedding model used at index time and the embedding model used at query time must be the same model (or at least compatible), because vectors from different models do not share a coordinate space.
This is a subtle but critical mechanical detail: swapping embedding models means re-embedding and re-indexing every chunk, not just the new queries.
Retrieval quality is usually described with recall (did the relevant chunk appear at all in the candidate set) and precision (how much of the candidate set is actually relevant), and these pull against each other.
A larger top_k improves recall but dilutes precision and increases the tokens sent to the generation step, which is exactly the problem reranking is designed to solve: retrieve broadly and cheaply, then narrow accurately and expensively.
In the agent loop, the critical mechanical detail is termination.
An LLM does not inherently know when to stop calling tools, so every real agent framework enforces a bound - a maximum number of iterations, a recursion limit, or an explicit "final answer" signal the model must emit.
Without that bound, a model that keeps deciding "I need more information" will call tools indefinitely, burning cost and time with no natural exit.
def run_agent_loop(model, tools: dict, state: list, max_steps: int = 8):
for _ in range(max_steps):
decision = model.decide(state) # LLM picks: answer or call a tool
if decision.is_final_answer:
return decision.content
result = tools[decision.tool_name](**decision.args) # act
state.append({"tool_result": result}) # observe -> feeds next decide()
return "stopped: exceeded max_steps" # explicit bound, not a hangThis snippet is not a production agent (real frameworks like LangGraph handle branching, persistence, and parallel tool calls), but it makes the loop's shape explicit: decide, act, observe, repeat, with a hard stop.
At scale, both models develop failure modes that do not show up in a demo with ten documents.
RAG pipelines built on a single flat vector index degrade as the corpus grows into the millions of chunks, because approximate nearest-neighbor search trades recall for speed, and that trade-off gets worse as the index grows unless you add filtering, hybrid search (combining keyword/BM25 with vector similarity), or hierarchical retrieval.
Agent loops develop a parallel failure mode: as you add more tools, the model's decision step gets harder, because it must now choose correctly among a larger, more ambiguous set of options, and small prompt or tool-description changes can shift behavior unpredictably.
Observability differs between the two models as well.
RAG pipelines are debugged by inspecting what was retrieved - if the answer is wrong, the first question is always "was the right chunk even in the candidate set?"
Agent loops are debugged by inspecting the trace of decisions and tool calls, since the failure is often several steps upstream of the final wrong answer, not in the last step itself.
Security also diverges: RAG's attack surface is data poisoning (malicious content injected into the corpus that gets retrieved and treated as trusted context), while an agent loop's attack surface is prompt injection through tool outputs - a tool result that itself contains instructions the model may follow.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
| Naive RAG (single retrieve + generate) | Simple, low latency, easy to reason about | No self-correction if retrieval misses | Small, well-curated corpora |
| Retrieval + reranking | Higher precision on ambiguous queries | Extra latency and a second model call | Larger corpora with noisy chunks |
| Agentic RAG (loop calls retrieval as a tool) | Can retry, reformulate queries, combine sources | Higher cost, harder to bound and test | Multi-hop questions, research-style tasks |
| Fixed single-tool agent | Predictable, easy to sandbox | Cannot adapt to unexpected sub-tasks | Well-scoped automations (one action, one purpose) |
| Multi-tool / multi-agent loop | Handles open-ended tasks | Larger blast radius, harder to evaluate | Complex workflows with human review gates |
A whole-document embedding blurs together many different topics into one vector, so a query about one paragraph will not reliably match. Chunking keeps each vector focused on one coherent unit of meaning.
There is no universal number; it depends on the embedding model's effective context and the granularity of your content. Most teams start around a few hundred tokens with modest overlap and tune based on retrieval evaluation results.
Vector search uses one embedding pass to approximate relevance quickly across a huge index. A reranker runs a more expensive model over just the shortlist, cross-referencing the query and each candidate directly, which produces more accurate ordering at a much smaller scale.
Not quite. A chatbot with plugins may call one tool per turn under human direction. An agent loop can make multiple internal decisions and tool calls before producing a single final answer, without a human in between each step.
The model is given tool schemas (name, description, argument types) alongside the conversation, and it is trained/prompted to emit a structured call instead of prose when a tool matches the task. The calling code then executes the real function.
An explicit bound set by the developer: a maximum iteration or recursion count, a token/cost budget, or a required "final answer" signal the model must produce to exit the loop.
Yes. Plain RAG (retrieve once, generate once) needs no loop at all. An agent loop can exist without any retrieval, using only other tools like a calculator or a calendar API.
Grounding means the model's answer is tied to specific retrieved evidence rather than generated purely from its training data, which is what makes citations and hallucination reduction possible.
They implement the same two mental models with different ergonomics: LlamaIndex leans toward data-loading and indexing convenience, LangChain toward composable chains, and LangGraph toward explicit stateful loops with cycles. The underlying pipeline and loop concepts do not change.
A mismatch or drift between the embedding model used at index time versus query time, or a chunking strategy that splits the actual answer across two separate chunks so neither one alone is retrieved as fully relevant.
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.
Reviewed by Chris St. John·Last updated Jul 15, 2026