The RAG Pipeline and Agent Loop Mental Model

Every page in this section, whether it covers LangChain, LlamaIndex, vector databases, or the Model Context Protocol, is an implementation detail sitting on top of exactly two mental models.

The first is the retrieval-augmented generation (RAG) pipeline: a fixed sequence of stages that turns a pile of documents into grounded, cited answers.

The second is the agent loop: a cycle that lets a language model decide what to do next, call a tool, observe the result, and decide again.

Understanding these two models first makes every specific library choice - which vector store, which reranker, which orchestration framework - a much smaller decision than it first appears.

Summary

• RAG grounds generation in retrieved evidence through a chunk-embed-retrieve-rerank-generate pipeline; agents wrap generation in a loop that can act on the world through tools.
• Insight: Neither a raw LLM call nor a single database query solves "answer this question using our data and take an action" - you need a pipeline and a loop, and most bugs come from treating either as a single step.
• Key Concepts: chunking, embedding, retrieval, reranking, generation, agent loop, tool use, grounding.
• When to Use: Domain-specific question answering, internal knowledge search, customer support over private docs, and any task requiring multi-step reasoning with external actions (booking, querying, writing files).
• Limitations/Trade-offs: RAG only helps when the answer actually lives in your corpus; agent loops trade determinism and cost predictability for flexibility, and both add latency stages that a single model call does not have.
• Related Topics: vector search, prompt engineering, function calling, orchestration frameworks.

Foundations

Retrieval-augmented generation exists because language models have two hard limits: a fixed context window and a training cutoff.

You cannot paste your entire knowledge base into a prompt, and the model cannot know about documents written after it was trained.

RAG solves both by retrieving only the relevant slice of text at query time and handing it to the model as context.

The pipeline has five conceptual stages, and each one exists to fix a specific failure of the stage before it.

Chunking splits long documents into retrieval-sized pieces, because embedding an entire PDF as one vector loses the fine-grained meaning of any single paragraph inside it.

Embedding converts each chunk into a vector, a list of numbers that places semantically similar text near each other in a high-dimensional space.

Retrieval takes a query, embeds it the same way, and finds the nearest chunk vectors, which is the pipeline's first real point of failure since "nearest" is not the same as "correct."

Reranking takes the top candidates from retrieval and re-scores them with a more expensive, more accurate model, because fast approximate search over millions of vectors necessarily leaves relevant results ranked lower than they should be.

Generation is the final step, where the LLM writes an answer using only the retrieved (and reranked) context, ideally with citations back to source chunks.

The agent loop is a different but complementary model.

Where RAG is a straight pipeline (data flows one direction, once), an agent loop is a cycle: the model receives state, decides on an action, an action executes, and the result feeds back in as new state.

The simplest mental picture is a thermostat: sense the temperature, decide whether to heat or cool, act, sense again.

For an LLM agent, "sense" is the conversation and tool results so far, "decide" is the model choosing a tool call or a final answer, and "act" is actually invoking that tool.

Tool use (sometimes called function calling) is what makes the loop useful: instead of the model only producing prose, it produces a structured request - a function name and arguments - that your code executes and feeds back as an observation.

Mechanics & Interactions

The two models interact constantly in real systems, because the most common agent tool is a RAG retriever.

An agent loop's "decide" step commonly resolves to "call the search_docs tool," which triggers the entire RAG pipeline, whose output becomes the "observe" step feeding the next decision.

This is why the two mental models belong on the same page: RAG without a loop is a single-shot Q&A system, and an agent loop without RAG has no way to ground its actions in your actual data.

Inside the RAG pipeline, the embedding model used at index time and the embedding model used at query time must be the same model (or at least compatible), because vectors from different models do not share a coordinate space.

This is a subtle but critical mechanical detail: swapping embedding models means re-embedding and re-indexing every chunk, not just the new queries.

Retrieval quality is usually described with recall (did the relevant chunk appear at all in the candidate set) and precision (how much of the candidate set is actually relevant), and these pull against each other.

A larger top_k improves recall but dilutes precision and increases the tokens sent to the generation step, which is exactly the problem reranking is designed to solve: retrieve broadly and cheaply, then narrow accurately and expensively.

In the agent loop, the critical mechanical detail is termination.

An LLM does not inherently know when to stop calling tools, so every real agent framework enforces a bound - a maximum number of iterations, a recursion limit, or an explicit "final answer" signal the model must emit.

Without that bound, a model that keeps deciding "I need more information" will call tools indefinitely, burning cost and time with no natural exit.

def run_agent_loop(model, tools: dict, state: list, max_steps: int = 8):
    for _ in range(max_steps):
        decision = model.decide(state)          # LLM picks: answer or call a tool
        if decision.is_final_answer:
            return decision.content
        result = tools[decision.tool_name](**decision.args)  # act
        state.append({"tool_result": result})   # observe -> feeds next decide()
    return "stopped: exceeded max_steps"          # explicit bound, not a hang

This snippet is not a production agent (real frameworks like LangGraph handle branching, persistence, and parallel tool calls), but it makes the loop's shape explicit: decide, act, observe, repeat, with a hard stop.

Advanced Considerations & Applications

At scale, both models develop failure modes that do not show up in a demo with ten documents.

RAG pipelines built on a single flat vector index degrade as the corpus grows into the millions of chunks, because approximate nearest-neighbor search trades recall for speed, and that trade-off gets worse as the index grows unless you add filtering, hybrid search (combining keyword/BM25 with vector similarity), or hierarchical retrieval.

Agent loops develop a parallel failure mode: as you add more tools, the model's decision step gets harder, because it must now choose correctly among a larger, more ambiguous set of options, and small prompt or tool-description changes can shift behavior unpredictably.

Observability differs between the two models as well.

RAG pipelines are debugged by inspecting what was retrieved - if the answer is wrong, the first question is always "was the right chunk even in the candidate set?"

Agent loops are debugged by inspecting the trace of decisions and tool calls, since the failure is often several steps upstream of the final wrong answer, not in the last step itself.

Security also diverges: RAG's attack surface is data poisoning (malicious content injected into the corpus that gets retrieved and treated as trusted context), while an agent loop's attack surface is prompt injection through tool outputs - a tool result that itself contains instructions the model may follow.

Common Misconceptions

• "RAG just means adding a vector database." A vector database is one component of one stage (retrieval); without deliberate chunking, a matched embedding model, and often reranking, a vector database alone produces mediocre, hard-to-debug answers.
• "More retrieved context is always better." Stuffing more chunks into the prompt increases the chance of irrelevant or contradictory text drowning out the correct answer, and it costs more tokens; recall and precision must be balanced, not maximized independently.
• "Agents are autonomous and unsupervised by design." Every production agent loop has an explicit bound - a step limit, a tool allowlist, or a human-approval gate - because unbounded loops are a liability, not a feature.
• "A bigger model fixes bad retrieval." If the right information never enters the context window, no amount of model capability recovers it; retrieval quality is often the higher-leverage fix over model size.
• "Tool use is just structured output." Structured output is necessary but not sufficient; the loop also needs a termination condition, error handling for failed tool calls, and a way to feed results back as new state.

FAQs

Related

• RAG Basics - hands-on walkthrough of the chunk-embed-retrieve-generate pipeline
• Retrieval & Re-ranking - deeper mechanics of the retrieval and reranking stages
• Vector Databases - where embeddings are stored and searched
• LangGraph & Agent Loops - implementing the agent loop as a state graph
• Tool Use & Function Calling - the mechanics behind the "act" step
• Evaluation & Guardrails - measuring retrieval and agent quality

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.