Skip to main content Skip to footer

Module 4 Overview:

Agents need access to accurate, current, and relevant information to make good decisions. A language model trained on a fixed dataset has a knowledge cutoff, cannot access private or internal data, and can generate confident but incorrect responses when it lacks context. This module introduces the technique that solves that problem: Retrieval-Augmented Generation.

RAG allows an AI system to pull in relevant information at the moment it is needed, rather than relying solely on what it learned during training. Agentic RAG takes this further by making retrieval a dynamic, iterative part of an agent's reasoning process rather than a fixed step.

In this module, you’ll learn:

  • What RAG is and how it works
  • How RAG differs from AI agents and why that distinction matters
  • Where traditional RAG falls short in real-world systems
  • What agentic RAG is and how it addresses those limitations
  • When agentic RAG is the right architectural choice

By the end of this module, you will have a clear understanding of what RAG is, how it differs from agentic RAG, and when to use each approach.

Estimated time length: 25 minutes

Lesson 1: What is RAG?

Learning objectives

By the end of this lesson you will be able to:

  • Define RAG and explain why it exists
  • Describe the core steps in a standard RAG pipeline
  • Explain the specific problems RAG is designed to solve

Language models are trained on large datasets, but that training has a cutoff. The model knows what it knew when it was trained. It does not automatically update as new information becomes available, and it has no inherent access to private, proprietary, or domain-specific data. For many real-world applications, that is a significant gap.

Think of it like a GPS that was loaded with maps from two years ago. It knows the roads it learned, but it has no way to know about a new highway, a recent road closure, or an updated speed limit. It can still get you somewhere, but not always by the most accurate or current route. Language models without external knowledge access have the same problem: confident, fluent responses that may be outdated, incomplete, or simply wrong.

Diagram of RAG vs no RAG

Retrieval-Augmented Generation, or RAG, addresses this by introducing a retrieval step before generation. Instead of relying only on the model's internal knowledge, the system retrieves relevant information from an external source at the time of the request and uses that information to ground the response. The model is no longer limited to what it learned during training. It can answer based on what is in your documents, your databases, or any external knowledge source you connect it to.

The problems RAG solves

Without retrieval, language models face four recurring problems in production environments.

  • Outdated responses: Training data has a cutoff date. Anything that happened, changed, or was published after that cutoff is invisible to the model. In fast-moving domains like finance, healthcare, or technology, this creates real accuracy problems.
  • No access to private data: Models trained on public data have no knowledge of your internal documents, proprietary systems, customer records, or organization-specific processes. RAG connects the model to that data at query time.
  • Incomplete answers: If the training data did not cover a topic thoroughly, the model may produce answers that are technically coherent but substantively incomplete. Retrieval fills that gap with actual source material.
  • Hallucinations: When a model lacks context, it sometimes generates responses that sound plausible but are factually incorrect. Grounding the model's response in retrieved content significantly reduces this risk.

How RAG works

A standard RAG pipeline combines two steps: retrieval and generation. The system first retrieves relevant information from a knowledge source, then uses that information as additional context for generating a response.

Step 1: Query: 

A user submits a request. The system processes that request to understand what information is needed.

Step 2: Retrieval: 

The system searches a knowledge base, which might be a document store, a vector database, an internal wiki, or any other structured source. It returns the content most relevant to the query, typically using semantic search techniques that match meaning rather than just keywords.

Step 3: Augmentation:

The retrieved content is added to the model's input alongside the original query. The model now has both the question and relevant context to work with.

Step 4: Generation: 

The model generates a response using both the query and the retrieved information. The output is grounded in actual source material rather than in the model's training data alone.

RAG working
Key insight

RAG does not make a language model smarter. It makes it better informed. The quality of the output depends directly on the quality of the knowledge base and the accuracy of the retrieval. A well-designed RAG system produces responses that are more accurate, more current, and more traceable than a model working from training data alone.

Where traditional RAG has limits

RAG works well when the task is straightforward: a clear query, a clean knowledge base, and an answer that lives in a single retrievable source. But standard RAG pipelines have a structural constraint worth understanding. They retrieve once. The query goes in, documents come back, and the model generates a response from whatever was returned. If that first retrieval is incomplete or off-target, there is no mechanism to try again. The system does not evaluate whether the retrieved content actually answers the question. It just uses what it has.

This works fine for direct question-answering and document lookup. It becomes a real limitation when information is distributed across multiple sources, queries are ambiguous, or a task requires more than one retrieval step to build a complete picture. Understanding this constraint is what makes agentic RAG, covered in the next lesson, a meaningful step forward rather than just a more complex version of the same thing.

Up next: RAG and AI agents are often mentioned together, but they are not the same thing. The next lesson draws a clear line between them.

Lesson 2: RAG vs AI Agents

Learning objectives

By the end of this lesson you will be able to:

  • Explain how RAG and AI agents differ in purpose and function
  • Describe the scenarios where each is the right choice
  • Understand how RAG and agents work together in a combined system

RAG and AI agents are frequently discussed in the same context, and in many systems they appear together. That proximity can make it easy to conflate them. They are not the same thing, and understanding the difference matters when you are deciding how to architect a system.

Here is the clearest way to put it: RAG helps a system know more. An agent helps a system do more. They solve different problems, and they are strongest when they work together.

What each one actually does

RAG is a technique for knowledge access. Its job is to retrieve relevant information from an external source and give that information to a model before it generates a response. RAG is fundamentally about improving the quality of what the model knows at the moment it needs to respond. It does not plan, act, or make decisions. It retrieves.

An AI agent is a system designed to work toward a goal. It perceives its environment, reasons about what needs to happen, decides which actions to take, uses tools to execute those actions, and evaluates the results. An agent may use retrieval as one of its tools, but retrieval is not the whole workflow. The agent decides when to retrieve, what to retrieve, and what to do with what it finds.

A useful way to frame the difference: RAG answers the question, what information does the model need? An agent answers a different question: what needs to happen next?

Rag vs agent diagram

A concrete example

Imagine a user asks: what is our refund policy for international orders?

A RAG system retrieves the relevant policy document and generates an answer based on that document. The interaction is complete.

An AI agent handling the same request might retrieve the policy, then check the customer's order details, determine whether the specific order qualifies under the policy, update the support ticket, and recommend or trigger the next action. The agent used retrieval as one step in a broader workflow, not as the entire workflow.

 

RAG

AI Agent

Primary purpose

Retrieve relevant information to improve a response

Work toward a goal and complete a task

Core function                

Finds and adds context to a generation request

Decides, plans, acts, and adjusts

Workflow        

Query > retrieve > generate

Understand goal → plan → use tools → act → adjust

Relationship to knowledge             

Provides external knowledge to the model

Uses knowledge as one input among many to make decisions

Autonomy

Limited – follows a fixed retrieval flow

Higher – adapts based on what it finds

Best suited for

Grounded Q&A, document lookup, knowledge search

Multi-step workflows, decision support, task execution

Example

Answering a policy question using a knowledge base

Resolving a support case using policy, account data, and workflow actions

RAG and agents are most powerful when combined. RAG gives the agent access to accurate, current knowledge. The agent decides how to use that knowledge in context, which actions to take, and how to complete the task. A RAG system without agency can answer questions. An agent with RAG can get things done.

Key insight

Not every RAG system is an agent. A system can retrieve documents and generate answers without any ability to plan, decide, or take action. And not every agent needs RAG, but most do, because good decisions require accurate information. Understanding which capability a problem requires is one of the more practical skills in agentic system design.

Up next: Now that the distinction is clear, the next lesson introduces agentic RAG and how it addresses the limitations of traditional RAG.

Lesson 3: What is Agentic RAG?

Learning objectives

By the end of this lesson you will be able to:

  • Define agentic RAG and explain how it differs from traditional RAG
  • Describe how the agentic RAG workflow operates
  • Explain the advantages agentic RAG offers over a standard RAG pipeline

In the previous lesson, you saw that RAG gives AI systems access to external knowledge, and that agents use that knowledge to make decisions and take action. Agentic RAG brings those two ideas together.

Standard RAG has a structural constraint. It retrieves once. The query goes in, documents come back, and the model generates a response from whatever was returned. If the first retrieval is incomplete, off-target, or only partially relevant, the system has no mechanism to course-correct. It proceeds to generation with whatever it has, even if that content does not fully address the question. Agentic RAG addresses this directly.

    Defining agentic RAG

    Agentic RAG is an approach where an AI agent controls the retrieval process dynamically, rather than following a fixed single-pass pipeline. Instead of retrieving once and generating from whatever comes back, the agent decides when to retrieve, what to search for, evaluates whether the results are sufficient, and retrieves again if they are not. Retrieval becomes one of the tools the agent uses as it reasons through a problem, rather than a preprocessing step that happens before any reasoning begins.

    The underlying technology is the same. The system still retrieves external content and uses it to ground a response. The difference is in how that retrieval is managed and how many times it can happen.

    Agentic RAG work

    How agentic RAG works

    The workflow for agentic RAG is iterative rather than linear. A typical sequence looks like this:

    1. Query analysis: 

    The agent receives a request and analyzes it. Rather than passing the raw query to a retrieval system, it determines what information is actually needed, identifies ambiguities, and decides on a retrieval strategy.

    2. Initial retrieval: 

    The agent sends a refined query to the most appropriate knowledge source and retrieves initial results. This might be a document store, a vector database, a structured data source, or an external system.

    3. Evaluation: 

    The agent reviews the retrieved content. Is it relevant? Is it complete? Does it actually answer the question, or does it raise new questions? If the results are insufficient, the agent does not proceed to generation.

    4. Refinement and iteration: 

    If the initial retrieval is not enough, the agent rewrites the query, searches a different source, or issues additional targeted queries. This loop continues until the agent has enough information to produce a reliable response.

    5. Generation: 

    Once the agent has sufficient, validated information, it generates a response grounded in that retrieved content. The output is more accurate and more complete than what a single-pass system could produce.

     

    Traditional RAG

    Agentic RAG

    Retrieval

    Fixed – happens once per request

    Dynamic – happens as many times as needed

    Query refinement

    None – uses the original query as-is

    Active – rewrites and refines queries based on results

    Multiple sources

    Usually tied to one source or fixed pipeline

    Can route to different sources based on the task

    Validation

    No – uses retrieved content without evaluating it

    Yes – evaluates quality before generating a response

    Iteration

    None – single pass

    Yes – loops until the information is sufficient

    Speed and cost                           

    Faster and less expensive

    Slower and more resource-intensive

    Best for

    Clear questions with answers in a known source

    Complex, multi-step tasks with uncertain or distributed information

    Why This Matters

    The key advantage of agentic RAG is that it allows systems to adapt. Instead of depending entirely on the first retrieval step, the system can adjust its approach based on what it finds. This leads to more complete and more reliable results, especially when information is distributed or incomplete.

    It also aligns naturally with how agents operate. Agents are designed to work toward a goal over multiple steps. Retrieval becomes one of the tools they use along the way, not the entire workflow. This shift, from one-time retrieval to iterative retrieval, is what enables more advanced use cases. It is also what sets up the next step, where we look at how this approach is applied in real systems.

    Up next: The final lesson looks at where agentic RAG shows up in practice and how to recognize when it is the right choice for a given problem.

    Lesson 4: Agentic RAG Use Cases and When to Use It

    Learning objectives

    By the end of this lesson you will be able to:

    • Identify the scenarios where agentic RAG creates the most value
    • Recognize when traditional RAG is sufficient
    • Apply the right retrieval architecture to a given problem

    Agentic RAG is a more capable approach than traditional RAG, but more capable does not always mean better for a given situation. It introduces additional complexity, requires more compute, and takes more time. For many straightforward use cases, a standard RAG pipeline is the right choice. Understanding when each approach is appropriate is one of the most practical things you can take from this module.

    When agentic RAG is the right choice

    Agentic RAG creates the most value in situations where the limitations of traditional RAG would otherwise compromise the output. A few signals that point toward the agentic approach:

    • The answer requires multiple retrieval steps: Some questions cannot be fully answered with a single query. A question like 'which of our products is most relevant for a customer in this industry with this use case?' requires retrieving product information, retrieving industry-specific context, and synthesizing across both. Agentic RAG can handle this. Traditional RAG cannot.
    • Information is distributed across multiple systems: In enterprise environments, relevant information is rarely in one place. An agentic system can route queries to the right source, whether that is a document store, a CRM, a database, or an external API, and combine the results. Traditional RAG is typically tied to a single source.
    • Queries are complex or evolve during the task: Some tasks start with a broad question that can only be refined after initial results come back. Agentic RAG supports this kind of iterative narrowing. Traditional RAG takes the query at face value and retrieves once.
    • Accuracy is critical and errors are costly: When the stakes of a wrong answer are high, such as in compliance, legal, medical, or financial contexts, the validation loop in agentic RAG provides a meaningful reliability advantage over single-pass retrieval.

    Real-world examples

    These patterns show up across industries in predictable ways.

    A basic RAG system retrieves a policy and answers a question. An agentic RAG system retrieves the policy, checks the customer's account history, determines whether their specific situation qualifies, and recommends or triggers the appropriate next action. The difference is that retrieval becomes part of a workflow rather than the entire workflow.

    A business question like 'what are the regulatory risks for this product in these three markets?' cannot be answered from a single document. An agentic system can issue multiple targeted queries across different knowledge sources, evaluate consistency across results, identify gaps, and synthesize a complete answer. This is the kind of work that traditionally required a human analyst.

    Clinical decision support systems need to draw on patient records, clinical guidelines, recent research, and drug interaction databases simultaneously. Agentic RAG allows a system to retrieve across all of these, validate consistency, and surface the most relevant information for a clinician's specific question.

    An agent reviewing code for compliance or quality can retrieve relevant documentation, past decisions, style guides, and dependency specifications iteratively, building context before making a recommendation rather than relying on a single lookup.

    When traditional RAG is enough

    Not every problem needs the complexity of agentic RAG. If the question is well-defined, the answer lives in a known source, and a single retrieval step consistently returns sufficient context, a standard pipeline is faster, cheaper, and easier to maintain.

    Direct question-answering from a single knowledge base, document search and summarization, and FAQ systems are all cases where traditional RAG performs well and agentic RAG would add cost without proportionate benefit. The rule of thumb: start with the simpler architecture and only move to agentic retrieval when the limitations of traditional RAG are actually showing up in your outputs.

    Key insight

    Agentic RAG and traditional RAG are not competing for the same job. Agentic RAG earns its added complexity and cost when a task involves multiple retrieval steps, multiple sources, or high stakes where a single pass cannot be trusted. Traditional RAG remains the right fit when the question is well-defined and a single retrieval step reliably returns what is needed. The goal is not to pick the more capable architecture by default, it is to match the architecture to what the problem actually requires.

    Up next: Before moving to the next module, test your understanding with a quick quiz.

    Module wrap-up

    The RAG and Agentic RAG module is now complete!

    You now have a solid understanding of how retrieval works, why it matters for AI agents, and how the architecture of retrieval changes when you move from a standard pipeline to an agentic approach. This knowledge is directly applicable to the system design decisions you will encounter in the later modules of this course.

    At this point, you should be able to:

    • Define RAG and explain the four core problems it solves
    • Describe the steps in a standard RAG pipeline and explain where its structural limits lie
    • Distinguish between RAG and AI agents and explain how they complement each other
    • Define agentic RAG and describe how its iterative workflow addresses the limits of standard pipelines
    • Recognize the scenarios where agentic RAG is the right architectural choice and where standard RAG is sufficient

    Put it to the test

    Take the short quiz below to check your understanding before moving on.

    Want to see agents in action?

    Everything you just learned comes to life in Neuro San, Cognizant AI Lab's open source framework for building and deploying multi-agent systems. If you want to explore how real agent networks are structured before diving into the next module, the repo is a great place to start.

    Star Neuro San on GitHub

    Next up:

    In the last module, we will cover responsible and safe agent systems: what responsible AI means in practice, how to handle data securely, and the design principles that make agentic systems trustworthy in production. 

    Next module: Responsible and Safe Agent Systems

    Pages

    Overview

    Overview

    1. RAG Definition

    1. RAG Definition

    2. RAG vs Agents

    2. RAG vs Agents

    3. Agentic RAG Definition

    3. Agentic RAG Definition

    4. Agentic RAG Use Cases

    4. Agentic RAG Use Cases

    Check your understanding

    Check your understanding
    Previous lesson
    Next lesson

    Continue Your Learning Journey

    Responsible & Safe Agent Systems
    Get started