Skip to main content Skip to footer

Module 5 Overview:

Building an agent system that works in a demo is one thing. Building one that works safely and reliably in production is another problem entirely. As agents become more autonomous and more deeply connected to real enterprise systems, the design decisions you make around security, data handling, observability, and testing start to have real consequences.

In an HFS Research report, they found that 22% of enterprises running multi-agent systems encountered emergent behaviors they did not anticipate, 21% experienced cascading failures where one broken agent took down others, and 18% discovered auditability gaps where thousands of decisions had been made with no interpretable lineage. These are not edge cases. They are the predictable result of deploying capable systems without the right foundations underneath them.

This module is about building those foundations. The concepts here are not abstract principles. They are practical design decisions that determine whether an agent system you build is trustworthy enough to run in production.

In this module, you’ll learn:

  • What responsible AI means and why it applies directly to agent systems
  • How to handle sensitive and personal data safely in agentic workflows
  • What observability means in the context of agents and why it is essential in production
  • How to think about security threats specific to agentic systems
  • How to approach testing in systems where behavior is not fully deterministic

By the end of this module, you will have a clear understanding of what goes into building responsible agentic systems that are safe for production.  

Estimated time length: 30 minutes

Lesson 1: What is Responsible AI?

Learning objectives

By the end of this lesson you will be able to:

  • Define responsible AI and explain why it matters for agent systems specifically
  • Describe the core principles of responsible AI and what each means in practice
  • Explain why responsible AI is a design concern, not just a compliance concern

Responsible AI is the practice of designing, deploying, and operating AI systems in ways that are safe, reliable, transparent, and aligned with human oversight. It is not a feature you add before release. It is a set of principles that shape how you build from the beginning.

For earlier AI systems, these concerns were real but limited in scope. A model that generates text can produce harmful or inaccurate outputs, but it cannot take action on its own. Agent systems are a different situation. They can retrieve information, interact with tools, trigger workflows, and make decisions across multiple steps with limited human involvement in between. That expanded capability is exactly what makes them useful. It is also what makes responsible AI genuinely important rather than theoretical.

An agent system that behaves unreliably, exposes sensitive data, or takes unintended actions does not just produce a bad output. It can have cascading consequences across every system and workflow it is connected to. The HFS report puts it plainly: enterprises treated agents like software, but they behave like employees. You would not give a new employee unrestricted access to every system on day one. The same logic applies here.

The core principles

Responsible AI in the context of agent systems is grounded in a set of principles that are now reflected across major industry frameworks, including NIST AI RMF, the EU AI Act, and ISO 42001. They are worth understanding clearly because they map directly to design decisions.

Safe and reliable

 Agent systems should behave consistently and as intended, even as they scale or encounter conditions they were not explicitly designed for. This requires ongoing testing, monitoring, and evaluation, not just a check before launch. Failures in production are often the ones no one anticipated in development.

Transparent and explainable

Organizations need visibility into how agent systems behave and why. For agents specifically, that means being able to trace what information was retrieved, which tools were called, what decisions were made, and how the system arrived at a particular output. Without this, debugging and accountability are nearly impossible.

Human-centered

Responsible AI is not about removing people from the process. It is about building systems that support human oversight, respect human values, and include clear escalation paths for decisions that should not be made autonomously. This is especially important in workflows where judgment is required.

Secure and privacy-protecting

Agents that are connected to enterprise systems, databases, and APIs have significant reach. Protecting sensitive information, controlling access, and designing against misuse are foundational requirements, not optional add-ons. More on this in Lesson 2.

Fair and inclusive

AI systems should produce equitable outcomes across different users, groups, and contexts. This includes testing for bias, thinking carefully about representation in training data and workflows, and ensuring systems do not systematically disadvantage any group.

Accountable and auditable

Organizations should be able to review what their agent systems did, when, and why. This requires governance processes, structured logging, and the ability to audit system activity when something goes wrong or when compliance requires it.

Scalable and sustainable

Systems that work at small scale should be designed to remain reliable as usage grows and workflows evolve. Responsible AI also includes thinking about long-term operational costs, maintenance overhead, and the broader organizational impact of automated decision-making.

Safe and reliable


Applying rigorous testing and monitoring to ensure our AI solutions perform as intended at any scale.

Transparent and explainable

Enhancing explainability, ensuring that AI practices promote trust, openness, traceability and clear human decision-making.

Human focused

Prioritizing human needs at the core of development, ensuring AI practices align with relevant laws to respect, empower and benefit people.

Secure and privacy-enhanced

Safeguarding personal information and protect AI systems from threats and vulnerabilities.

Fair and inclusive


Designing AI solutions that are fair, accessible and representative, striving for equitable outcomes.

Accountable and auditable

Ensuring appropriate human oversight and taking responsibility for the outcomes and impacts of AI solutions.

Scalable and sustainable

Producing AI solutions that can grow and adapt over time, while also considering environmental and societal impacts.

Key insight

Responsible AI is not a layer you add before deploying. It is a set of design decisions you make throughout the build. The HFS report found that the enterprises running the most successful multi-agent systems are not the ones with the most agents. They are the ones that deliberately designed for autonomy, accountability, and human-machine collaboration from the start.

Up next: The next lesson digs into one of the most concrete responsible AI challenges in agent systems: how to handle sensitive and personal data safely.

Lesson 2: Securing Sensitive and Personal Data

Learning objectives

By the end of this lesson you will be able to:

  • Explain why data protection is a specific concern in agentic systems and how exposure happens in practice
  • Describe the key practices used to protect sensitive and PII data in agent workflows
  • Understand what Sly Data is and why separating sensitive data from LLM prompts matters

AI agents often work with information that is sensitive, private, or business-critical. In enterprise workflows, that frequently includes personally identifiable information, known as PII: names, email addresses, account numbers, phone numbers, government-issued identifiers. Because agents can retrieve, process, and pass information between tools and systems dynamically, protecting this data requires deliberate design rather than assumptions.

The challenge is that agents, unlike traditional applications, do not follow a fixed, predictable path. They decide what data to retrieve based on context. They pass state between tools. They maintain memory across steps. This flexibility is what makes them powerful. It also creates data exposure risks that would not exist in a conventional rule-based system.

Why the stakes are higher than they appear

One of the core problem with agents is that they look at the utility of data, not the risk of data. A human customer support representative who has access to PII knows not to share customer A's data with customer B. An agent without explicit guardrails may surface that data in a context where it should not appear, simply because it was retrieved as part of completing a task. The agent is not being malicious. It is just optimizing for task completion.

This is compounded by where sensitive data ends up. One of the most significant risks in agentic systems is that enterprise data gets embedded directly inside LLM prompts. The language model is a probabilistic reasoning environment, not a controlled execution environment. Data that enters a prompt can be reproduced, reinterpreted, or included in outputs in ways that are difficult to predict or audit. Microsoft's security team has documented cases where sensitive PII, credit card numbers, and custom data types in prompts were processed and surfaced in ways that violated data loss prevention policies, which is why their Purview platform added runtime DLP controls specifically for agent prompt flows in 2026.

What the data shows

How sensitive data gets exposed

Understanding the failure patterns is the first step to designing against them. There are four common ways sensitive data escapes its intended boundaries in agent systems. In each case, the exposure is predictable once you know what to look for, and in most cases, entirely preventable with the right design decisions in place. Let's take a closer look at the different ways so we know how to avoid them.

4 box diagram
  • Oversharing in responses: An agent with broad retrieval access may surface confidential information in a response the user was not supposed to receive. If access controls are not scoped carefully to the specific workflow and user, the model may retrieve and include data it should not have touched.
  • Uncontrolled data flow between tools: As agents pass context from one tool to the next, sensitive information can travel further through a system than intended. Without clear data boundaries at each handoff, PII or confidential state can reach external systems or logs where it should not appear.
  • Persistence in memory and logs: Some agent architectures maintain memory across sessions or log intermediate reasoning steps. If sensitive data ends up in those logs without appropriate redaction or access controls, it creates a long-term exposure risk that is difficult to detect and even harder to remediate.
  • Direct prompt exposure: Sensitive enterprise data embedded inside a prompt sent to a language model is the highest-risk pattern. The LLM processes everything in its context window. Separation between sensitive operational data and the probabilistic reasoning layer is not just good practice. It is the design principle that makes agentic systems genuinely safe to deploy in enterprise environments.

How to protect sensitive data in agent systems

Secure data handling in agent systems requires controls at multiple layers of the workflow, not just at the perimeter.

  • Least privilege access: Every agent should only be able to access the data it actually needs for the specific task it is performing, scoped to the workflow. Broad access granted for convenience is a security liability. This principle is consistent across OWASP, NIST, and Microsoft's published guidance for agentic systems.
  • PII masking and redaction: Sensitive fields should be detected and masked before they reach parts of the system where they are not needed, including LLM prompts. Tools like Microsoft Presidio and AWS Macie provide automated PII detection and redaction that can be integrated into agent workflows.
  • Separate sensitive state from LLM context: Rather than embedding operational state directly in prompts, well-designed systems use controlled data channels that allow the LLM to reason about a task without seeing the raw sensitive data. This is the architectural principle behind neuro-san's Sly Data pattern, covered below.
  • Restrict memory persistence: Not all agent state needs to persist between sessions. Systems should be explicit about what gets stored, for how long, under what conditions, and who can access it. Session isolation, where each conversation runs in a separate context with no memory carryover, is appropriate for many enterprise use cases.
  • Monitor and audit data access: Organizations need visibility into what data their agents are accessing, when, and in what context. This supports both real-time security monitoring and after-the-fact compliance auditing. Audit logs should capture data access events with enough context to reconstruct what happened.
Spotlight: Sly Data in Neuro San

One of the core challenges covered in this lesson, keeping sensitive data out of LLM prompts, is something Neuro San addresses directly with a pattern called Sly Data. Sly Data (Structured Layered Exchange Data) is a deterministic, schema-controlled data contract used to exchange state between agents and systems without exposing sensitive information to the language model. Rather than embedding enterprise data directly inside a prompt, agents exchange that information through controlled private data channels, ensuring every exchange is controlled, predictable, traceable, and auditable.

This pattern is covered in depth when you build with Neuro San in later courses.

Up next: Now that you understand how to protect data, the next lesson covers how to see what your agent is actually doing while it runs.

Lesson 3: Observability in Agent Systems

Learning objectives

By the end of this lesson you will be able to:

  • Define observability and explain why it is essential for agents specifically
  • Describe what teams instrument and monitor in production agentic systems
  • Explain the relationship between observability, debugging, and trust

When a traditional application fails, you trace through its code, find the function that broke, and fix it. The path is deterministic. The same input reliably produces the same output, which means failures are reproducible and traceable.

Agent systems work differently. The same user request can trigger different tool sequences, retrieve different documents, and produce different responses each time it runs. A single request might travel through a knowledge search, several tool calls, a reasoning loop, and a handoff between agents before a response is produced. The final answer might be wrong, but the path that led to it is invisible unless you designed the system to expose it.

This is the problem observability solves. Observability is the practice of instrumenting a system so you can see inside it while it runs. For agent systems specifically, it means having a structured record of what the agent retrieved, which tools it called, how it reasoned at each step, and what decisions it made along the way. Without that record, you are left guessing. You can see the output was wrong. You cannot reliably explain why, fix the root cause, or prove to an auditor that the system behaved appropriately.

Gartner's research found that over 40% of agentic AI projects are predicted to be canceled by 2027 due to escalating costs and monitoring gaps. The failure mode is not that the technology stops working. It is that teams cannot see what is happening, cannot control costs, and cannot defend the system's behavior when something goes wrong.

Why traditional observability is not enough

Traditional software observability is built on three pillars: metrics (quantitative signals like latency and error rates), logs (timestamped records of events), and traces (records of how a request moved through a system). These are well-understood and widely used. They are also insufficient for agents on their own.

Traditional observability relies on three foundational pillars: metrics, logs, and traces. These provide visibility into system performance, help diagnose failures, and support root-cause analysis. They are well-suited for conventional software systems. However, AI agents are non-deterministic and introduce new dimensions: autonomy, reasoning, and dynamic decision-making that require a more advanced observability framework. 

The gap is decision-making visibility. Standard traces can tell you that a tool was called. They cannot tell you why the agent chose that tool over another, what it was reasoning about when it made that decision, or whether the reasoning was sound. When your agent invokes three tools, loops twice, and produces an incorrect answer, standard traces show what happened but not why. Agent observability closes this gap by instrumenting the decision-making layer itself.

The five pillars of agent observability

Agent observability builds on traditional methods and adds two critical components: evaluations and governance. Together they form five pillars that every production agent system needs. 

    5 boxes in a diagram
    1. Metrics

    Metrics are quantitative signals that tell you how the system is performing over time. For agent systems, the most important metrics to track are:

    • Token usage and cost: how many tokens are consumed per request and per agent, and what that costs. Agents that chain multiple LLM calls and tool calls can incur costs that are unpredictable without per-trace cost attribution. Agents that autonomously chain multiple LLM and API calls can incur unpredictable costs, making real-time cost tracking and per-trace cost attribution essential for production deployments.
    • Latency: end-to-end response time broken down by agent, tool, and retrieval step. This tells you where bottlenecks are.
    • Success rate: the proportion of agent runs that complete without errors, hallucinations, or policy violations.
    • Hallucination rate: how frequently the agent produces outputs not grounded in retrieved evidence.
    2. Logs

    Logs are timestamped event records. For agents, logging needs to be structured and semantically rich, not just raw text. The industry is shifting from reactive log-based monitoring to proactive, structured tracing with typed observation data. Rather than parsing unstructured logs after failures, teams now instrument agents with rich semantic types: tool calls, retrieval steps, guardrail checks, for real-time insight into agent behavior.

    Every significant event in the agent's workflow should produce a structured log entry: the query received, the retrieval queries issued, each tool call made, each decision point reached, and the final output returned. These logs are what compliance teams, auditors, and security teams rely on.

    3. Traces

    Traces are the most important pillar for agent systems. A trace captures the full execution path of a single request from start to finish, including every step the agent took, in order, with timing and context at each step.

    End-to-end dynamic traces capture the full chain of thought: decisions, tool calls and interactions, data usage, and outcomes. They make AI agents observable at the ecosystem level, not just per prompt. 

    In practice, a trace is structured as a hierarchy of spans. A parent span represents the overall request. Child spans represent individual steps: an LLM call, a retrieval step, a tool call, a sub-agent handoff. Each span records its inputs, outputs, duration, and any errors. This parent-child relationship is what allows you to reconstruct the complete execution path and identify exactly where in a multi-step workflow something went wrong.

    4. Evaluations

    Evaluations assess the quality and correctness of agent outputs, not just whether the system ran without errors. An agent can complete every step of a workflow successfully and still produce a wrong or harmful answer. Evaluations catch this.

    In practice, evaluations can be automated or human-reviewed. Automated evaluations score outputs against criteria like factual accuracy, relevance, adherence to guidelines, and task completion. They run on every request in production, flagging outputs that fall below a threshold for human review. Over time, evaluation scores are tracked as a time series. When a prompt change, model update, or new tool version causes quality to drop, that shift shows up in the evaluation trends before users start complaining.

    5. Governance

    Governance is the layer that connects observability to accountability. It ensures agents operate within defined boundaries, comply with organizational policies, and produce an audit trail that satisfies regulators.

    The EU AI Act's high-risk AI obligations, in effect from August 2026, require lifecycle logging, traceability, and the ability to demonstrate a system behaved as intended. NIST AI RMF makes similar demands. Governance in observability means tagging traces with policy context, flagging outputs that violate guardrails, and maintaining records that answer not just what happened but whether it was appropriate.

    How to build observability into an agent system

    Making a system observable means designing it to record what it does as it runs. Every time the agent takes a meaningful action, such as searching a knowledge source, calling a tool, or reaching a decision point, the system needs to capture a record of that action: what triggered it, what happened, how long it took, and what the outcome was. That process of building in those recording points is called instrumentation.

    How you approach instrumentation depends on how you are building. Some agent frameworks have observability built in and emit this data automatically. Others require you to add recording points yourself, explicitly telling the system what to capture at each step. In either case, the goal is the same: a complete record of everything that happened during a request, from the moment it came in to the moment a response was returned.

    A few practical things to keep in mind when building this out:
    • Every request should have a unique identifier so you can pull up the full record for any specific interaction if something goes wrong. Without this, finding the trace for a particular user complaint is very difficult at scale.
    • Structure your records to show the sequence of what happened. The overall request is the parent. Each step within it, each tool call, each retrieval, each reasoning decision, is a child of that parent. This hierarchy is what lets you reconstruct the exact path the agent took.
    • Capture inputs and outputs at every step, not just the final response. The final response tells you the outcome. The intermediate records tell you why.
    • When a new model version, updated prompt, or changed tool causes the system to behave differently, you need to know what changed. Tagging each record with the versions of the components that were active at the time makes this possible.
    • Finally, observability infrastructure is not exempt from the data protection principles covered in Lesson 2. Logs and traces that contain raw personal data create the same risks as any other system that stores sensitive information. Apply masking and redaction here too.
    Key insight

    You cannot improve what you cannot see, and you cannot be accountable for what you have not recorded. Agent observability is not a feature you add before launch. It is infrastructure you design from the start. Teams that treat observability as a core capability, not an afterthought, achieve measurably better outcomes. 

    Up next: Now that we covered visibility into what agents are doing, the next lesson looks at how to make sure what they are doing is secure.

    Lesson 4: Security in Agent Systems

    Learning objectives

    By the end of this lesson you will be able to:

    • Understand why agent systems introduce a different security threat model than traditional software
    • Identify the core security risks specific to agentic systems and how they manifest in practice
    • Describe the design principles that make agent systems more secure from the start

    Earlier in this module, we covered two important aspects of maintaining safe and responsible agent systems: protecting sensitive data and building in observability. Both are foundational, but there is more to securing an agentic system than controlling what it knows and what you can see.

    The deeper challenge is what agents can do. Unlike earlier AI systems that generate outputs for a human to act on, agents can execute actions directly: sending emails, modifying records, calling external services, writing and running code, triggering multi-step workflows. That execution capability is what makes them powerful in enterprise environments. It is also what makes a security failure in an agentic system fundamentally different from one in a traditional application.

    The risks are real. For example, an AI coding agent deleted a company's entire production database in nine seconds, then produced a written explanation of exactly which safety constraints it had decided to override. No confirmation was requested. The agent reasoned its way to a decision and acted on it autonomously, causing thirty hours of downtime for every business depending on that platform. The problem was not in the data layer. It was in the absence of any boundary between what the agent could decide and what it could do without human approval.

    That is the risk category this lesson addresses.

    Why the threat model is different

    Traditional application security is designed around deterministic systems – those where no randomness is involved. You define what the software can do, restrict everything else, and verify that inputs and outputs conform to expected patterns. That model works when behavior is predictable and fixed.

    Agent systems are adaptive. They process natural language, respond to retrieved content, and make decisions based on context that changes with every request. This creates a different kind of attack surface. You cannot fully enumerate what an agent might do in advance, because its behavior depends on inputs you have not seen yet. An attacker who understands this can craft inputs specifically designed to push the agent toward unintended behavior, not by exploiting a code vulnerability, but by manipulating the agent's reasoning through the content it processes.

    Prompt injection is the clearest example of this. It is not a traditional code exploit. It is an attack that embeds malicious instructions in content the agent retrieves or receives as part of a normal workflow, effectively hijacking the agent's reasoning and redirecting its behavior. A document that contains hidden instructions telling the agent to forward data to an external address, execute a command, or skip a guardrail it would otherwise enforce: these are real attacks that have been executed against production systems.

    The core security risks

    Understanding the specific risk categories in agentic systems is the foundation of designing against them.

    Prompt injection is when malicious instructions are hidden inside content the agent retrieves or receives, such as a document, a user message, or a tool response, designed to override the agent's intended behavior. Instead of following its instructions, the agent follows the attacker's. It is the most actively exploited attack in production agentic systems today. The defense is architectural: treat everything the agent retrieves or receives from outside the system as potentially hostile, keep the agent's core instructions separate from user-provided content, and filter inputs before they reach the model.

    Every capability you give an agent that it does not strictly need for a task is a liability. If that agent is compromised, behaves unexpectedly, or is manipulated through an attack, those extra capabilities expand what an attacker can do with it. An agent that only needs to read documents should not also have the ability to send emails or modify records. Scope what each agent can do to exactly what the specific task requires, and nothing beyond that.

    Agents that remember information across sessions can be manipulated by corrupting what they remember. If an attacker can insert false or malicious information into an agent's memory, the agent will act on it in future interactions without knowing it has been tampered with. The effects can persist silently across many sessions. Keeping each session isolated from the next, and validating what gets written to persistent memory, are the primary defenses.

    Every external system an agent can interact with is a potential entry point for an attack. An agent connected to email, a database, a CRM, or an external API can cause real harm if those connections are not designed with proper controls. A poorly secured tool integration can allow an attacker to feed the agent bad instructions through tool responses, or allow the agent to take actions in external systems it should not have access to. Each tool integration should be reviewed with the same rigor as any external API connection.

    Some actions cannot be undone: a sent email cannot be unsent, a deleted record may not be recoverable, a triggered financial transaction may not be reversible. Agents that can take these kinds of actions need an explicit approval step before execution, a moment where a human confirms the action before it happens. Without that gate, a single error, misunderstanding, or successful attack can cause damage that cannot be fixed. This is not optional in production environments.

    How to build more secure agent systems

    The mitigations for these risks are not complicated. They require deliberate decisions made early in the design process.

    Scope permissions to the task, not the agent

     What an agent needs permission to do varies by step. Reading a document requires different access than updating a record or sending a message. Rather than giving an agent broad permissions that cover everything it might ever need, grant access step by step, only for what each specific action requires.

    Treat all inputs as potentially untrusted

    Content the agent retrieves from a knowledge base, receives from a user, or gets back from a tool should all be treated with caution, not assumed to be safe. Filtering for adversarial patterns before content reaches the model's reasoning layer is now standard practice in production agent security.

    Require explicit approval before irreversible actions

     Any action the agent cannot easily undo should pause and ask a human to confirm before it executes. This is the single highest-impact control for preventing errors from becoming permanent. Design it in from the beginning, not as an afterthought.

    Keep credentials out of prompts entirely

     API keys, authentication tokens, and passwords must never appear inside a prompt or anywhere in the agent's reasoning context. They belong in dedicated secure storage systems with controlled, logged access. If a credential ends up in a prompt, the model may reproduce it in an output, log it, or expose it through the agent's responses.

    Watch for behavior that does not look normal

     Monitoring for security in agent systems means tracking patterns, not just individual events. A sudden spike in tool calls, access to data the agent does not usually touch, or outputs that are significantly different from the norm can be early signals of an attack or a system behaving in an unintended way.

    Define what the agent is allowed to do, and enforce it

    Every agent should have a clear, bounded scope of operation. When a request falls outside that scope, the agent should escalate to a human rather than try to handle it independently. An agent that can expand its own boundaries when it seems useful is an agent that cannot be reliably controlled.

    Practice

    What it means

    Scope permissions to the task                      

    Grant access step by step, only for what each specific action requires. An agent that reads documents should not also have permission to send emails or modify records.

    Treat all inputs as untrusted                   

    Content retrieved from a knowledge base, received from a user, or returned by a tool should all be filtered before reaching the model's reasoning layer. Do not assume it is safe.

    Require approval for irreversible actions

    Any action the agent cannot easily undo should pause for human confirmation before it executes. Design this in from the start, not as an afterthought.

    Keep credentials out of prompts

    API keys, tokens, and passwords must never appear in a prompt or reasoning context. Store them in dedicated secure systems with controlled, audited access.

    Monitor for abnormal behavior

    Track patterns over time: unusual spikes in tool calls, unexpected data access, or outputs that differ significantly from normal can signal an attack in progress.

    Define and enforce operational boundaries

    Every agent should have a clear scope. When a request falls outside it, the agent should escalate to a human rather than improvise. Agents that expand their own boundaries cannot be reliably controlled.

    Key insight

    Security in agent systems is not a feature you add before launch. It is a set of architectural decisions you make from the start about what the agent can access, what it can do, when it needs approval, and what happens when something unexpected occurs. The agents that cause the most damage are not usually the ones that were attacked. They are the ones that were given too much autonomy without enough structure around it.

    Up next: The final lesson covers testing: why reliable agent systems require a fundamentally different approach to testing than traditional software, and how to build confidence in systems whose behavior is not fully deterministic.

    Lesson 5: Testing Agent Systems

    Learning objectives

    By the end of this lesson you will be able to:

    • Explain why testing agent systems requires a different approach than testing conventional software
    • Understand what teams evaluate when testing agentic workflows and why each dimension matters
    • Recognize testing as a continuous practice and understand how it connects to building trustworthy systems

    Everything covered in this module so far – responsible design, data protection, observability, security – comes together in the question of how you know a system is actually working the way you intend. For traditional software, the answer is testing: run the code, check the outputs, verify they match expectations. For agent systems, that approach is necessary but not sufficient, because agent systems do not always produce the same output twice.

    This is not a flaw. It is a fundamental property of how large language models work. Given the same input, a language model may produce multiple valid responses that differ in phrasing, structure, or reasoning path. That flexibility is part of what makes agents useful. It also means that testing them requires thinking differently about what you are measuring and what confidence actually means for a non-deterministic system.

    Why traditional testing approaches fall short

    Traditional software testing is built on determinism. You write a test that sends a specific input and asserts a specific output. If the output matches, the test passes. This works because the software is deterministic: the same input always produces the same output. Agent systems break this model in several ways.

    Boxes in grid

    First, outputs vary. Two runs with identical inputs may produce responses that are both correct but worded differently, structured differently, or arrived at through different reasoning paths. Exact output matching fails immediately.

    Second, failures can be invisible at the component level. An individual retrieval step, tool call, or reasoning step can each work correctly in isolation while the composed system behaves unexpectedly. Some failures only appear when the full workflow runs end to end, when one agent's output becomes another agent's input, when memory from a previous session influences the current one, or when a sequence of individually reasonable decisions leads to a collectively wrong outcome.

    Third, the environment changes. Agents run against live data, real APIs, and dynamic knowledge sources. A workflow that behaves correctly today may behave differently next week when underlying data changes, a tool returns a different response format, or a model update shifts how the reasoning layer interprets inputs.

    This means testing agent systems requires evaluating the consistency and reliability of behavior across a range of conditions, not verifying exact outputs. The question is not whether the agent produced this exact response. It is whether the agent is behaving in a trustworthy, consistent, and safe way across the situations it will encounter.

    What teams evaluate

    Effective testing for agent systems focuses on the behavior of the full workflow, not just isolated components. There are six dimensions that matter.

    1. Workflow completion: Does the system complete the intended task reliably across a range of inputs, including inputs at the edges of what it was designed for? Does it fail gracefully and communicate that failure clearly when it cannot complete a task, rather than producing a plausible-looking but wrong result?

    2. Retrieval accuracy: Is the agent retrieving the right information for each request? Is that information relevant, complete, and correctly used in generating the response? Retrieval failures are one of the leading causes of incorrect agent outputs, and they are often invisible in the final response without dedicated evaluation at this layer.

    6 boxes stacked

    3. Output consistency: When an agent runs multiple times with equivalent inputs, do its outputs fall within an acceptable range of quality and correctness? Significant variation across runs is a signal that the workflow is fragile, and that small changes to inputs or context can produce unpredictably different outcomes.

    4. Factual accuracy and groundedness: Are the agent's outputs grounded in the information it retrieved and the tools it used, rather than in hallucinated or invented content? Detecting when an agent is producing responses that are not supported by evidence is one of the most important things testing can catch, and one of the hardest to evaluate at scale.

    5. Safety and boundary compliance: Does the system stay within its defined operational scope? Does it handle sensitive data appropriately? Does it escalate to a human when it should, rather than proceeding autonomously into territory it was not designed for?

    6. Behavior under adversarial conditions: How does the system respond to ambiguous, malformed, or deliberately adversarial inputs? Testing what happens at the edges and in failure modes, not just in the clean cases, is what distinguishes systems that are ready for production from systems that have only been tested in controlled conditions.

    Testing as a continuous practice

    One of the most important shifts in thinking about agent testing is moving from treating it as a pre-deployment gate to treating it as an ongoing practice. Agent systems are not static. Prompts change. Models are updated. Knowledge sources evolve. Tool interfaces shift. Any of these changes can affect system behavior in ways that are not obvious until the system is running in production.

    This means the testing infrastructure needs to run continuously alongside the system, not just before it goes live. Evaluation pipelines that capture real usage, measure quality across the dimensions above, and surface regressions before users notice them are as much a part of production infrastructure as the agent system itself.

    Testing is also the mechanism by which teams build justified confidence in their systems. Not confidence that the system will never produce an unexpected output, which is not achievable for non-deterministic systems, but confidence that the system behaves reliably enough, safely enough, and consistently enough to be trusted in the context where it is deployed. That confidence has to be earned through evidence, not assumed.

    In later courses, you will build and test multi-agent systems using Neuro San, Cognizant AI Lab's open-source framework. Neuro San includes a testing framework specifically designed for the properties covered in this lesson: measuring consistency across runs, evaluating whether agent outputs are grounded in their source material, and enabling agents to test other agents through controlled multi-turn interactions. The concepts introduced here are the foundation for that work.

    Key insight

    A system that has not been tested across a meaningful range of conditions is a system whose behavior in production is unknown. Testing agent systems is harder than testing conventional software, but it is not optional. It is how you move from an agent that works in a demo to an agent that can be trusted in production, at scale, with real users and real consequences.

    Up next: Before finishing the course with the certification exam, test your understanding with a quick quiz.

    Module wrap-up

    You have now completed the last module of the Agent Foundations course!

    The concepts you learned in this module are not abstract principles – they are the design decisions that determine whether an agent system you build is trustworthy enough to run in production, at scale, and in real workflows. At this point, you should be able to:

    • Define responsible AI and explain why its core principles apply specifically to agent systems
    • Describe the data protection risks in agentic workflows and the practices used to address them
    • Explain what observability means in the context of agents and why it is essential for debugging and accountability
    • Identify the key security threats in agentic systems, including prompt injection and excessive permissions, and describe the design practices that address them
    • Explain why agent testing requires a different approach from traditional software testing and describe what reliable testing looks like in practice

    Put it to the test

    Take the short quiz below to check your understanding before moving on to your certification exam.

    Want to see agents in action?

    Everything you just learned comes to life in Neuro San, Cognizant AI Lab's open source framework for building and deploying multi-agent systems. If you want to explore how real agent networks are structured before diving into the next module, the repo is a great place to start.

    Star Neuro San on GitHub

    Complete your certification with an exam:

    You are now ready to get certified in Agent Foundations. Please take the exam to receive your certification. 

    Complete the certification exam

    Pages

    Overview

    Overview

    1. Responsible AI Definition

    1. Responsible AI Definition

    2. Securing Sensitive Data

    2. Securing Sensitive Data

    3. Observability in Agent Systems

    3. Observability in Agent Systems

    4. Security in Agent Systems

    4. Security in Agent Systems

    5. Testing Agent Systems

    5. Testing Agent Systems

    Check your understanding

    Check your understanding
    Previous
    Next

    Congratulations! You've Completed Agent Academy!

    Get certified in Agent Foundations

    Take the exam