Continuous evaluation of AI agents: a practical method

Reliable AI agents are measured by the outcomes they produce, not by the fluency of their responses. This distinction seems simple, but it changes how automations are designed. An agent can write a convincing response, call tools without technical errors, and still cancel the wrong order, classify a lead inaccurately, or create a ticket in the wrong system.

The pressure to put agents into production has made this issue more visible. In LangChain’s State of Agent Engineering 2026 report, 57% of respondents said they already have agents in production. Yet while 89% report some level of observability, only 52.4% run offline evaluations with test sets. The gap is critical: logging events is not the same as proving quality.

Continuous evaluation is the system that connects business intent, agent behavior, and operational outcomes. It should run before, during, and after every meaningful change: model, prompt, tool, policy, integration, knowledge base, or routing rule.

What should be evaluated in an AI agent

The direct answer is: evaluate the task, the trajectory, and the effect on the process. An isolated final response is rarely enough.

An agent operates through a sequence. It interprets a request, selects context, decides whether to use a tool, executes actions, handles partial returns, and closes or transfers the case. Each step can introduce failures. Therefore, the unit of analysis is not just the generated text. It is the complete execution.

There are three useful layers:

• Outcome quality: was the objective achieved? For a support agent, was the request resolved correctly? For a sales agent, was the lead qualified using the right criteria?
• Trajectory quality: did the agent use the right sources, tools, and permissions? A trajectory is the ordered record of decisions, tool calls, messages, and intermediate states.
• Operational quality: did the agent comply with cost, latency, security, and retry limits?

The study Monitoring Agentic Systems Before They're Reliable, published in June 2026, proposes observing agentic systems through quality, adequacy, and efficiency across three scopes: within an execution, across executions, and in the system structure. The practical implication is straightforward: an API error dashboard alone cannot detect inappropriate decisions that were technically executed correctly.

Define success with observable criteria. “Provide good service” is an ambition. “Correctly classify intent, record the request in the CRM, and route exceptions to a human within two minutes” is an evaluable specification.

How to turn business objectives into test cases

The direct answer is: build an evaluation set from real decisions and relevant exceptions, not just ideal examples.

An evaluation set is a versioned collection of cases representing situations the agent must solve. Each case should include the input, available context, allowed tools, approval criteria, and, when applicable, the expected state after execution.

Start with five groups:

1. Frequent cases: represent the main volume and protect process efficiency.
2. Critical cases: occur infrequently but have high financial, legal, or reputational impact.
3. Ambiguous cases: require confirmation, additional context retrieval, or human routing.
4. Adversarial cases: include conflicting instructions, incomplete data, and attempts to induce improper tool use.
5. Production failures: every understood incident should become a permanent regression case.

This last group is the most valuable. It turns an isolated failure into operational memory. If an agent applied an outdated sales policy, the case should not disappear after the fix. It becomes part of the suite that will run before future changes.

Avoid tests that accept any “reasonable” text. In transactional tasks, the criterion must reach the final state. If the agent claims it updated a record, validate that the field changed in the correct system. If it says it sent a proposal, validate the recipient, the document, and the applied commercial rule.

The research AlphaEval: Evaluating Agents in Production, from April 2026, reinforces this distinction by evaluating complete agentic products, not just models. Perceived performance depends on the combination of model, instructions, tools, interface, permissions, and environment.

Which metrics reveal whether the agent is improving

The direct answer is: combine business, quality, risk, and efficiency metrics. A single accuracy rate creates blind spots.

The first metric should reflect the purpose of the workflow. It may be first-contact resolution, qualified conversion rate, handling time, revenue recovery, rework reduction, or document compliance. This is the outcome metric.

Then, track quality metrics:

• correct completion rate;
• appropriate human routing rate;
• adherence to policies and business rules;
• correct use of sources and tools;
• recurrence of known failures.

Also include risk metrics. Measure actions blocked by guardrails, out-of-scope access attempts, reverted changes, cases involving sensitive data, and decisions that required later review. A guardrail is a technical or process rule that limits dangerous actions, such as approving discounts beyond authorization levels or accessing data outside an authorized purpose.

Finally, track efficiency: end-to-end latency, cost per completed task, number of tool calls, number of retries, and abandonment rate. An agent that resolves more cases but triples its cost and latency can degrade the overall process.

Do not set a universal target of “95% accuracy.” Tolerance should vary based on the action. An agent that summarizes meetings can accept a more subjective evaluation. An agent that updates records, releases payments, or advises customers on contracts requires stricter criteria, state confirmation, and autonomy limits.

How to evaluate subjective responses without blindly trusting another model

The direct answer is: use automated evaluators for scale, but calibrate them with recurring human review.

Not every task has a single answer. Tone, clarity, completeness, usefulness, and contextual appropriateness are subjective dimensions. In these cases, an LLM as a judge can compare the agent’s output against a structured rubric. The rubric must state exactly what will be assessed, which evidence matters, and which conditions cause the response to fail.

But the evaluator also makes mistakes. It may favor long answers, confuse confidence with accuracy, or reproduce biases from the model itself. Therefore, automated judgment should not be treated as autonomous truth.

A robust practice has four controls:

1. Use task-specific criteria rather than requesting a generic quality assessment.
2. Separate content, safety, and execution evaluation. One judge for everything reduces diagnostic clarity.
3. Conduct periodic human sampling to compare automated verdicts with expert judgment.
4. Measure disagreement. If human evaluators disagree with one another, the quality rule is still too vague.

Anthropic reported in an April 2026 postmortem that quality issues in agentic products were initially difficult to distinguish from normal variation in user feedback and were not immediately reproduced by internal evaluations. The lesson is important: evaluations must evolve when production reveals signals that the lab did not capture.

How to monitor agents without turning observability into noise

The direct answer is: monitor by exception, distribution shifts, and business impact.

Observability is the ability to reconstruct what a system did and why it reached a given outcome. For agents, this requires tracking the model version, prompt, retrieved context, tool calls, permissions, intermediate outputs, final outcome, and subsequent feedback.

The most common mistake is storing every trace without defining priorities. A mature operation needs alerts tied to failure hypotheses. Examples include:

• sudden increase in transfers to human support;
• growth in responses without sources on topics that require evidence;
• use of a tool outside its historical pattern;
• decline in completion rate after a model change;
• increase in retries per execution;
• divergence between what the agent claims and the state confirmed in the system.

Also monitor drift, or behavioral deviation over time. It can emerge from changes in the request mix, knowledge base updates, a new model version, an altered integration, or degradation in an external API. The agent may continue to “work” while still delivering worse outcomes.

Observability should enable comparison across versions. Without versioning the prompt, policy, tool, and data set, the team cannot attribute a regression to a likely cause. The goal is not to accumulate logs. It is to reduce the time between detecting an anomaly, understanding its origin, and applying a safe fix.

What is the operating cycle for correcting failures without creating new regressions

The direct answer is: treat every change as a testable hypothesis and move forward through progressive release.

The cycle begins with a baseline. Before optimizing, record current performance by task type, user segment, tool used, cost, latency, and human intervention rate. Without a baseline, improvements are only impressions.

Then, follow a disciplined cadence:

1. Classify the failure

Determine whether the issue involved understanding, context retrieval, policy, tool, permission, integration, or experience. “The agent made a mistake” is not a useful diagnosis.

2. Create or update the evaluation case

Reproduce the failure with safe data. Define the correct outcome and the signals that prove approval. The case then protects the operation against recurrence.

3. Change one variable at a time when possible

Changing the prompt, model, tool, and rule simultaneously makes causal attribution difficult. In critical workflows, favor controlled experiments.

4. Run offline evaluations and integration tests

Offline evaluation verifies behavior on known cases. Integration testing confirms that tools, permissions, and external states work in a realistic environment.

5. Release gradually

Use a limited share of traffic, a reduced action scope, or expanded human oversight. Autonomy should grow alongside performance evidence, not based on stated confidence.

6. Reassess the effect on the process

A local improvement can make the operation worse. Reducing human transfers, for example, is harmful if the agent begins resolving fewer cases correctly.

This method brings agents closer to a discipline already familiar in critical systems: quality is a continuously verified property. For operations that orchestrate multiple steps, platforms such as Centriu Flow can centralize rules, approvals, and control points, but governance still depends on clear outcome criteria and an active evaluation cycle.

Where to start in the next 30 days

The direct answer is: choose a bounded workflow, define an outcome metric, and build a small but representative suite.

In the first week, map the decision the agent makes and the expected effect on the process. Identify where an incorrect decision generates the highest cost. In the second week, gather 30 to 50 real cases, including errors, exceptions, and ambiguous requests. In the third week, implement automated criteria for outcomes, policy, cost, and latency. Reserve human review for subjective and high-impact cases.

In the fourth week, run the suite with every change and establish a routine for analyzing production failures. Do not try to measure everything at once. The initial gain comes from making explicit what was previously implicit: what decision the agent can make, under what conditions, and how the company knows it was correct.

Reliable agents are not those that appear most autonomous. They are those that accumulate evidence that they operate well, recognize their limits, and improve without repeating the same mistakes.