An AI agent can produce a convincing answer in a demonstration and still fail when it is connected to a real business process.

It may select the wrong tool, act on incomplete data, repeat an operation after a timeout, fail to escalate, or perform well on normal requests but poorly on unusual ones.

That is why agent evaluation should test the complete workflow, not only the language model’s output.

This guide provides a practical structure for a pre-production test plan. The exact tests and thresholds should be selected for the system, the consequences of failure, and the business outcome.

Start with the workflow and expected outcome

Before writing test cases, document the trigger, input data, expected steps, tools and systems, output or action, approval boundary, failure path, and business owner.

Define what successful completion means. For a customer support workflow, success might include correct classification, use of an approved knowledge source, a suitable response draft, correct escalation when required, and a complete audit record.

Do not define success as “the response sounds good.” A polished answer can still be wrong, unsupported, or operationally incomplete.

Build a representative test set

Use realistic examples from the intended workflow. Remove or anonymize sensitive information and make sure the test data can be used appropriately.

Include common requests, less frequent but important requests, ambiguous requests, requests with missing information, conflicting data, out-of-scope requests, requests requiring human review, tool/API errors, and adversarial or misleading inputs.

Do not use only easy examples. The purpose is to understand where the system fails and whether those failures are acceptable.

Evaluation dimension 1: Task success

Can the agent reach the intended business outcome? Define a pass condition for each test. A task may require several conditions to pass, such as correct classification, correct tool action, and the required audit record.

Record partial completion separately from full completion. If a human must finish the task, include that explicitly rather than treating it as fully automated.

Evaluation dimension 2: Factual and source correctness

If the agent answers a question, test whether the answer is supported by approved sources. Include cases where information is missing, stale, contradictory, or outside scope.

A useful test asks the system to say it cannot determine the answer when evidence is insufficient. The agent should not invent details to fill a gap.

For retrieval-based workflows, check whether the correct source was retrieved and whether the response accurately reflects it. For action-based workflows, check the data used to justify the action.

Evaluation dimension 3: Tool selection and execution

A tool-enabled agent can fail even when its explanation sounds correct. Test correct tool selection, arguments, input schema, handling of tool errors, absence of unauthorized calls, prevention of duplicate actions, interpretation of tool responses, and safe behaviour when a tool is unavailable.

Use test doubles or sandbox systems where appropriate. Do not run destructive test actions against production data.

Evaluation dimension 4: Permissions and security

Test whether the agent can access data outside its scope, refuses unauthorized actions, stops when approval is rejected, resists attempts to redirect the workflow through untrusted content, protects credentials, records tool calls, and allows access to be revoked.

Security testing should be planned with the responsible technical and security owners. Do not use live systems or third-party targets without authorization.

Evaluation dimension 5: Ambiguity and missing data

Real requests are not always clean. A customer may omit an identifier, two systems may disagree, or a record may be outdated.

The expected behaviour might be to ask a clarifying question, return a limited answer, or route the case to a person. Define the acceptable path in advance. The agent should not silently choose an arbitrary interpretation when consequences are material.

Evaluation dimension 6: Human escalation

Test whether the agent recognizes escalation triggers, provides relevant context, identifies uncertainty, preserves task state, stops taking actions while waiting, routes the case correctly, and records the final decision.

A system that knows when to ask for help can be more dependable than one that tries to complete every task.

Evaluation dimension 7: Reliability and recovery

Test timeouts, network errors, rate limits, invalid responses, and partial failures. For actions that could be duplicated, test idempotency or another appropriate duplicate-prevention mechanism. Confirm whether the workflow can resume safely after a failure.

Also test how operators pause the system, disable a tool, revoke access, and restore the workflow. Recovery is part of production readiness.

Evaluation dimension 8: Latency, cost, and human rework

Measure the operational cost of completing the task, not only the cost of a model call. Track end-to-end completion time, model and tool usage cost, retry frequency, human review time, human correction or rework, escalation rate, failure severity, and successful completion rate.

Compare against a documented baseline. If the goal is to reduce manual work, include the human work that remains after automation. Avoid presenting an ROI claim without explaining assumptions and measurement period.

Evaluation dimension 9: Consistency and regression

Repeat important tests across multiple runs where behaviour can vary. Keep a regression set so changes to prompts, models, tools, retrieval, or data do not silently break previously working behaviour.

Version the test set and record the system configuration used. When an evaluation result changes, the team should be able to identify what changed and decide whether the difference matters.

Build a practical scorecard

For each test case, record test ID, scenario, risk category, expected result, observed result, pass/fail, severity, evidence or trace ID, remediation owner, and release-blocking status.

Avoid hiding serious failures inside a single average score. A high overall score should not compensate for a critical unauthorized action.

Set release gates

Before evaluation begins, agree on release conditions. Examples include no unresolved critical permission failures; required tool actions pass agreed tests; high-impact actions follow the approval path; failure and escalation handling are tested; monitoring and ownership are in place; known limitations are documented; and the business owner accepts the residual risk.

The exact threshold should depend on the use case. A low-risk summarization tool and an agent that changes customer records should not have the same release requirements.

After launch: keep evaluating

Evaluation should continue after release. Monitor real-world failures, user corrections, tool errors, latency, cost, and changes in the underlying workflow.

Re-test when the model changes, prompt changes materially, a new tool is added, permissions change, data sources change, business rules change, or a serious incident occurs. Use incident findings to improve the test set. The objective is a continuous feedback loop, not a one-time certification.

Conclusion

A strong AI agent evaluation plan answers three questions: does the workflow achieve the intended outcome; does it stay within the agreed boundaries; and can the organisation detect, understand, and recover from failures?

Testing should reflect the real task, real tools, and consequences of error. It should also include the human work that remains after automation.

If your team is moving an AI agent from prototype to production, start by documenting the workflow and building a test set before expanding permissions.

Wority Technology can help structure that evaluation and connect it to business outcomes.

Visit us