AI Agent Evaluation: Test Ecommerce Work Before Expanding Authority
A useful agent evaluation gives an AI employee real workflow cases, checks the outcome and actions it took, and records the failures that matter to your business.
Each case has an expected outcome, required evidence, and an action boundary.
An AI support agent handles ten test tickets and writes ten polished replies. That result tells you little about whether it can do the job. It may have used the wrong order, missed a return-policy exception, or prepared a refund when the case needed a replacement.
Evaluate the workflow rather than the writing. Give the agent representative cases, define the correct business outcome, and inspect the sources and tool actions behind each result. The test should show which work can run, which work needs review, and where the agent fails.
Measure the finished job
A model benchmark measures general capability. Your evaluation needs to measure the work you plan to delegate. OpenAI recommends grading the full agent trace, including tool calls, handoffs, and guardrails, so teams can see whether a workflow reached the right outcome for the right reasons (agent evaluation guide).
Start by writing the completion condition. A support agent resolving a damaged order may need to identify the correct order, verify delivery and policy details, choose a remedy, and prepare a reply. If the order falls outside the policy or the records conflict, successful completion means escalation with the evidence attached.
Keep outcome and process separate. An agent can reach a reasonable answer after reading the wrong customer record. It can also follow the right process and stop because a source is missing. The first run should fail. The second may pass if your policy says missing evidence requires a person.
Build a small evaluation set from real work
Use cases that resemble the queue the agent will receive. Write the expected result and checks before you run the test.
A DTC support evaluation
Consider a skincare store testing an agent on damaged-order requests. The brand offers a replacement when a customer reports damage within 30 days and supplies adequate evidence. Refunds above $50 need approval. Orders with conflicting delivery records go to a support lead.
Build an evaluation set with recent, de-identified cases. One customer sends a clear photo of a broken pump four days after delivery. Another writes 45 days after delivery. A third case has a carrier scan marked delivered while the store record says the shipment remains in transit. A fourth customer omits the order number.
The first case should produce a replacement recommendation and a drafted reply. The second should explain the policy and escalate any exception request. The third should stop and show both shipping records. The fourth should ask for the minimum information needed to find the order without inventing an identity or matching by name alone.
Run each case several times if the workflow contains judgment or model-based routing. Record whether the outcome changes. A case that passes once and fails twice is not ready for unattended work, even if the successful transcript looks excellent.
Use a scorecard tied to business risk
Score task completion first. Did the agent choose the policy-compliant remedy or escalation path? Then check factual grounding: did it use the correct order, policy version, shipment record, and customer message? These checks can often use exact values and tool-call logs.
Add action-boundary checks. The agent should not issue a refund, send a reply, or change an order during an offline test. In a live approval workflow, confirm that it pauses at the expected threshold and sends the case to the named owner. A strong written answer does not cancel an unauthorized action.
Use human judgment where rules cannot settle the result. A support lead can grade whether the response explains uncertainty without confusing the customer. Anthropic recommends deterministic graders where possible, model graders where rules need flexibility, and human review for validation (agent eval design guide).
Report failures by type rather than hiding them inside one average score. A wrong tone and a wrong refund amount do not carry the same cost. Track factual errors, policy errors, missed escalations, unauthorized actions, and unclear customer communication separately.
Keep the evaluation set current
The first set reflects what your team already knows. Production work will find cases you missed. When an agent uses an expired promotion, confuses two orders, or escalates a common request that should be routine, remove customer details and add that pattern to the evaluation set.
Run the same set after changing a prompt, tool connection, policy, or model. A change that fixes one case can break another. Repeatable cases let you compare versions before customers experience the regression.
Expand authority by action. An agent may earn permission to send shipping updates while refunds remain behind approval. Evaluation evidence should support that specific boundary rather than a broad claim that the agent is ready.
Where ShopDucky fits
ShopDucky gives ecommerce teams AI employees that work across tools such as Shopify, support platforms, Meta Ads, and Slack. Teams can review the source records and proposed actions behind a workflow, keep costly changes behind approval, and expand authority as real evaluation and production evidence improves.
AI agent evaluation, answered
What is an AI agent evaluation?+
An AI agent evaluation is a repeatable test that gives an agent a defined task, records the result and actions it took, then scores them against written criteria. For ecommerce work, the criteria should cover the business outcome, source use, policy compliance, and escalation behavior.
How many cases should an evaluation set contain?+
Start with enough cases to represent the normal workflow and its important failure modes. A small set of carefully chosen real cases is more useful than a large set of generic prompts. Add cases when production work reveals a new edge condition.
Should a person grade every agent run?+
No. Use deterministic checks for facts such as order IDs, refund amounts, tool calls, and whether a restricted action occurred. Use a clear rubric or human review for judgment calls such as tone, ambiguity, and whether an escalation explanation is adequate.
When can an ecommerce agent receive more authority?+
Expand one action at a time after the agent passes the relevant evaluation set and performs well on reviewed production cases. Keep expensive, customer-visible, or hard-to-reverse actions behind approval until the evidence supports a specific change.
