Grade support agent conversations against your refund policy
Replay support agent traces and have Jev answer a few policy questions per trace, logged as metrics you can compare across versions.
The use case
Teams shipping an AI support agent need to know whether a new prompt or model still follows policy, for example on refunds. This worked example replays saved traces, each holding the customer message, the policy text, every tool step and the reply. Plain code checks the easy facts first, then Jev answers a few rubric questions in one request, such as whether the refund matched policy and whether the agent asked for the order number. Each answer becomes a separate metric in the evaluation tool. A person reviews failing traces before a release goes out.
Example workflow
- Save 10-100 representative support traces with the policy text
- Check deterministic facts such as tool call order in code
- Ask Jev three to five rubric questions per trace in one call
- Log each answer as a metric in Opik
- A person reviews failures before approving a release
Tools
- Jev (TypeSafe AI)
- Opik
- Python
Risks and guardrails
- A small trace set misses rare failures
- Rubric wording changes shift scores between runs
- Grading the grader is needed before trusting the metric
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- jev-as-judge by Akshay Pachaar on Made with Jev: madewithjev.com/builds/jev-as-judge
- jev-as-judge: original post or repo by Akshay Pachaar: github.com/patchy631/jev-as-judge