Why this matters
An HR agent that answers 60 people is only useful if its answers are the handbook answers. One wrong benefits or vacation answer, delivered politely, creates a problem HR has to clean up, and people stop trusting the tool.
Testing by hand in the Playground after every change is slow, so it gets skipped. An Airia Evaluation saves your test questions and the answers you expect, runs them against the agent in one click, and has a judge model score each answer. In this module you build one for the HR Policy Agent and learn to read past the score.
What you’ll build
An Evaluation in Airia with your agent, two test cases (one the handbook answers, one it must hand to HR) and the Airia default evaluator with a small judge model.
A completed run you read row by row: Expected Output next to Actual Output, with the accuracy score, latency and cost per question.
What you need
- Nothing to start the practiceThe practice is built from real Airia screens and touches no account.
- An Airia account and an agent to test, to do it for realBuild the HR Policy Agent in module 3 first, or test any agent you already have. Check airia.com for plans and limits.
- Five questions and the right answersWrite them as HR (or the process owner) would answer, and include at least one the agent must refuse or hand to a person.
Concepts
2 ideas you’ll use in the practice.
Test cases are your definition of "right"
Each test case is an Input (the question) and an Expected Output (the answer you want). Together they are the dataset. You can type them or upload a CSV from Airia's template.
The most important case is the one the agent should not answer. An agent that guesses about pay or benefits is worse than no agent.
A score is a hint, not a pass
The judge model compares each actual answer with the expected one and gives accuracy and bias scores. It can still rate a confident general answer highly: in our run the vacation answer never said 15 days and scored 84%.
Read Actual Output for every case. Fix the agent (for example, check the handbook data source is connected), then use Run Again and Compare Results.
Practice
Do every step yourself in a practice copy of the real tool. It checks your work, and “Do it for me” finishes any step you get stuck on.
Do it for real
Now repeat it in your own account. Tick each step as you go.
Quick check
3 questions to check you’ve got it.
1. What should every evaluation dataset include?
- A question the agent must refuse or hand to a person
- Only easy questions
- The agent's instructions
Show the answer
A question the agent must refuse or hand to a person. The refusal case is where a guessing agent does the most harm.
2. The evaluation shows 92% accuracy. What next?
- Read each Actual Output against the Expected Output
- Publish to everyone
- Delete the test
Show the answer
Read each Actual Output against the Expected Output. A judge model can score a polished but wrong answer highly; the rows tell you what staff would really read.
3. Why use a small judge model?
- It keeps the cost of each run low
- It is always more accurate
- Airia requires it
Show the answer
It keeps the cost of each run low. Our two-question run cost less than a cent with GPT-5 Nano as the judge.