Why this matters
Before a billing team relies on an appeal-letter agent, someone has to try to make it leak. Typing a test into the Playground after every change is slow and easy to skip. An Airia Evaluation saves the test messages, runs them all in one click and has a judge model score each answer.
In this module you build that test with five cases, two of them seeded with a made-up name, date of birth and MRN. Our real run is the lesson: it scored 97.78% while two letters printed the identifiers, and the judge gave both 100%. You read past the score, fix the agent and run it again.
This lesson is training, not legal advice. A guardrail does not make a tool HIPAA compliant on its own: your organization also needs a Business Associate Agreement with any vendor that handles PHI, its own policies, access controls and staff training. Ask your compliance officer before any real patient data goes near an AI tool. Every patient in this lesson is made up (Pat Example, MRN 000-TEST).
What you’ll build
An Airia Evaluation, Appeal letter leak check, with the appeal-letter agent, five test cases and the Airia default evaluator with a small judge model.
Two runs you read row by row: one that leaked and scored 97.78%, and one with placeholders in every letter that scored 80%.
What you need
- Nothing to start the practiceThe practice is built from real Airia screens and touches no account.
- The appeal-letter agent from module 1, to do it for realBuild it in module 1 first, or test any agent you already have. Check airia.com for plans and limits.
- Made-up test values you can search forNames like Pat Example, MRNs starting 000-TEST, phones in the 555-01xx range.
Concepts
2 ideas you’ll use in the practice.
Leak cases
A leak case is a test message that contains identifiers, with an Expected Output that names the exact values that must never appear. Put one at the start of a message and one buried at the end, where a chart note usually is.
Include ordinary cases too, and one with missing facts, so you also see whether the agent invents treatments.
The score is not the test
The judge model rates how good a letter looks against your expected answer. In our first run it gave 100% to a letter that printed the name, date of birth and MRN, and even noted the privacy problem in its own summary.
After the fix, the score went down to 80% because the judge gave a correct letter 0% (it said it had no answer to grade). Scores move both ways. Read Actual Output for every leak case, every run.
Practice
Do every step yourself in a practice copy of the real tool. It checks your work, and “Do it for me” finishes any step you get stuck on.
Do it for real
Now repeat it in your own account. Tick each step as you go.
Quick check
3 questions to check you’ve got it.
1. The evaluation shows 97.78% accuracy. What do you do?
- Read Actual Output for every leak case
- Roll it out
- Lower the expectations
Show the answer
Read Actual Output for every leak case. In our run two of those letters printed the made-up name and MRN and still scored 100%.
2. What makes a good leak case?
- Made-up identifiers in the input and the exact values listed as must-not-appear
- A question with no identifiers
- A real patient’s note
Show the answer
Made-up identifiers in the input and the exact values listed as must-not-appear. Searchable made-up values show a leak at a glance, and no real patient is involved.
3. After the fix the score dropped to 80%. Is the agent worse?
- No: no letter leaked; the judge mis-scored one correct letter
- Yes, roll back
- The test is broken, delete it
Show the answer
No: no letter leaked; the judge mis-scored one correct letter. The letters are the test. The judge’s number is a hint that can be wrong in either direction.
You reached the end of AI for Healthcare Teams with Airia: Protect Patient Data
Agents built and tested? Next, the rest of the front office: scheduling, insurance questions and patient messages with everyday AI tools.
Next course: AI for Medical & Dental Office Admin →