What is LLM as a judge?
Quick answer
LLM as a judge is a testing method where an AI model grades answers, often from another model or prompt, against criteria you write down: did it use only the given facts, stay under the word limit, avoid promises? It lets you check dozens of outputs quickly instead of reading each one yourself.
Last updated
Take the free module →Module 6 of the free Better Prompts course, about 7 minutes. No account needed.
Updated · By Robert Breen
Why it matters for a small business
Once a prompt runs in an automation or a chatbot, you need to know whether it keeps working as inputs change. Reading every output doesn't scale. A judge prompt with a clear checklist can flag the ones that fail so a person reviews those.
Judges have known weaknesses: they can favor longer answers, answers in their own style, or the first option shown. Keep the criteria concrete and yes or no, give the judge the source facts, and spot-check its grades against your own.
In a real lesson: Catch Made-up Facts Before They Reach Customers using ChatGPT
In Catch Made-up Facts Before They Reach Customers, you finish with a six-step fix-it checklist saved to a Prompt Templates Google Doc: any fee, price, date or promise you didn't give it, any claim that says nothing, any safety, law, health or money fact you haven't checked.
The lesson has a person run that checklist. Turning it into a judge is the next step for volume: paste Dana's real facts, the checklist and a draft into a fresh chat and ask for pass or fail on each line with a quote. You still open the source for any fact the judge flags.
Try this lesson free or read the step-by-step guide.
Common confusions
LLM as a judge vs self-verification
Self-verification asks the same model to check its own answer in the same chat. A judge is a separate grading step, ideally with fresh context.
LLM as a judge vs human review
The judge filters at scale. A person still decides on anything that goes to a customer and checks a sample of the judge's grades.
Tips
- Write criteria as yes or no questions with the source facts attached.
- Ask the judge to quote the line that fails each check.
- Compare its grades with your own on ten examples before trusting it.
Related terms
More AI basics terms
Where you use it: free lessons
- Catch Made-up Facts Before They Reach Customers using ChatGPT (ChatGPT, 7 min)
- Give Examples: Few-Shot Prompts That Copy Your Style (ChatGPT, 6 min)
Frequently asked questions
- Can AI grade AI reliably?
- Reasonably well on clear, factual criteria, less well on taste or subtle accuracy. Spot-check its grades.
- Should the judge be a different model?
- Often it helps to use a different or stronger model and a fresh chat, so it doesn't share the first answer's blind spots.
- What should a judge prompt include?
- The source facts, the answer to grade, a short list of yes or no criteria, and a request to quote the line behind each failure.