# What is LLM as a judge?

> LLM as a judge means using one AI model to grade another's answers against a written checklist. It speeds up testing but still needs human spot checks.

Source: https://learn.ynteractive.com/content/glossary/llm-as-a-judge · Updated 2026-10-20 · Free from Stepthrough (https://learn.ynteractive.com)

[AI basics](https://learn.ynteractive.com/content/glossary#topic-ai-basics) · [AI glossary](https://learn.ynteractive.com/content/glossary)

Quick answer

LLM as a judge is a testing method where an AI model grades answers, often from another model or prompt, against criteria you write down: did it use only the given facts, stay under the word limit, avoid promises? It lets you check dozens of outputs quickly instead of reading each one yourself.

Last updated October 20, 2026

[Take the free module →](https://learn.ynteractive.com/courses/better-prompts/6)Module 6 of the free [Better Prompts](https://learn.ynteractive.com/courses/better-prompts) course, about 7 minutes. No account needed.

Updated October 20, 2026 · By [Robert Breen](https://learn.ynteractive.com/content/about)

## Why it matters for a small business

Once a prompt runs in an automation or a chatbot, you need to know whether it keeps working as inputs change. Reading every output doesn't scale. A judge prompt with a clear checklist can flag the ones that fail so a person reviews those.

Judges have known weaknesses: they can favor longer answers, answers in their own style, or the first option shown. Keep the criteria concrete and yes or no, give the judge the source facts, and spot-check its grades against your own.

## In a real lesson: Catch Made-up Facts Before They Reach Customers using ChatGPT

In [Catch Made-up Facts Before They Reach Customers](https://learn.ynteractive.com/content/how-to-fix-bad-chatgpt-answers-push-back-constraints-fact-check), you finish with a six-step fix-it checklist saved to a Prompt Templates Google Doc: any fee, price, date or promise you didn't give it, any claim that says nothing, any safety, law, health or money fact you haven't checked.

The lesson has a person run that checklist. Turning it into a judge is the next step for volume: paste Dana's real facts, the checklist and a draft into a fresh chat and ask for pass or fail on each line with a quote. You still open the source for any fact the judge flags.

[Try this lesson free](https://learn.ynteractive.com/modules/chatgpt-fix-bad-answers) or [read the step-by-step guide](https://learn.ynteractive.com/content/how-to-fix-bad-chatgpt-answers-push-back-constraints-fact-check).

## Common confusions

### LLM as a judge vs self-verification

Self-verification asks the same model to check its own answer in the same chat. A judge is a separate grading step, ideally with fresh context.

### LLM as a judge vs human review

The judge filters at scale. A person still decides on anything that goes to a customer and checks a sample of the judge's grades.

## Tips

- Write criteria as yes or no questions with the source facts attached.
- Ask the judge to quote the line that fails each check.
- Compare its grades with your own on ten examples before trusting it.

## Related terms

[AI evals](https://learn.ynteractive.com/content/glossary/ai-evals) · [Self-check prompt](https://learn.ynteractive.com/content/glossary/self-verification) · [Prompt testing](https://learn.ynteractive.com/content/glossary/prompt-testing) · [AI scorecard](https://learn.ynteractive.com/content/glossary/ai-test-scorecard) · [Human in the loop](https://learn.ynteractive.com/content/glossary/human-in-the-loop)

## More AI basics terms

[Local-first software](https://learn.ynteractive.com/content/glossary/local-first) · [Machine learning](https://learn.ynteractive.com/content/glossary/machine-learning) · [Max tokens](https://learn.ynteractive.com/content/glossary/max-tokens) · [Metadata filtering](https://learn.ynteractive.com/content/glossary/metadata-filtering)

## Where you use it: free lessons

- [Catch Made-up Facts Before They Reach Customers using ChatGPT](https://learn.ynteractive.com/content/how-to-fix-bad-chatgpt-answers-push-back-constraints-fact-check) (ChatGPT, 7 min)
- [Give Examples: Few-Shot Prompts That Copy Your Style](https://learn.ynteractive.com/content/few-shot-prompting-chatgpt-examples-your-style) (ChatGPT, 6 min)

## Frequently asked questions

**Can AI grade AI reliably?**

Reasonably well on clear, factual criteria, less well on taste or subtle accuracy. Spot-check its grades.

**Should the judge be a different model?**

Often it helps to use a different or stronger model and a fresh chat, so it doesn't share the first answer's blind spots.

**What should a judge prompt include?**

The source facts, the answer to grade, a short list of yes or no criteria, and a request to quote the line behind each failure.

[All AI glossary terms, A to Z](https://learn.ynteractive.com/content/glossary) · [Free prompt templates](https://learn.ynteractive.com/content/prompts)