What is a golden dataset in AI testing?
Quick answer
A golden dataset is a fixed set of sample inputs, each paired with an answer a person has checked and agreed is right. It works like an answer key: every time you change a prompt, model or workflow, you run the same inputs again and compare the results with the agreed answers.
Last updated
Take the free module →Module 22 of the free n8n Jump Start course, about 11 minutes. No account needed.
Updated · By Robert Breen
Why it matters for a small business
Without an answer key, every AI test turns into a debate about whether the output "looks fine". A golden dataset settles that. If your invoice workflow should produce a total of 120.0 for a given client, that is written down before the test, and the test either matches it or it doesn't. Because the set stays the same, a score from this month can be compared fairly with a score from last month.
The word "golden" means trusted, not large. A useful set for an office task can be 20 to 50 real examples, including a few awkward ones. The hard rule is where the answers come from: a person writes them from the source, such as a timesheet or a policy, and never copies them from the AI output being tested. Otherwise the test just agrees with itself.
In a real lesson: Truquence: Catch an n8n Run That Says Success but Gets It Wrong
In Truquence: Catch an n8n Run That Says Success but Gets It Wrong, Priya Nair's invoice workflow reports success, but the Total node passed along 12.0 instead of 120.0. You run truquence init to make a contract.json with one check per node, then replace null on the Total check with [{"total": 120.0}]. The lesson doesn't call this a golden dataset, but it is the same idea in miniature: an expected answer written down before the check.
Truquence's docs are firm on the golden rule too: write the expected output yourself, for example from the client's timesheet, and never copy it from the run you are checking. When truquence accept compares the real run with that answer, it returns FAIL and names the Total node. After the fix, the same contract returns PASS.
Try this lesson free or read the step-by-step guide.
Common confusions
Golden dataset vs training data
Training data teaches a model. A golden dataset only tests it. Keep the two apart, or the test measures memory instead of skill.
Golden dataset vs ground truth
Ground truth is the correct answer for one item. A golden dataset is the whole curated collection of inputs and their ground-truth answers.
Tips
- Start with 20 real examples from your own work, plus a few unusual ones, and write each correct answer yourself.
- Freeze the set once it is agreed, and add new cases as new versions, so old scores stay comparable.
- Strip names and account numbers from real examples before you store them as test cases.
Related terms
More AI basics terms
Where you use it: free lessons
- Truquence: Catch an n8n Run That Says Success but Gets It Wrong (Truquence, 11 min)
- The Prompter AI: Prove a Better Prompt Works on Your Own Meeting Notes (The Prompter AI, 8 min)
Frequently asked questions
- How big should a golden dataset be?
- Big enough to cover your common cases and your known tricky ones. For one office task, a few dozen well-chosen examples often beat hundreds of random ones.
- Who should write the answers?
- Someone who knows the work, from the original source. Never copy answers from the AI output you are testing.
- When should I update it?
- When the business rules change, or when a real mistake slips through. Add that mistake as a new case so it is tested every time from then on.