What is A/B testing?
Quick answer
A/B testing means showing two versions of something, such as a subject line, ad headline or signup page, to two similar groups at the same time and comparing a single result, like clicks or signups. Version A is usually the current one and version B changes one thing.
Last updated
Updated · By Robert Breen
Why it matters for a small business
Marketing debates are often settled by whoever argues hardest. A/B testing replaces opinions with evidence from your own audience. Instead of guessing whether a friendly or a direct subject line works better, you send both and look at what happened.
The value comes from discipline. Change one thing so you know what caused the difference. Split people randomly so neither group is special. Decide in advance which number you will judge by, and wait for enough results before calling a winner. A test with ten people per side mostly measures luck. AI is useful for writing variations quickly, but it cannot tell you which one wins. Only the test can.
In a real lesson: n8n AI Agent Tutorial: Save Social Media Ideas to Google Sheets
In the AI Social Media Idea Agent lesson, you build an n8n agent for BrightPath Marketing, a made-up digital marketing agency. One idea the agent produces is an Instagram post explaining how ChatGPT can "brainstorm audience pain points, write ad copy variations, and suggest A/B test ideas." Another uses the hook "These three prompts took a local shop from guessing to testing."
That is the right division of labor. AI drafts the variations. A test decides between them. The lesson stops at saving ideas to the Social Media Ideas sheet with columns for Platform, Post Topic, Hook, Description and Call to Action, but a column like Hook is exactly what you would test.
To turn that sheet into a test, pick one row, ask the agent for a second hook with everything else kept the same, and run both versions to similar audiences on the same days. Compare one number you chose in advance, such as clicks on the call to action. The lesson does not run tests or report results.
Try this lesson free or read the step-by-step guide.
Common confusions
A/B test vs multivariate test
An A/B test changes one thing between two versions. A multivariate test changes several things at once in many combinations and needs far more traffic to say anything useful.
A/B test vs before and after
Changing a page and comparing this month with last month is not an A/B test, because the season, the news or the traffic may have changed too. A real test runs both versions at the same time.
Tips
- Write the test as one sentence first: "If we change X, Y will go up."
- Test changes big enough to matter, like a different offer or headline, not a comma.
- Keep a log of every test and its result so the team does not repeat old experiments.
Related terms
More Business terms
Where you use it: free lessons
- n8n AI Agent Tutorial: Save Social Media Ideas to Google Sheets (n8n, 12 min)
- Build an n8n AI Agent That Plans Your Content Calendar (n8n, 12 min)
Prompt templates that use it
Frequently asked questions
- How long should an A/B test run?
- Until each version has enough results to compare and at least a full business cycle, often a week or more, has passed. Stopping as soon as one side is ahead is a common way to pick the wrong winner.
- Can I A/B test with a small email list?
- You can, but only large differences will show up clearly. With a small list, test bold changes and treat close results as a tie.