What is pretraining in AI?
Quick answer
Pretraining is the first and biggest stage of building a language model. The model reads an enormous collection of text and learns to predict the next word, which teaches it grammar, facts and writing styles. Later stages make it follow instructions, but pretraining is where most of its general knowledge comes from.
Last updated
Take the free module →Module 6 of the free Better Prompts course, about 7 minutes. No account needed.
Updated · By Robert Breen
Why it matters for a small business
Pretraining explains two habits you see every day. First, the model knows a lot about the world in general but nothing about your business unless you tell it. Second, when a fact is missing, it fills the gap with whatever usually comes next in text like yours, which is how a confident made-up policy appears.
It also explains the knowledge cutoff. Pretraining happens on text collected up to a date, so anything newer is unknown unless the tool searches the web or you paste it in.
In a real lesson: Catch Made-up Facts Before They Reach Customers using ChatGPT
In Catch Made-up Facts Before They Reach Customers, Dana at Juniper Lane Bakery asks for a cake pickup email but gives only the pickup hours. In the lesson's example draft, ChatGPT fills the gaps with a "next business day" policy and a line about enjoying the cake within 2 to 3 days. Those are typical bakery phrases from general text, not Dana's rules.
The fix is to supply what pretraining can't: Dana's real policy (cakes held until closing, call to set a new day), a request to list every assumption, and a web search with official food safety sources for the storage fact.
Try this lesson free or read the step-by-step guide.
Common confusions
Pretraining vs fine-tuning
Pretraining builds general knowledge from a huge dataset. Fine-tuning comes after and adjusts the model for a narrower job with a much smaller one.
Pretraining vs training on your chats
Your chats aren't part of the original pretraining. Some services may use chats to train future models unless you turn that off in data controls.
Tips
- Give the model your own facts: prices, hours, policies. It only knows the general version.
- Ask "what did you assume?" after any draft that goes to a customer.
- For anything recent, use web search or paste the source.
Related terms
More AI basics terms
Where you use it: free lessons
- Catch Made-up Facts Before They Reach Customers using ChatGPT (ChatGPT, 7 min)
- ChatGPT: Keep Client Data Safe, Set Up Your Team's Data Controls in 10 Minutes (ChatGPT, 10 min)
Frequently asked questions
- What does pretrained mean in GPT?
- It means the model learned from a large body of text before being shaped for any specific job, such as chatting.
- Can I pretrain my own model?
- In practice, very few businesses do; it takes huge amounts of data and computing. Most use a pretrained model and give it their facts in the prompt, files or a knowledge base.