What is latency in AI?
Quick answer
AI latency is the time between sending a request to an AI model and getting the answer back. It depends on the model's size, how much text you send, how long the answer is, extra steps such as retrieval or thinking, and how busy the service is. For chat it is a nuisance; on a phone call it is the experience.
Last updated
Take the free module →Module 5 of the free Lovable Jump Start course, about 8 minutes. No account needed.
Updated · By Robert Breen
Why it matters for a small business
People accept a few seconds' wait in a chat window, but a pause on a phone call feels broken. That is why voice agent platforms push fast models. In automations, latency adds up when an AI step runs on every row of a sheet.
Speed is a trade-off you choose: larger models, higher effort settings, research modes and big documents give better answers on hard tasks and take longer. Pick the fastest setup that still passes your tests.
Long waits can also hide errors. If an automation times out waiting for a slow AI step, the run may fail or retry, so set sensible timeouts and an error alert, as the n8n error handling lesson shows.
In a real lesson: Add an AI Chatbot to Your Website with Lovable AI
In Add an AI Chatbot to Your Website with Lovable AI, you test the Hazelbrook Hair Studio assistant, then open More, then AI. Each test question appears as a request with the model it used, its tokens, the credits it cost and how long it took. That last figure is latency.
The voice course shows why it matters even more on calls. In Give Your Voice Agent Knowledge, ElevenLabs says turning on RAG adds about 250 milliseconds to each answer, which is one reason the lesson keeps two short documents in full context instead.
Try this lesson free or read the step-by-step guide.
Common confusions
Latency vs throughput
Latency is how long one answer takes. Throughput is how many answers a system can handle at once.
Latency vs rate limit
A rate limit caps how many requests you may send. Hitting it causes errors or waits, which can look like latency.
Tips
- For live calls and chat, try a small, fast model first.
- Keep prompts and documents short; every extra token takes time.
- Check the time per request in your tool's logs after real use.
Related terms
More AI basics terms
Where you use it: free lessons
- Add an AI Chatbot to Your Website with Lovable AI (Lovable, 8 min)
- Give Your Voice Agent Knowledge: A Price List and FAQ in ElevenLabs (ElevenLabs Agents, 9 min)
- Your First Voice Agent in ElevenLabs: An After-Hours Receptionist (ElevenLabs Agents, 9 min)
Frequently asked questions
- Why is my AI chatbot slow?
- Common causes are a large model, long instructions or documents, a high effort or thinking setting, retrieval steps, or a busy service.
- How fast should a voice agent respond?
- Fast enough to feel like a normal conversation. Test with real calls and pick a fast model, then read the transcripts.
- Does a longer prompt make AI slower?
- Yes, a little. The model has to read every token you send, so long instructions and documents add time and cost.