AI chat response time: measure, don’t guess

How fast is your assistant? The honest answer to that question is rarely a single figure. A visitor does not experience one response time but several: the time until the first character appears on screen, the time until the answer stands complete, and, hidden inside both, the time the search in the knowledge base takes. Blend the three into one value and you get a number that reads well and steers nothing. More often the measurement is missing altogether: one second sounds plausible, so one second goes on the slide. This article describes which times can actually be measured on an AI assistant, why the mean is the most misleading of them, what median and 95th percentile show instead, and how to spot a figure with no measurement series behind it. The numbers in this text come from our own series against our own assistant, documented under measurement series.

Enlarge graphic

Labels in the illustration

• Course of one answer: where the time is spent
• first character
• complete answer
• Request leaves
• Knowledge search
• Model starts
• Answer streams out
• Distribution instead of mean
• Median
• Mean
• 95th percentile
• Outlier
• The mean moves with the outlier, the median stays put.
• Without these three details, no claim
• Measuring point: first character or complete answer
• Metric: median, 95th percentile or mean
• Scope: how many questions, which period, which network
• If one of them is missing, the figure cannot be checked.
• Own measurement series: 1.3 s to the first character, median
• 64 questions to our own assistant, not to customer projects

Measuring response time instead of guessing

Three times a visitor actually experiences

Several things happen one after another in a conversation with an assistant, and the visitor notices almost none of them. They type a question, press send and wait. What they perceive first is emptiness: the input is gone, nothing is there yet. That emptiness ends with the first character of the answer. After that a completely different perception begins, namely reading, and reading no longer feels like waiting, even if the text keeps running for a while. This is exactly why the time to the first character is the most important measuring point, not the total duration. An answer that begins after one second and keeps writing for five seconds is experienced as fast. An answer that appears complete in one go after four seconds is experienced as slow, even though it finished earlier.

This distinction is not hair-splitting; it decides which number you should measure in the first place. Anyone who starts the stopwatch only at the end of the answer inevitably measures the length of the answer as well. A detailed, well-supported reply then looks worse than a curt half-sentence, although it is worth more. Conversely, the bare time to the first character hides an answer that trickles out painfully slowly afterwards. It therefore makes sense to record both times separately and to look at them separately. How an assistant gets onto the page at all, and which paths that creates, is described in the article on embedding in website and shop.

Why a mean hides the outlier

The mean is the metric everybody knows, and the only one of the usual set that a single value can shift arbitrarily far. Take twenty answers, nineteen of which arrive quickly and one after a very long wait. The mean rises noticeably, although nineteen out of twenty visitors noticed nothing. It works the other way round just as well: a batch of very short answers pulls the mean down and makes a series look good in which individual outliers occurred. In both cases the number describes nobody's experience. It describes a sum divided by a count, and a visitor experiences neither sums nor counts.

The median solves this problem because it does not compute, it sorts. You lay out all measured times side by side in order of size and take the time in the middle. Half the answers were faster, half slower. A single extreme value at the edge barely shifts that middle, because it is only one position in the row and not a weight in a sum. This is precisely why the median is the figure you quote to a customer when you want to describe what they experience as a rule. What it does not describe is the edge: anyone who only knows the median knows nothing about the cases in which it took a long time.

Median, 95th percentile and the maximum

The 95th percentile answers the question the median leaves open: how long did it take in the unfavourable cases? It is the value below which nineteen out of twenty answers stayed. Put differently: one twentieth of visitors waited longer than that figure. This is the quantity complaints attach to, because whoever is annoyed by slowness almost always belongs to that twentieth. Median and 95th percentile together describe a distribution precisely enough for any decision that comes up in a mid-sized project. If the two lie close together, the assistant runs evenly. If they are far apart, there is a case that regularly falls out of line, and it is worth hunting down.

• Median: the middle of the sorted row. Half the answers were faster, half slower.
• 95th percentile: the value below which nineteen out of twenty answers stayed. The quantity for the unfavourable case.
• Maximum: the longest single case in the series. A pointer, not a control value.
• Minimum: the shortest single case. Popular as a marketing figure, meaningless for operations.
• Number of questions and period: without both, each of the four figures is an anecdote.

It matters that these metrics come from the same series. A median from the morning and a 95th percentile from the night do not form a picture together, they form a confusion. Nor may you mix metrics taken at different measuring points: a median to the first character next to a maximum to the complete answer looks like an outlier where in fact two different things were measured. So keep every figure together with its measuring point, the way a temperature without a unit is useless.

Where the time in the sequence is spent

An answer is produced in sections, and each section has its own duration. First the question leaves the browser and reaches the server. Then the knowledge base is searched for the sections that match the question. Our own knowledge base holds 2,670 sections from 70 documents; that figure applies to the corpus of the assistant answering on our site. Whether searching such a corpus costs noticeable time depends less on the volume than on the cut: sections that are too large return blurred hits, sections that are too small tear the context apart and force you to collect more of them. After that the model starts phrasing, and at that moment the first character drops.

After that the duration depends mostly on the length of the answer and on the amount of context handed to the model. A long history, many supplied sections and extensive instructions lengthen every single step. At the end comes the rendering in the window, and that costs time too if the widget only loads once somebody clicks on it. This last span appears in no server-side measurement and is therefore regularly overlooked. It is part of what the visitor experiences, though, and so it belongs in the measurement. Why an answer must not only be fast but also bound to your own content is covered in the article on hallucinations and the knowledge base.

No measurement series, no claim

The most common time figure in documents about assistants is estimated. It comes about like this: somebody opens the window, types a question, finds it fast and writes down one second. That figure has three problems. It rests on a single attempt, it usually comes from your own network and often from the same location as the server, and it names no measuring point. Later it appears in the quote, in the manual and eventually on the website, and nobody can say any more what it referred to. The way out is not heavier measuring equipment but a short, repeatable series with a defined procedure.

As soon as such a figure goes public it is also a statement about a feature of the service. Section 5 (1) of the German Act Against Unfair Competition (UWG) provides that a misleading commercial practice is unlawful if it is apt to cause the consumer or other market participant to take a transactional decision they would not otherwise have taken. A time figure nobody measured is exactly the risk that sentence describes. Attach measuring point, metric and scope to it instead and an assertion turns into a checkable statement, and that can be defended.

• The measuring point: first character or complete answer
• The metric: median, 95th percentile, mean or maximum
• The number of questions and the period of the series
• The scope: which assistant on which knowledge base
• The path of the measurement: through the public interface or internally
• What was excluded: warm-up runs, repetitions, failed attempts

Which deadlines really apply

There is no statutory requirement for how long a chat answer may take. That does not mean no deadlines run around an assistant; they simply attach to other procedures. As soon as an assistant processes personal data, and it does so with the conversation history alone, the deadlines of the General Data Protection Regulation apply. If a data subject asks about their data from the chat, a hard deadline runs from receipt of the request, and it is not measured in seconds but in one month.

A second deadline sits in Article 33 (1) GDPR: a personal data breach must be notified to the competent supervisory authority without undue delay and, where feasible, within 72 hours after having become aware of it. Anyone storing conversation histories should know that deadline before needing it. Nothing about answer duration can be derived from the AI Regulation, Regulation (EU) 2024/1689, but a useful benchmark can: Article 15 (3) requires high-risk systems to declare accuracy levels and the relevant metrics in the instructions for use, and Article 72 requires post-market monitoring. A chat assistant on a company website will as a rule not fall under the high-risk classification. The underlying idea still holds: name a metric and keep observing it in operation.

How we measure without fooling ourselves

Our series runs through the same public interface a widget on a customer site uses. That is the first and most important decision: measuring internally, past the application, leaves out exactly the paths that cost time in operation. Each question gets its own conversation so that no growing history is measured along with it, because with every further question in the same window a larger context would ride along and the series would measure something other than at the start. There is a pause between questions; without it, from a certain question onwards only our own throttle would be measured.

• Each question in its own conversation so that no history is measured along with it
• Timestamps on send, at the first character and at the last character
• A pause between questions so that the throttle does not become the measurement
• Questions inside and outside the knowledge base in the same series
• The answer text is recorded so that a fast wrong answer does not count as a success
• Median, 95th percentile, minimum and maximum from the same series, not from different ones

The last point is the one most often missing. A response time without a look at the content rewards exactly the wrong behaviour: an assistant that fires off a short standard sentence at every difficult question wins any pure time measurement. That is why every time series needs a second column stating whether the answer was supported and whether an uncovered question reached a human being. How that handover runs cleanly is described in the article on handover to staff; how to build the catalogue of test questions is shown in the article on acceptance and test cases.

What the figure steers in operation

A measured figure is only worth something if it triggers a decision. Three uses have proven themselves, and all three assume the figure comes from an ongoing series and not from a one-off measurement before go-live. The median becomes the observed quantity, the 95th percentile becomes the threshold, and together they say when a change to the knowledge base or to the extent of the instructions has actually had an effect, and when only the feeling has changed.

So that the series does not fall asleep, it belongs in a fixed rhythm: once in full before go-live, again after every larger change to the knowledge base, and regularly thereafter. Which questions are actually asked, and where the answers lag behind, is shown by the anonymised conversation analytics; the current state of operations is shown on the status page. How the case where the assistant does not answer at all can be rehearsed in advance is covered in the article on acceptance testing.

Limits: what a measurement does not say

A measurement series describes one assistant, on one knowledge base, in one period, across a certain number of questions. It says no more than that. It cannot be transferred to another project, because different content, different questions and a different network path come together there. Nor does it say anything about tomorrow: a change of model, a grown corpus or an altered instruction shifts the distribution, and then the old figure no longer holds. And it says nothing about the quality of the answer. A fast wrong reply is worse than a slow correct one. That is why time measurement stands next to the question of whether the answer was supported by the knowledge base. And because waiting does not affect everyone alike, accessibility belongs in this picture too: someone using a screen reader experiences slowly incoming text differently from someone skimming it. How our own series is set up, which questions it asks and where it stops is documented under measurements, so you can lay your own figures next to it.

Frequent questions