The measurement series behind the figures on this website

The series asks 32 questions through the same public interface a widget on a customer site uses. 24 of them lie inside the knowledge base, eight deliberately outside — that is the only way to tell whether the assistant stops or invents something.

Each question gets its own conversation. A long history would fold an ever larger context into every further question and would therefore measure something other than the first question did.

There is a 5.5 second pause between questions. The throttle allows twelve requests per minute; without the pause, from the thirteenth question onwards the series would only be measuring our own brake.

Measurement happens character by character on the live stream. The first value is the time to the first character of the answer — measuring only at the end would fold the length of the answer into the number.

An answer counts as covered only when the assistant sends the passages it rests on in the same stream. An answer that is factually right but names no passage counts as not covered in this series.

The value is below the number of in-scope questions. That is not a defect we argue away: these are questions the assistant assembles from several passages, or answers with a question of its own without naming a single passage.

The second figure counts the out-of-scope questions that ended without a handover to a person. The lower it is, the more reliably an uncovered question reaches somebody who can answer it.

Eight questions in the series are chosen so that they cannot appear in any document: turnover figures, a private phone number, a forecast, a rating of a competitor. A measurement that only counts events could not tell, at this point, whether the assistant honestly declined or invented something.

That is why the answer text is recorded and read. A refusal counts as a refusal only when the text says the information is missing — not merely when no citation event arrived.

This is the most important figure on the page. An assistant that answers ninety-nine percent of questions quickly and invents something in the remaining one percent is worse in daily operation than one that hands over more often.

The size of the knowledge base is not asked about, it is counted: the number of sections our own material was split into. It says how much substance sits behind an answer, and nothing about how good that substance is.

The weight of the loader script is measured on the delivered file, not on the source. What is measured is the file that arrives in the browser, with the transfer compression the web server actually sends.

Both figures apply to our own installation. On a customer site the size depends on the material that was indexed and the weight on that web server's compression — which is why every figure states what it refers to.