Test dialog and acceptance

Before an assistant goes onto your website, you can check it in two places. In the Test dialog you ask questions yourself and see the answer, the citations and the final state before a visitor gets to see them. In Acceptance you add check questions with a model answer and run them as a batch against the published version; the verdict is computed and asks no model.

Test dialog

You find the test dialog in the menu under Work › Test dialog (app.xicbot.de/probe). In an assistant's setup path, the Test step between Connect knowledge and Embed leads straight here and preselects that assistant. Every test dialog is a conversation of its own in the Test channel. The answer is produced the same way as in the widget, including the stored fallback answer.

Starting

If you have several assistants, choose one in the Assistant field; with only one, its name appears after Testing. An address with ?a= and an assistant's ID preselects that assistant; an unknown ID falls back to the first assistant without a message. Start test dialog creates the conversation and shows Loading … meanwhile. If the assistant has a greeting, it appears as the first line.

The conversation keeps the version that was published when it started. If you publish a new version afterwards, you only see its effect after New test dialog. Without a published version the service refuses the start with 422 no_version, and the test dialog shows There is no published version of this assistant. Publish a version and start again. A newly created assistant, however, already has its first version published.

Sending a message

Your question is stored in the conversation right away, before the answer starts. While the assistant is writing, the button shows Answering …; the lines are labelled You and Assistant. Up to three citations appear below an answer, further ones behind + n more. While an answer is running, the conversation accepts no second message (409 reply_in_progress); a running answer holds it for 180 seconds at most. If the service refuses a message for this reason, the test dialog shows An answer to this question is already running. If the message is longer than your account’s maximum length, it shows The message is too long: at most … characters. In place of the dots, your account’s limit appears as a number.

Cancel stops the answer on the service as well, so it does not keep running unnoticed. The text written so far stays visible; no citations arrive after a cancellation, and the status line shows cancelled. When no answer is under way, two options sit next to Send: Open this conversation in the list opens it under Conversations, New test dialog starts an empty conversation. The old one is kept and can be found in the list with the Channel filter set to Test.

After the first answer the page shows If the assistant answers well, the next step is embedding it on your website. with the link Embed the widget. How that works is described in Embed the widget.

The test dialog shows the text, citations and final state of an answer. Tool cards, follow-up questions, the contact offer and a handover do not appear here; in the widget, follow-up questions and the offer to talk to a person appear as buttons below the answer. You confirm a proposed tool call under Open this conversation in the list. If the service corrects an answer after writing it, for example because a statement without a source gives way to a placeholder (“[nicht belegt]” in a German answer, otherwise “[not backed]”) or a mandatory notice is replaced by the promised one, the test dialog takes over the corrected wording, as it is stored and as the widget shows it. If the model fails in the middle of an answer, the test dialog shows only the stored fallback answer, not the cut-off beginning.

Final state of an answer

If an answer does not end normally, its final state appears below the conversation. An answer that ends normally shows nothing extra.

Every answer the assistant phrases with the model is a model call and counts against your plan's quota, in the test dialog too. If a question matches an active question and answer pair, that pair's answer goes out word for word without a model call, and nothing is booked; see Knowledge. Once the quota is used up, the fallback answer comes and nothing is booked. Details are in Plans and quotas. Test dialogs and the conversations of acceptance runs belong to the Test channel and follow their own Test conversations retention period; if nothing is set there, the general period for conversations applies. See Retention and deletion.

Permissions and throttling

With the Read role, the choice under Assistant, Start test dialog, the Your message field and Send are disabled. At the top of the test dialog, even while no assistant has been created yet, the note With the Read role this area is view-only. Changes need at least the Edit role. appears. The service itself refuses start and send for this role with 403 role_too_low; if such a refusal arrives when sending, the test dialog shows the same message. Exceeding the throttle returns 429 and the message Too many attempts. Please wait a moment. Roles are explained in Team and roles.

Acceptance

You find acceptance under Records › Acceptance (app.xicbot.de/abnahme). Every check question is asked once, the way a visitor would ask it, against the assistant's published version. The app recommends a run before going live and after every change. When the page opens, the first assistant is selected; if you have several, switch in the Assistant field.

Adding a check question

Fill in the fields and choose Add check question; while saving, the button reads Saving …. Afterwards Question, Model answer, Must contain and Must not contain are empty again, while Minimum coverage and Runs along keep their values. A new check question goes to the end of the list.

Below the term fields it says One term per line, at most 10. Case and umlaut spelling do not matter. Spaces at the edges and empty lines are dropped. The service stores duplicate terms only once if they match character for character: “Price” and “price” both stay, even though the comparison with the answer ignores case. It checks the limit of 10 terms before removing duplicates: 11 lines are refused even if two of them are the same.

How the verdict is reached

A check question counts as passed when three conditions are met at the same time. If one of them is missed, it has failed.

• Every term from Must contain appears in the answer. Whatever is missing is listed in the result under Missing.
• No term from Must not contain appears. Any that does is listed under Forbidden but said.
• The coverage reaches the Minimum coverage. This condition is dropped when Minimum coverage is 0 and when the model answer contains no meaningful word.

Terms are searched for as a string of characters, not as a whole word: ship also matches "shipping". The comparison is in lower case, with umlauts and accents folded (ä as ae, ß as ss, é as e) and with runs of spaces collapsed, so a term broken across lines still matches.

Coverage is the share of the model answer's meaningful words that also appear in the answer. A word is a sequence of letters, digits and hyphens with 3 to 64 characters; filler words such as "der", "die" or "und" are dropped, and each word counts once. A hyphenated compound counts as its parts and as the joined whole. The value is rounded to three decimal places and shown as a percentage.

not measurable means that no answer could be evaluated, for example because the quota was used up, the fallback answer was given or the answer stayed empty. Such a line has not failed. The reason is shown next to it, and the rate leaves it out.

Starting an acceptance run

1. 1

   Step 1

   Start the run

   Start acceptance run queues a run; the service answers right away with 202 and the work happens in the background. The button is disabled while a run is open, without a published version, without a ticked check question and with the Read role. If a run for the assistant is already queued or in progress and is less than an hour old, you get that one back instead of a second run. While it starts, the button shows Loading ….

2. 2

   Step 2

   Ask the questions

   The run takes the ticked check questions, at most 50, ordered by position and then by creation time, together with the version published when it begins. For each check question it creates a conversation of its own in the Test channel and asks the question along the same path a visitor question takes in the widget, with search and tools; tools count as Visitor (public chat) there. The answer language follows the language the service detects in the question. If detection fails, the check question's language applies, which can only be set via the API, and without it the version's language.

3. 3

   Step 3

   Follow the run

   During the run the page shows The run is queued. This page refreshes itself while it runs. It checks every 3 seconds as long as the window is visible. The counters grow after each check question; the list of check questions reloads once after the end.

4. 4

   Step 4

   Review

   Under Last run you see Questions, Passed, Failed, Not measurable, Rate and Duration, plus status, time and a reason where there is one. The time is when the run was queued. Duration shows Not measured while the run is queued or running, and stays that way if the run was stopped. Below follow the Results for each check question. In the list every check question shows its latest verdict and, under Last, when it was reached. Before the first run the page says No run yet.

The Rate is the number of passed lines divided by passed plus failed lines; lines that are not measurable are left out. If no line was measured there is no rate, and the page shows Not measured. When lines are not measurable, it says A line that could not be measured has not failed. The reason is shown next to it and the rate leaves it out. The app only shows the latest run; the API returns the 30 newest.

Reading the results

States, verdicts and reasons

An identifier without a display text, such as tenant_unknown, is shown by the app as it is rather than being hidden.

Changing and deleting check questions

In the list every check question shows Question, Model answer, its latest verdict with Last, the terms under Must contain and Must not contain and the Minimum coverage as a percentage. The Runs along tick box is saved on its own right away. Text and terms cannot be edited in the app: delete the check question and add it again, or change it via the API.

Delete removes the check question at once and without asking. Earlier results stay readable with their question and model answer but lose their link to the check question. If the assistant is deleted, its check questions and runs go with it.

Roles and throttling

Every role may view. Adding, changing, deleting and Start acceptance run require at least Edit. With the Read role you see check questions, the run and the verdicts, but the form, the tick boxes and Start acceptance run are disabled, Delete is missing, and the top of the page says With the Read role this area is view-only. Changes need at least the Edit role. Adding, changing and starting are throttled to 30 requests per minute and person. The counter is the same one used for adding and changing assistants, their versions and tools. Deleting has no throttle of its own. More under Team and roles.

API

All paths start with /api/v1/ and require a sign-in with a confirmed email address; see API basics, Authentication and Limits. The test dialog uses POST /api/v1/conversations with assistant_id and then POST /api/v1/conversations/{id}/send with text; both are described in Conversations and management. Acceptance uses the endpoints below. If the assistant {id} is unknown, each of them answers 404 not_found.

A check question in check and checks carries id, question, expected, must_contain, must_not_contain, min_overlap, language, position and is_active, plus last_result (pass, fail, error, or null before the first run) and last_run_at. Only a run sets the last two, even for a result that is not measurable; they cannot be written via the API.

Frequent questions