Analytics: metrics, gaps and the weekly report

Analytics shows what your assistants achieved over a period and where they ran into limits: 32 metrics in five groups, along with the timeline, topics, open questions, detected gaps, drop-off points, restriction cases and recommendations. Every figure is counted. Where a metric lacks a basis, it reads Not measured rather than 0. Once a week the account owner can receive a summary by email.

Where to find analytics

In the menu, Analytics sits in the Records group, right after Acceptance. The page lives at https://app.xicbot.de/auswertung and requires a sign-in with a confirmed email address, see Sign-in and access. Conversations through the chat window on your website and via the API are counted. Conversations from the test dialog and the check conversations that a check run under Acceptance creates are not counted here; against your plan’s quota they count like any other conversation, see Plans and quotas. Testing in the dialog is covered under Test and acceptance.

Metrics and periods

At the top of the page you choose what the figures cover. All cards, lists and the timeline follow that choice.

Below the controls you see the start and end of the window and the time zone used for the calculation. The time zone comes from your browser; if it cannot be read, Europe/Berlin applies. If the browser reports an older name for a zone, such as Asia/Calcutta, the service calculates with the name the time zone database uses today, here Asia/Kolkata, and the page shows that name. Next to it you see Last measured at … or, before the first measurement, Not measured yet. Once the last measurement is 90 seconds old or more, not refreshed for … seconds appears. The service keeps each result for 60 seconds, separately per account, section, period, time zone and assistant, so Measure again within that time can return the same figures. If the service does not recognise the time zone, the page shows no figures but the message The service does not recognise the time zone of this device. The figures cannot be assigned to days and are therefore not shown.

If you change the period, a comparison you have already selected is recalculated with the new window length. If you change the assistant, the selected version is cleared. Retrieval quality and Probe set found measure the knowledge base shared by all assistants; selecting an assistant does not change them.

Reasons for “Not measured”

Usage

Answer quality

Tools

Outcomes

Boundaries and blocks

What to do next

This section lists one suggestion per issue found in the period, together with the area where you make the change. Only issues that occurred at least once in the window are listed, sorted by the number of cases. Each entry shows One case or … cases and a link Open …. If nothing stood out, it reads None of the observed anomalies occurred in this period. Whether a change moves the figure usually shows only in the comparison before and after the next version.

Timeline and quality over time

The Over time chart shows conversations per day in the selected time zone. The table below lists, for each day, Conversations, Messages, Handovers, With a source, Without an answer, Handover rate, Coverage and Retrieval quality. Quality over time plots the daily share of replies with a source on a fixed scale from 0 to 100 percent. A day without a reply gets no bar instead of being plotted as 0. Below this chart you see the highest and the lowest share with their dates and a pointer to the With a source column of the Over time table; it has no table of its own. Without data it reads There is no history for this period.

Language and screen width

This section shows where conversations come from: By language lists the language of the conversation, most frequent first, and By screen width the visitor’s window width in the steps Phone, Tablet, Desktop and Wide screen. Shares refer to conversations that have a value; the rest appear as … without a value. Without any counted conversation it reads There was no conversation on your website or via the API in this period., with a note below that test dialogs and acceptance runs are not counted.

Topics

Topics group your visitors’ questions. The basis is up to 5000 visitor questions from the period, newest first. Each question is masked before grouping (see Visitor texts); similar wordings form a topic, named after up to three leading terms and shown with an example of up to 300 characters. Up to 15 topics are shown, sorted by number of questions.

If more than 300 distinct question wordings come together, the 300 most recently asked are grouped; older wordings drop out even if they were more frequent. The page reports this as The … most frequent question wordings were bundled; rarer ones are not included. If no question was asked, it reads No question was asked in this period.; if there were questions but no recognisable topic: There were questions in this period, but none with a recognisable topic.

Questions without an answer

This lists the visitor questions for which the assistant could not generate an answer and delivered the Fallback answer instead. Up to 50 such replies are read, newest first; each shows the question before it, masked and cut to 300 characters. Similar questions are grouped with up to ten variants, the number of conversations and the share that was handed over; groups with the most conversations come first, with handed-over conversations counting twice. Open conversation takes you to Conversations. A fallback answer points to a failed answer run; what is missing from your knowledge is listed under Detected gaps. If every question was answered, it reads Every question in this period was answered.

Detected gaps

A row is created when the assistant has finished delivering a reply that cites no source and made no tool call while doing so, even if the search found passages in your knowledge; this applies whether or not a notification is set up for it. Replies from a question-and-answer pair and replies with a tool call, even one that was only proposed, do not create a gap. The kind reads Declined if the assistant declined to answer, otherwise Without a citation. No gap is recorded for questions shorter than twelve characters, such as a greeting, or while your account holds no knowledge yet. Up to 50 gaps are shown, each linked to its conversation. If the question has already been deleted under your retention period, it reads The question was deleted when its retention period ran out; the gap stays counted., see Retention and deletion. With no results: No gap was detected in this period.

Drop-off points

A conversation counts as abandoned when all of the following apply:

• It is still open and was not handed over to a person.
• The visitor wrote at least one message.
• More than 30 minutes have passed since the last message.
• The conversation produced neither an enquiry nor an appointment nor a cart.

The position tells you after which message the conversation stopped; the first message is position 1. Drop-offs by position in the conversation spreads the cases over Message 1, Message 2 and so on, next to the number abandoned and the outcome events by kind (Enquiry, Appointment, Cart). Outcomes without a conversation, such as an enquiry sent through the API, have no position and are missing from this list; with All assistants selected they still count in their card, unless they came from a deleted test dialog or check conversation. The list shows up to 50 cases with the last question, masked and cut to 300 characters. If the question has been deleted, it reads The last question was deleted when its retention period ran out; the drop-off stays counted.; if the position cannot be determined, Position cannot be determined. With no results: No conversation was abandoned in this period.

Restriction cases

Each row is a case in which a boundary took effect: Turned away if the request was ended before the model was called because the quota was used up or the conversation had reached the maximum number of visitor messages, or Boundary touched if the output check hit a restricted topic in a reply. The topics are Health, Legal, Finance, Allergies, Emergency, Complaint, Data protection and Payment data; restricted subjects carry the label Restricted topic. For rejected requests the topic is missing: No topic recorded. The reason a request was turned away is not stored on the message. Up to 50 cases are shown, along with the total of both kinds. With no results: No restriction took effect in this period.

Before and after

This section only appears when a single assistant is selected at the top. Under Version you pick a released version; the 50 most recent are offered, and if there are more it reads The 50 most recent versions are offered. Two windows of equal length around the release are compared, each as long as the selected period: … days before and after the release. If the window after has not yet reached its full length, it reads The window after it covers … days so far. Versions are released under Assistants.

Green with better means the figure moved in the desired direction, red with worse in the undesired one. Higher is better for Solved on its own, Answers with a source, Question coverage and Rated helpful; lower is better for Handed to a human, Without an answer, Conversations with friction and Abandonment rate. Number and length of conversations carry no rating. The table cannot tell you whether the version caused a difference.

Report as a file

Under Report, Report as a file downloads the figures for the selected period and assistant as a JSON file named after the period, for example xicbot-auswertung-30d.json. The file states when the data was collected and contains all metrics, the timeline, up to 15 topics and up to 50 each of questions without an answer, detected gaps, restriction cases and drop-offs. Language and screen width, recommendations and the version comparison are not included. Each download recalculates; an account can make 30 downloads per minute.

Weekly report by email

At the end of the page you find the section Weekly report by email: Every Monday a summary of the past seven days goes to the confirmed address of the account owner. New accounts have the report ordered. Below it you see Recipient: … with the address, or the warning No confirmed address is on file. Until then the service sends no report. How to confirm your address is explained under Account security.

The account owner can switch it on and off. Anyone with a different role sees the current state instead of the switch, The weekly report is ordered. or The weekly report is not ordered., and the sentence The weekly report is ordered by the account owner. It goes to the owner’s confirmed address. Through the API, other roles get status 403 with role_too_low. If the audit log entry for the change cannot be written, the weekly report stays as it was; the API responds with status 500 and not_logged, and the page shows Not saved: the entry in the audit log could not be written. Please try again. Roles are explained under Team and roles.

1. 1

   Step 1

   Open analytics

   Sign in as the account owner and open Analytics.

2. 2

   Step 2

   Find the section

   Scroll to Weekly report by email at the end of the page.

3. 3

   Step 3

   Tick or untick the box

   Tick or untick Receive the weekly report. The change is saved immediately; if saving fails, the error message appears and the box shows the stored state again.

4. 4

   Step 4

   Check the recipient

   Check the Recipient: … line. If it is missing, confirm your address first.

What the report contains

The report covers seven calendar days in the Europe/Berlin time zone, including the day it is sent, across all assistants. It counts like analytics: test dialogs and acceptance runs are left out. The subject names the Weekly report and the period. Its language follows the owner’s interface language, German or English, otherwise German, and applies to the whole report: subject, period, labels, numbers, reasons for missing values and notes. The Open analytics button leads to this page. At the end the mail states that the weekly report is switched on and can be switched off in analytics under “Weekly report by email”.

Below the figures, notes follow depending on what the week produced:

• Without a counted conversation, including a week with only test dialogs or acceptance runs: a sentence saying there was no conversation on your website or via the API this week, that test dialogs and acceptance runs are not counted, and that the analytics therefore stay empty without the measurement having failed.
• If your plan includes analytics: up to five most frequent topics with their number of questions, and the topics that were more frequent than in the equally long period before.
• For fallback answers, their number and, if your plan includes analytics, up to five grouped questions, each cut to 90 characters; without a grouped question, a note pointing to Without an answer in analytics. Without analytics in the plan, the note names the plan from which conversation analytics shows these questions.
• For handovers, the number of conversations that went to your team.
• If fewer than 50 percent of replies carry a source reference, a note pointing to a thin knowledge base.
• If none of this applies: a sentence saying there were no unanswered questions and no handovers to your team this week.

When no report is sent

• The account is not active.
• The weekly report is switched off.
• The account owner has no confirmed email address.

Visitor texts in analytics

Topic names and examples, questions without an answer, the last question at drop-off points and the questions in the weekly report are visitor text. It is masked before display: email addresses become [E-Mail], phone numbers starting with + or 0 and having seven or more digits [Telefon], account numbers in IBAN format [IBAN], street names with a usual German ending such as -straße, -weg or -platz plus house number, and a postcode with town at the start of a line or after a comma [Adresse]. Recognition works with patterns and usually catches the common spellings, not every one. If a text cannot be masked, it shows [nicht maskierbar]. Masking happens at output, not at storage; the stored conversation stays unchanged, see Conversations and Privacy and legal.

Limits at a glance

Through the API

The page reads its data through the same endpoints that are open to you. All read calls accept period (7d, 30d, 90d, default 30d), zone (zone identifier, default Europe/Berlin) and assistant (an assistant’s ID). Older names from the time zone database, such as Asia/Calcutta or Europe/Kiev, are accepted, and the service calculates with today's name (Asia/Kolkata, Europe/Kyiv); where a response carries the zone field, it holds that name. Upper and lower case do not matter, and the fixed offsets Etc/GMT+1 to Etc/GMT+12 and Etc/GMT-1 to Etc/GMT-14 are accepted (Etc/GMT+1 is one hour behind UTC). An unknown time zone, including an offset such as +01:00, returns 422 unknown_timezone, an unknown assistant 404 unknown_assistant. Authentication and keys are described under API: conversations and management.

Frequent questions