Letting Customers Rate Answers: Using Feedback from the Chat

Below an answer in the chat sits a small question: was this helpful? Clicking yes or no takes the visitor hardly any effort and costs the business nothing. Even so, this feedback often goes unused, because nobody has decided who reads the ratings, how to interpret them and what happens next. This article describes the loop that turns a simple rating per answer into a quality measure: group poorly rated answers by topic, find the cause, update the knowledge base and check the same question again. It also covers the traps that distort a rate and the rules that let the evaluation work without personal profiles.

Enlarge graphic

Labels in the illustration

• Answer rating
• Yes or no below
• every answer
• Group by topic
• Separate one-offs
• from patterns
• Find the cause
• Read history and
• sources of the answer
• Fix knowledge
• Add content or
• update the page
• Check again
• Same question in
• test and probe set
• Next time window: same topic, fewer not-helpful ratings?
• Four causes and one special case: what to do
• Cause
• Correction
• Knowledge missing
• Add content to the knowledge base
• Page out of date
• Update the source and re-index it
• Wrong source used
• Resolve conflicts, exclude old material
• Question unclear
• Add a clarifying question to instructions
• Unhappy with the matter
• No fix to the answer: explain the rule
• Read the coverage first
• Replies with a rating
• How much has been rated at all
• Visitor ratings
• How many ratings fell in the window
• Rated helpful
• Meaningful only with enough ratings

Rating answers: from a no to a corrected answer

A low-cost quality measure in a chat

An assistant can record a lot about itself: which sources it used for an answer, how well the question was covered by the stored material, whether it handed over to a person. These figures measure the answer from the inside. Whether it actually helped the person who asked appears in none of them. An answer can carry a source and still miss the question, because the source is outdated or the question was meant differently. A rating per answer closes exactly this gap: it measures from the outside, costs the visitor one click and is attached to exactly the point in the conversation it concerns.

The rating is therefore not a substitute for the other figures but their counterpart. How an evaluation is structured as a whole, with topics, knowledge gaps, drop-offs and buying signals, is described in the article on conversation analytics. The focus here is narrower: the loop from rating via cause to correction and back to measurement.

What the rating looks like in the chat

The simpler the rating, the more likely it is to be used. Two states are enough: helpful or not helpful. A star scale asks for a weighing-up that hardly anyone wants to do in the middle of a conversation, and it produces in-between values that are hard to interpret. More important than the form is the place: the rating belongs to the individual answer, not to the whole conversation. Only then can you later tell which piece of information did not help, rather than merely that something in the exchange did not fit.

• Below every answer, not only at the end of the conversation, when many visitors have already closed the window.
• Voluntary and unobtrusive: no mandatory field, no pop-up window, no repeated request after every message.
• Neutrally worded: the question is whether the answer helped, not whether the visitor was satisfied or would recommend the business.
• Reversible: anyone who clicked by mistake or later sees the answer differently can change or withdraw the rating.
• Nothing in return: no vouchers or perks for a rating, and no rating as a condition for continuing to write.

Reading the figure: coverage first

A rate of helpful ratings looks like a clear figure, but without its denominator it is hard to interpret. Those who rate are a selection, and not a random one: whether someone clicks may also depend on the mood in which they read the answer. The XICBOT evaluation therefore shows the ratings in three cards. How the metric cards count as a whole, with numerator and denominator for every share, is described on the page about XICBOT analytics. If there is not a single rating in the period, the card for the share shows no value but the reason.

Grouping poorly rated answers by topic

A single no says little. Perhaps the visitor was looking for something else, perhaps it was a misclick. It becomes meaningful when several not-helpful ratings concern the same topic: delivery times to Austria, returning made-to-order items, opening hours on public holidays. That is why the evaluation does not start with the rate but with the rated answers themselves. In XICBOT, the suggestions under “What to do next” in the evaluation include one of its own for answers rated not helpful, with the number of cases and the knowledge area as the place where the change is made. In the conversation view each conversation opens in a side panel; there every answer shows the visitor's rating alongside its sources and tool calls.

• Collect the not-helpful ratings: all answers in the period rated as not helpful, with the question before them and the answer word for word.
• Assign a topic: allocate each answer to a topic, in your customers' language rather than that of your departments.
• Look for patterns: topics with several not-helpful ratings first; set single cases aside until they recur.
• Run a cross-check: are there yes ratings on the same topic too? Then the problem often lies in one variant of the question, not in the topic.
• Match against the gaps: if the same topic also appears among the knowledge gaps or the unanswered questions, the cause is usually found quickly.

Four causes behind a no

Once a topic has been found, the question of why follows. A look at the conversation pays off: the question word for word, the answer, the sources used and what the visitor wrote afterwards. As a rule the no can be attributed to one of four causes, and each calls for a different correction. A fifth group looks like a defect but is not one.

For the first cause there is a second signal that arises independently of the rating: answers without a source reference. In our own measurement series of 11 September 2026, 42 of 48 answers carried a source reference; the count covered questions whose answer is in the knowledge base, and six remained without one. Answers like these are the first candidates when not-helpful ratings pile up on a topic. How the source reference itself should be designed is described in the article on citing sources in chat.

Updating and checking again

Corrections are made where the cause lies, and as a rule not in the assistant's wording. If knowledge is missing, a section goes into the knowledge base; if a page is outdated, it is corrected on the website and re-indexed. In XICBOT every indexing run writes into a new version and switches over only at the end; if it fails, the previous state keeps answering. If a piece of information should go out exactly as you worded it, for returns or withdrawal for example, store a curated question-and-answer pair: its wording goes out unchanged, without a model call.

• In the test area, ask the same question in the visitor's words and check whether the new answer cites the right source.
• As a probe line, store the question together with the expected document. The next probe-set run shows whether that document is found and in which position.
• Check variants as well, including the short and the long-winded version of the question, because visitors rarely write the way the material does.
• In the next time window, see whether the not-helpful ratings on this topic decline, with comparable coverage.
• Record what was changed and when, so that a movement in the figures can be attributed to a change.

The loop does not replace regular maintenance; it sets its priorities. What is checked on a fixed schedule anyway, prices, deadlines, opening hours, is covered in the article on keeping the knowledge base up to date. The ratings additionally show which content visitors actually miss, and therefore where an extra round pays off first.

Three traps in the evaluation

The third trap deserves a second look, because it turns a correct answer into an apparent defect. An assistant that finds nothing in its material on a question should say so instead of guessing. In our own measurement series of 11 September 2026, 8 of 8 questions without a basis were declined instead of guessed. Such a refusal is unsatisfying for the visitor in the moment and may receive a no. Anyone who counts this no as a defect and wants to push the rate up creates an incentive to water the refusal down. Why that is the wrong path is described in the article on declining instead of guessing.

Displeasure with the matter is different from annoyance within the conversation itself. If someone repeats the same question, writes in capital letters or asks for a person, a handover is often more helpful than another answer; that belongs to detecting displeasure during the conversation, not to the evaluation afterwards. How an assistant recognises such signs and hands over is described in the article on detecting frustration in chat. The rating complements this detection: it is given even where not a single word of annoyance was written.

Data protection: evaluate in aggregate, no personal profiles

A rating on its own names no one. But it is attached to an answer in a conversation, and that conversation may contain personal data: a name, an order number, an email address. As long as the rating can be linked to a person via the conversation, it is part of that data. Totals over a period are different, such as the number of ratings in a week and the share of helpful ones. According to Recital 26 of the GDPR, the principles of data protection should not apply to anonymous information, meaning information that does not relate to an identified or identifiable person. Caution is needed with very small numbers: a week with a single rating may allow conclusions about a particular conversation. What should not arise, by contrast, is a picture of individual visitors. The GDPR defines profiling in Article 4(4) as follows:

• Do not collect ratings per visitor: no list of who clicks no how often, and no classification of visitors as critical or satisfied.
• Record the purpose: the ratings serve to improve the answers, not advertising and not the assessment of employees. Article 5(1)(b) GDPR requires that data be collected for specified purposes and not further processed in a manner incompatible with them.
• Show little: for finding causes, the masked conversation is usually sufficient. Article 25(2) GDPR requires default settings under which only the data necessary for each specific purpose are processed.
• Observe retention periods: in XICBOT the rating is a field on the answer, not a separate collection. When the conversation is deleted after the period you set for conversations, the rating goes with it; earlier periods then show correspondingly fewer ratings. How such periods are set is described in the article on retention periods for chat logs.

When ratings reflect on employees

A rating assesses an answer from the assistant. But it can quickly turn into a rating of people: of the person who maintains an area of the knowledge base, or of the team that helps out after a handover. In businesses with a works council, this raises the question of co-determination, and not only at the evaluation stage. Section 87(1) no. 6 of the German Works Constitution Act (BetrVG) lists as a matter subject to co-determination:

Whether a rating function is such a device depends, according to the case law of the Federal Labour Court, not on the intention with which it is introduced. What matters is whether it is objectively suitable for collecting information about the behaviour or performance of employees, for example when ratings can be attributed to individual people via responsibility for an area or via a handover. On this view, monitoring begins with collecting such information, not only with evaluating it (decision of 13 December 2016, 1 ABR 7/15). Evaluating only by topic therefore does not automatically keep a rating function outside co-determination; even introducing it may be subject to co-determination.

For the loop, a simple rule still applies: answers are rated, not people. The ratings lead to topics and causes, and the correction is made by whoever is responsible for the content, without a rate per person emerging. In businesses with a works council, it is advisable to discuss the rating function with the works council and the data protection officer before it goes into use, all the more so if ratings are to be used to assess employees. Whether a specific set-up is subject to co-determination is a question for professional advice in the individual case. How responsibilities and approvals are sensibly distributed in the team is described in the article on roles and approvals.

Embedding the loop in day-to-day work

The loop only holds if it runs regularly and someone is responsible for it. As a rule, a fixed round each week or every two weeks is enough, in which the rated answers are reviewed, the topics maintained and the corrections recorded. Before the first round, it should be settled what the loop is meant to measure and what it is not.

• The rating sits below every answer, is voluntary, neutrally worded and can be withdrawn.
• One person is responsible for the round; corrections are made by whoever maintains the content in question.
• Coverage is read first, then the not-helpful ratings by topic, and the rate last.
• Every topic with not-helpful ratings is given a cause from the four groups, or a note that the displeasure concerned the matter itself.
• Every correction is checked with the same question: in the test area, as a probe line and in the next time window.
• The evaluation is aggregated; there is no rate per visitor and none per employee.

Frequent questions