AI Assistant Acceptance: Test Cases Before Go-Live

Accepting an AI assistant that rarely answers the same way twice: test cases from real enquiries, release criteria and regression runs. How to plan go-live.

In plain language

A language model does not use the same words each time. So you test its behaviour, not its wording. The test cases come from real questions, typos included. Before the build, it is set when the assistant may go live. Some errors stop the release on their own. The test covers right answers, clear refusals on legal and price questions, timely handover and topic changes. Two people check borderline cases, one of them from the department. After each change the test runs again, which lowers the risk but does not remove it.

July 24, 2026 13 min read By XICBOT

Key points

A language model words the same answer differently each time, so a literal target-actual comparison misses the point. Acceptance checks behaviour, and because the input space is open it works as a prioritized sample by frequency and potential damage.
The test catalogue holds the 50 most frequent real enquiries taken from phone notes, the shared inbox and search terms, kept in their original wording including typos, versioned and frozen. Invented ideal dialogues only test what was built anyway.
Release criteria are written down before the build: a score threshold across the catalogue, for example 45 of 50 points, plus knock-out criteria that individually block release. ISO/IEC 42001 requires acceptance criteria to exist before testing starts.
Four dimensions carry the assessment: factual accuracy backed by the knowledge base, clean refusal on legal statements and price commitments, timely escalation with context, and behaviour under topic switches, provocation and foreign languages.
Scoring follows a rubric from 0 to 3 points defined with examples beforehand. Borderline cases and every downgrade go through a two-person review, with the second reviewer coming from the department and not involved in building the system (NIST AI Risk Management Framework).
Every knowledge-base change triggers a regression run against the unchanged catalogue, plus a full run each quarter, recorded with date, version and score. An acceptance procedure lowers the risk of wrong answers, it does not remove it.

Why an AI assistant cannot be signed off like a form

Conventional software is deterministic: same input, same output. That is exactly what standard acceptance is built on. You define a target value, compare it with the actual value and record the deviation. A language model works differently. It picks every word from a probability distribution, which is why the same question produces differently worded answers. Two replies can sound entirely different and both be correct. Conversely, an answer can sound familiar and still contain an invented deadline. Testing for word-level identity measures the wrong thing. The question is not whether the assistant produces the expected sentence, but whether its behaviour in a given situation is defensible.

The second peculiarity is the open input space. A form has eight fields with defined value ranges; a chat accepts any conceivable sentence, in any phrasing, in any mood, in any language. Complete coverage is therefore out of reach. That is not a reason to skip testing, it is the reason for a deliberate, prioritised sample. The NIST AI Risk Management Framework states precisely this in its MEASURE function: metrics and test methods are selected for the highest-priority risks first, rather than blanket coverage of everything (NIST AI Risk Management Framework). Translated for a mid-market project: test what happens often, and what hurts when it goes wrong. The order matters more than the volume.

Deriving test cases from real enquiries

The most common mistake in acceptance preparation is well meant and still damaging: a team sits down and invents dialogues. What emerges are wish dialogues. They hit the tone you hoped for and the questions you are prepared for. What ends up being tested is what was built anyway. Real enquiries are blunter, shorter, more ambiguous, and they ask exactly the questions the concept missed. An acceptance set made of wish dialogues certifies a quality the assistant is not required to sustain in daily use, and therefore does not sustain.

The robust alternative is unglamorous: collect the 50 most frequent questions your business actually receives, from the places where they land. Phone notes from recent weeks, the shared inbox, the search terms from your on-site search, and the enquiries your team would rather not answer again. Those 50 questions are your test-case set. They hold one advantage no invention can match: they have already been asked, in exactly the wording in which they will be asked again. Which content belongs in the knowledge base as a result is covered in our article on building a knowledge base.

Phone notes

The questions your team answers on the line are the most honest source. They are unfiltered, often incompletely phrased, and show what customers genuinely worry about.
Shared inbox

E-mails deliver whole enquiries with context and attachments. They show how fully or how briefly people describe their situation before anyone asks back.
Search terms

On-site search and search queries reaching your pages reveal the wording people start with on their own, with no regard for your menu structure.

These sources supply the weighting as a by-product. If 20 of the 50 questions come from the phone, 18 from the inbox and 12 from search, you know the distribution of your real enquiry volume. An assistant already in service supplies this data continuously, which is what our conversation analytics is built for: it shows which questions are actually asked and where answers break down. For a first go-live, the existing channels are enough.

Collect: gather enquiries from the past four to eight weeks across phone, mail and search, without filtering.
Consolidate: bundle equivalent questions into one test case and note the frequency.
Prioritise: sort by frequency and by potential damage if answered wrongly, not by convenience.
Keep the wording: retain the original phrasing, typos and half sentences included, instead of smoothing it out.
Extend: deliberately add edge cases with no pleasant answer, such as legal and pricing questions.
Freeze: version the set so later runs measure against the same yardstick.

Defining release criteria before the build

Anyone who formulates release criteria after the build inevitably formulates them along the lines of what the system can already do. That is human, and it makes the acceptance worthless. The sequence is therefore not a formality, it is the actual mechanism. ISO/IEC 42001, the international standard for AI management systems, states exactly this in its control on verification and validation of AI systems: acceptance criteria should be defined before testing begins, in order to avoid subjective decision-making (ISO/IEC 42001 — AI management systems). The standard also requires release criteria that a system must satisfy before deployment, including a determination of which error rate is acceptable in the given context (ISO/IEC 42001 — AI management systems).

The question that comes before the build

Not: has the assistant turned out well? But: how exactly would we recognise that it is not yet good enough, and what happens then? Answering that in writing before the first line of configuration means the acceptance has essentially already been conducted. Everything after it is measurement.

In concrete terms: a score threshold across the whole set, for example 45 out of 50 possible points, plus hard knock-out criteria that individually block release. An assistant scoring 48 out of 50 but making a price commitment in one case is not released. The score averages; the knock-out criterion does not. That separation is the difference between an acceptance that captures risk and one that averages it away. Which capabilities are on the table in the first place depends on the scope, which we describe under custom functions.

The four review dimensions in detail

Four dimensions cover the practically relevant part of the behaviour. They are cut deliberately so that each can be scored on its own and none masks another. An assistant can be technically brilliant and still fail because it does not know its boundaries. The MEASURE function of the NIST framework dedicates a category of its own, with several subcategories, to the evaluation of trustworthy characteristics, from performance through safety to explainability. For a mid-market project, these four are the workable core.

1. Factual accuracy

Is the statement true, and is it covered by the knowledge base? What is checked is not merely whether the answer happens to be right, but whether it is sourced. A correct answer drawn from general model knowledge is a passed test case with a failed procedure.
2. Boundaries

What must the assistant not answer? Legal advice, individual price commitments, binding promises on dates or scope. Only a clean decline counts here. This dimension is a knock-out criterion, not a point collector.
3. Escalation

Does it hand over to a human in good time, with context, without making the person explain their situation twice? Escalating too late is as much a fault as escalating too early, because both cost trust.
4. Behaviour under stress

An incomplete question, an abrupt topic switch, a provocation, a foreign language. This is where you see whether the configuration holds or whether only the sunny side was covered. Disruption is the normal case, not the exception.

The second dimension deserves particular attention because it touches the legal frame. An assistant's statements can be attributed to the business, which our article on chatbot liability after the OLG Hamm ruling puts in context. The test case therefore is not: does it answer cleverly here? But: does it decline the commitment cleanly and offer the route to a human? Where assistants act on their own, in the cart for instance, that boundary shifts again, as our article on AI agents in the shop shows.

The fourth dimension is routinely underestimated in projects. An assistant that handles clean questions with confidence can lose the thread mid-sentence on a topic switch, or respond to a provocation with wording nobody wants to see published. A question in another language is a real test case too, which our multilingual assistant covers by design. Deliberately manipulative inputs are a field of their own, addressed in our article on prompt injection.

Scoring with a rubric instead of gut feeling

As soon as several people judge answers, the scoring drifts apart. One finds a reply too terse, another finds it precise. Without an agreed yardstick, whoever talks longest decides. A rubric replaces that discussion with a scale written down before the run. Three points per test case are entirely sufficient: 3 means defensible without reservation, 2 defensible with a minor weakness, 1 problematic, 0 not acceptable. What matters is not the granularity of the scale but that the levels are illustrated with examples in advance.

Aspect	Acceptance by gut feeling	Acceptance with rubric and test set
Test cases	Improvised during the meeting	50 real questions, versioned in advance
Yardstick	Feels good enough	Points per dimension, defined upfront
Edge cases	Rarely come up at all	Fixed in the set as knock-out criteria
Disagreement	Whoever argues more convincingly	Second review against the same scale
Result	An impression in the minutes	Score, status and open items
Repeatability	Different the next time round	Same set, comparable value

The side effect is the real gain: a rubric makes disagreement productive. When two reviewers score the same test case differently, it is as a rule because the level description is too vague. That discussion sharpens the criterion instead of persuading the person. After two or three runs the rubric is precise enough that the scoring itself costs barely any time.

Two-person review for critical answers

Not every test case needs two pairs of eyes, but every critical one does. Critical means: everything falling under the boundaries dimension, everything that could act as legally or financially binding, and everything a reviewer scored below full marks. The NIST framework explicitly recommends internal experts who did not serve as front-line developers of the system, or independent assessors (NIST AI Risk Management Framework). The reason is mundane and effective: whoever built a configuration reads the answer knowing how it was meant. That leniency is unavoidable, which is exactly why you design it out instead of fighting it.

Whoever built an answer reads its good intentions into it. Whoever sees it for the first time reads only what is there. That second reading is the one your customers get.

In practice a lean rule suffices: reviewer A works through the entire set, reviewer B takes the critical cases and every downgrade. Where they disagree, the stricter score stands rather than a compromise, until the criterion has been sharpened. Reviewer B should come from the business side, not from engineering. Someone who spends their day explaining on the phone why a deadline runs the way it does spots a skewed answer faster than any metric. How the handover to that team works technically is described in our article on handover to human agents.

A regression run after every knowledge-base change

Here lies the trap projects miss most often. The knowledge base is not a finished artefact, it grows. A new price list is added, an FAQ entry is extended, a PDF is handed in later. Every one of these additions can flip answers that were previously clean. Not because the new source is wrong, but because it competes with an old one, because a passage matches the question better than the one actually intended, or because a detail overrides a general statement. An assistant that quoted the delivery time correctly yesterday quotes the one from the old data sheet today, because that data sheet has just been added.

So the rule is: every change to the knowledge base triggers a run against the existing set. That is the regression run, and it is the reason the effort spent on the set pays off at all. The first time it is work; from the second time on it is a safety net. The NIST framework explicitly requires approaches, personnel and documentation in its MEASURE function to regularly identify and track existing, unanticipated and emergent risks (NIST AI Risk Management Framework). A repeated run against a frozen set does exactly that. Why source maintenance is so central to answer quality is explored in our article on hallucinations and the knowledge base.

A new or changed source in the knowledge base triggers a run, not an assumption.
The test set stays unchanged so that values remain comparable across runs.
Deviations from the previous run are reviewed individually, including the improvements.
Contradictory sources are resolved in the content itself, not papered over with an extra rule.
Prices, deadlines and availability count as especially prone to flipping and are checked first.
The run is recorded with date, version and score, even when nothing stands out.

The acceptance record as documentation

An acceptance record is first of all a plain working tool: it captures what was tested, with what result, and what remained open. That very plainness is what makes it valuable. ISO/IEC 42001 requires, in its control on verification and validation, the documentation of anomalies, test results, corrective actions and retesting (ISO/IEC 42001 — AI management systems). The standard comprises 38 controls (ISO/IEC 42001 — AI management systems), grouped into topic areas from AI policy to third-party relationships. No mid-market project needs all of them, but the pattern carries over: what you did not write down, you arguably did not test.

Then there are the legal obligations. Under Article 50 of the EU AI Act, systems interacting directly with people must be designed so that users are informed they are dealing with an AI, and this at the latest at the time of the first interaction (EU AI Act, Article 50). The transparency obligations apply from 2 August 2026 (EU AI Act, Article 113), and as a rule the disclosure duty in the chat also covers systems placed on the market before that date. The Digital Omnibus grants such older systems a transition period only for the machine-readable marking of AI-generated content. Infringements of Article 50 can be sanctioned with up to 15 million euros (EU AI Act, Article 99) or 3 percent (EU AI Act, Article 99) of total worldwide annual turnover, whichever is higher; for small and medium-sized enterprises and small mid-caps, the lower amount applies. A test case checking the disclosure takes two minutes and belongs in every set. Our article on the EU AI Act disclosure duty provides the context.

Acceptance is not a date, it is a recurring run

The most widespread misconception in projects is that acceptance is a date in the plan. A meeting where everyone looks at a screen, nods and signs. After that the system counts as accepted and the topic as closed. With an AI assistant that assumption does not survive three months. Content shifts, the offering changes, questions move with the season, and the underlying model is developed further by its provider. A test case that passed in January can fail in July without anyone having done anything wrong.

That this movement is real shows in the capabilities themselves: on real computer tasks, AI agents jumped from 12 to around 66 percent (Stanford HAI — AI Index Report 2026) task success, while documented incidents rose to 362 (Stanford HAI — AI Index Report 2026) in parallel. Capability and risk grow together, and both move faster than an annual rhythm. ISO/IEC 42001 therefore lays out its review structure as a cycle as well: clause 9 requires monitoring, measurement and internal audits at planned intervals, clause 10 turns the results into continual improvement (ISO/IEC 42001 — AI management systems). The standard deliberately names no interval, it requires a risk-based determination.

A pragmatic rhythm

The full set before go-live. A regression run after every change to the knowledge base. Quarterly, a full run plus an update of the set with new real questions from conversation analytics. On a model change or a larger functional extension, the full set again. A mid-market project rarely needs more structure than that.

Governance frameworks scaled to mid-market projects

Two frameworks shape the discussion. The NIST AI Risk Management Framework is organised into the four functions GOVERN, MAP, MEASURE and MANAGE (NIST AI Risk Management Framework), with MEASURE covering precisely the testing at issue here. ISO/IEC 42001 is the certifiable standard for AI management systems and works on a risk basis: applicable controls are documented and justified in a Statement of Applicability, and Annex A is explicitly a reference set rather than a mandatory checklist (ISO/IEC 42001 — AI management systems). That construction is why both frameworks can be scaled down without turning into a certification project.

Prioritise instead of cover

MEASURE 1 becomes: test the most frequent and the riskiest questions first. The rest follows if time allows. Completeness is not an attainable quantity here.
Criteria before testing

The verification control becomes: write down the release threshold and the knock-out criteria before anyone reads the first answer.
Cycle instead of date

Clauses 9 and 10 become: one run after every change, one full run per quarter, recorded. Without an auditor, but with the same logic.

The effort is smaller than its reputation, and the trend points that way regardless: the share of businesses with no responsible-AI policies at all fell from 24 to 11 percent (Stanford HAI — AI Index Report 2026). The biggest obstacle to implementation is not cost but knowledge and training gaps, cited by 59 percent (Stanford HAI — AI Index Report 2026) of respondents, ahead of resource or budget constraints at 48 percent (Stanford HAI — AI Index Report 2026) and technical limitations at 38 percent (Stanford HAI — AI Index Report 2026). Translated: it fails less often on budget than on the fact that nobody knows the procedure. A documented acceptance procedure closes exactly that gap, in days rather than quarters.

How we run acceptance in a project

We do not build assistants on the switch-it-on-and-hope pattern. Before the first configuration exists, the release criteria are fixed in writing: the score threshold across the set, the knock-out criteria for boundaries, and the rule for when escalation is mandatory. We derive the test-case set together with you from your real enquiries, not from wish dialogues. What the assistant may say at all comes from your approved knowledge base; where it should act, retrieving availability for instance, our tool control applies with clearly bounded permissions.

Release criteria and knock-out cases before the build, in writing and agreed with you.
Test-case set drawn from your business's real enquiries, versioned and frozen.
Scoring against a fixed rubric across all four review dimensions.
Two-person review for boundaries, escalation and every downgrade.
Regression run after every change to the knowledge base, recorded.
Acceptance record with score, rework items and the release decision.

An honest qualification belongs with this: testing reduces risk, it does not remove it. Even after a passed acceptance an assistant can receive a question missing from the set and give an unfortunate answer. That is precisely why boundaries, escalation to your team and the recurring run are not optional extras but part of the procedure. How a project runs overall is described in our article on going from idea to a live assistant; where we work and with which approaches is shown in our references. For an assessment of your case, a conversation about accepting your assistant is enough.

NoteSources and studies

This article is based on data from: Stanford HAI (AI Index Report 2026, on risk perception, organisational adoption, documented incidents, agent performance and the spread of responsible-AI policies), the NIST AI Risk Management Framework (AI RMF 1.0, in particular the MEASURE function and its categories, TEVV processes, testing before and during operation, and independent assessment), ISO/IEC 42001 (AI management systems: verification and validation of AI systems, acceptance and release criteria, documentation duties, Annex A controls and the clauses on performance evaluation and improvement) and the EU AI Act (Article 50 on transparency obligations for systems interacting directly with people, Article 113 on applicability from 2 August 2026 and Article 99 on the penalty range, as amended by the Digital Omnibus), together with our own project experience. Figures may differ depending on the survey, the model and the point in time. An acceptance procedure lowers the risk of faulty answers, it does not eliminate it; boundaries, escalation and recurring runs therefore remain part of operations.

Frequent questions

Why can an AI assistant not be signed off like ordinary software?

Because it does not answer deterministically. A language model picks every word from a probability distribution, so the same question produces differently worded answers. The classic target-versus-actual comparison on word-level identity therefore measures the wrong thing: two entirely different formulations can both be correct, while a familiar-sounding answer can still contain an invented deadline. On top of that comes the open input space, since a chat accepts any conceivable sentence. Acceptance therefore examines behaviour in typical situations rather than agreement with an expected text, and works with a prioritised sample instead of a claim to completeness.

How many test cases does an acceptance need, and where do they come from?

In mid-market projects an order of magnitude of around 50 test cases has proven workable. What matters is not the number but the origin: the questions should come from real enquiries, that is phone notes, the shared inbox and the search terms on your website, not from invented wish dialogues. Real enquiries are shorter, more ambiguous and less comfortable, and they hit exactly the gaps in the concept. The set is then extended with deliberate edge cases that have no pleasant answer, such as legal and pricing questions. Finally it is versioned and frozen so that later runs measure against the same yardstick.

Which four dimensions should the review cover?

First, factual accuracy, where what counts is not only whether the statement is true but whether it is covered by the knowledge base. Second, boundaries, meaning everything the assistant must not answer: legal advice, individual price commitments, binding promises on dates or scope. Third, escalation, that is whether handover to a human happens in good time and with full context. Fourth, behaviour under stress, for example with incomplete questions, topic switches, provocations or a foreign language. The boundaries dimension is best run as a knock-out criterion, because a good overall score cannot offset a problematic commitment.

Why must release criteria be fixed before the build?

Because criteria created after the build inevitably orient themselves around what the system can already do. The acceptance then merely confirms the result instead of examining it. ISO/IEC 42001 states explicitly, in its control on verification and validation of AI systems, that acceptance criteria should be defined before testing begins in order to avoid subjective decision-making, and it requires release criteria including a determination of acceptable error rates. In practice that means a score threshold across the set plus hard knock-out criteria that individually block release, regardless of the overall result.

What is a regression run and why is it necessary?

A regression run re-tests the unchanged test-case set after the knowledge base has changed. It is necessary because a new source can flip answers that were previously clean: it competes with an older source, a passage formally matches the question better than the one actually intended, or a detail overrides a general statement. An assistant that quoted the right delivery time yesterday may quote the one from a newly added old data sheet today. The NIST AI Risk Management Framework correspondingly requires approaches and documentation in its MEASURE function to continuously track existing and emergent risks.

Is an acceptance before go-live sufficient?

It is the starting point, not the conclusion. Content, offering and questions shift, and the underlying model is developed further by its provider, so a test case that passed in January can fail in July without anyone having made a mistake. Both the NIST framework, with its requirement to test before deployment and regularly while in operation, and ISO/IEC 42001, with monitoring at planned intervals and continual improvement, lay out testing as a cycle. A pragmatic rhythm: the full set before go-live, a regression run after every change to the knowledge base, and a full run each quarter with an update of the set. Risk-free operation cannot be assured on that basis, but traceable operation can.

Ask it yourself, Assistant of this website

Automated assistant

Settings