Updating an AI assistant without taking it offline

Instruction and knowledge base change while the assistant is live. How to update with version identifiers, a preview, a staged switch and a rollback.

In plain language

An assistant changes while it runs. The instruction, the knowledge, the rights or the model can be new. Each version gets a fixed name. Before the switch, you test the new version with the old questions. Then you switch in small steps. First you decide when you will stop. The way back to the old version must be quick. Practise this way back.

August 29, 2026 13 min read By XICBOT

Key points

An assistant in service consists of four layers that change independently of one another: the system instruction, the knowledge base, the tool permissions and the runtime including the model version. An update rarely touches all four, but it has to name all four.
Every version gets an identifier pointing at an immutable state: a checksum of the instruction, the documents with their date and count, the tool permissions, the model designation, plus the date, the responsible person and the reason for the change.
Before the switch, the new version runs in a preview against the same test catalogue as the old one. What is checked is not only whether behaviour changes where it should, but above all whether it stays the same where it should stay the same.
The switch happens in stages: your own people first, then a small share of conversations, then everyone. The stop criterion is written down before the first share is moved over, otherwise every anomaly gets argued away later on.
That the new version is answering is not proven by a tool's success message but by the state: the identifier in the script that is served, the same identifier in the conversation record, and a question whose answer differs between the versions.
Rolling back to the previous version is part of the procedure, not an admission of failure. It has to be possible in minutes, without the person who built the change, and it is rehearsed. Conversations, records and deletion periods do not roll back with it.

What actually changes on a live assistant

When people talk about updating an assistant, they usually mean a single thing: a new sentence in the instruction or a new document in the corpus. Technically there are four separate layers, and they change at completely different rhythms. The system instruction often stays untouched for months, the knowledge base shifts weekly, the tool permissions jump whenever a new function arrives, and the runtime occasionally changes without any action on your part because a provider develops its model version further. A single identifier for the assistant as a whole is therefore too coarse. A version is the combination of all four layers at one point in time.

That distinction is not a formality. It decides whether a later question can be answered at all. If a customer complains in October about an answer they received in September, the first question back is: which instruction applied at the time, which documents were in the corpus, which queries was the assistant allowed to make, and which model version produced the answer? Anyone who has not written those four items down together cannot reconstruct the case and starts guessing. Anyone who has them looks them up in a minute.

System instruction

Role, tone, boundaries and escalation rules. It changes rarely but bears on every single answer. A deleted sentence can remove a boundary that nobody checks afterwards, because it is still there in the concept document.
Knowledge base

Documents, sections and how they are cut. This changes most frequently and often in passing. An added file can flip an answer that was previously clean, because it competes with an older source over the same question.
Tool permissions

Which systems the assistant may query and which fields come back. This changes in bursts, usually alongside a new function. It is where you decide what becomes visible in a failure case and what does not.
Runtime

The model version, the libraries and the script that loads on your pages. Parts of it change without any action from you, which is why the state is recorded rather than assumed.

The knowledge base is the layer most often underestimated, because a change to it looks so small. Our own knowledge base holds 2,670 sections from 70 documents; that figure applies to the corpus of the assistant answering on this site. In a corpus of that size, swapping in a single price list is not a minor intervention but a shift in the competition over every question that list touches. You cannot see it from the outside, which is exactly why it deserves the same treatment as a change to code. Which content belongs in such a corpus at all is covered in our article on building a knowledge base.

Why every change needs an identifier

An identifier is not a version number for its own sake. It is a pointer to a state that no longer changes. Only then do three questions become answerable that come up regularly in operation: what was live while this conversation was running? What has changed since the last state that ran without incident? And exactly what do you fall back to if the new version turns out to be worse? Without an identifier all three are questions of memory, and a team's memory for this sort of thing lasts about two weeks.

The effort is small as long as it happens while you change something and not afterwards. Anyone who assigns the identifier only once it is needed reconstructs a state that never existed in that form. In practice one entry per version is enough, recording the four layers plus the three details nobody remembers later: when, by whom, and why. The reason is the most valuable line, because it stops a change that was taken back from returning unchanged two weeks later. None of it has to be publicly visible; for our own product we keep the history in the changelog.

A checksum or version number of the system instruction, not merely the date it was last edited.
The list of documents in the knowledge base with their date, section count and origin.
The tools and their permissions: which query is allowed and which fields come back.
The designation of the model version used to produce the answers.
The date, the responsible person and the reason for the change, in one sentence.
The identifier of the version to fall back to if the new one does not hold up.

A version without an identifier is a claim

If you cannot say which version was answering at two o'clock on Tuesday, you can neither trace nor refute a complaint from Tuesday. The identifier costs one line while you make the change, and weeks later it decides whether a query becomes a glance at the record or an investigation.

A preview against the real test cases

A preview is the new version running under the same code, at an address only your team knows. It is quick to set up and still regularly misused: somebody asks three questions, the answers sound good, the version goes live. That tests whether the change can do what it was built for. The more important half stays untested, namely whether everything else has remained unchanged. So the preview is run against the same test catalogue the assistant was accepted with, unchanged. How that catalogue comes about is described in our article on acceptance with test cases.

What helps here is a measure that can be compared across versions and does not depend on the taste of the person reviewing. Source binding works well for that: the share of answers that come with a reference to their source. In our own measurement series 42 of 48 answers carried a source reference; the count covered questions whose answer is in the knowledge base, and six carried no source reference. If that share drops noticeably in the preview, the new instruction has given the model more freedom than intended, or the corpus no longer carries the questions. Either way, the figure shows it earlier than the individual answers do. Why binding answers to sources matters so much is explored in our article on hallucinations and the knowledge base.

What the preview shows	What only live operation shows
Whether the new answer is factually right	How visitors actually ask when nobody guides them
Whether the old test cases still pass	Whether a cache is still serving the old answers
Whether the AI disclosure survives the change	Whether the script really reloads on every page
Whether boundaries and escalation still hold	How the answer path behaves under load
Whether source references still come along	Whether real data takes different paths than test data
Whether two sources contradict each other	Which questions arrive that nobody anticipated

Switching over in stages instead of all at once

Moving from one version to the next is not a switch, it is a share. First the new version answers only your own people, then a small part of the conversations, then everyone. The point of this staging is not caution as an attitude but the plain fact that the first real conversations reveal things no catalogue contains. Keeping the share small limits the number of people who receive a poor answer to a size you can still follow up on personally the same day.

The new version answers your own people first, in the same system and not on a playground.
Then a small share of conversations, selected at random rather than by time of day.
The stop criterion is written down beforehand: which anomaly pulls the share back immediately.
The share grows in steps, between which somebody actually reads the conversations instead of only looking at figures.
Every step is noted with the time, the share and the identifier, even when nothing stands out.
The previous version is kept intact until the final step has run quietly for several days.

One rule is easily overlooked: a conversation in progress stays on its version. Switching the instruction mid-dialogue creates a break the visitor notices even if they cannot name it. The form of address changes, a boundary suddenly applies differently, a route offered earlier disappears. The switch therefore takes effect from the next conversation onwards, and the identifier is bound to a conversation with its first message. That matters for the evaluation too: it is the only way to compare two versions side by side without the conversations blurring into each other. What such evaluations can tell you is shown in our article on conversation analytics.

How you know the new version is really answering

Between written and delivered there are three layers in an assistant, and each of them can be left on the old state on its own without any error being reported anywhere. That is the most common reason for the situation where somebody has rolled the change out, the assistant answers exactly as before, and two hours of searching in the wrong part of the system begin. The search stays short if you know the three layers in advance and prove each of them separately.

The script in the browser

It loads from the visitor's cache or from a server in front of yours. A new state on the server does not mean a returning visitor receives it. The proof is the identifier in the file actually served, fetched with the cache bypassed.
The answering service

New code takes effect only once the running process has loaded it. A changed file on disk is no proof that the service reads it. The proof is a restart with a timestamp and an answer that names the identifier.
The knowledge-base index

Changed documents take effect only once the search index has been rebuilt. Until then the new instruction answers from old sections, which is particularly hard to spot. The proof is a question that can only be answered from the new document.

From those three layers comes a simple proof that needs no tooling. First: the file served in the browser names the version identifier, checked with the cache bypassed. Second: the record of the next conversation names the same identifier. Third: a question whose answer demonstrably differs between the old and the new version returns the new answer. That third question should be agreed before the switch rather than looked for afterwards, otherwise you reliably find a question whose answer was right either way.

Rolling back to the previous state

The rollback is the property that makes fast switching possible in the first place. Someone who knows they can take a change back in minutes decides on the merits instead of hesitating. Someone who does not know postpones every update until enough of them have piled up to make a project out of it. The difference between those two ways of working is not one of diligence, it is simply whether the previous state exists in full or would have to be rebuilt from parts.

The previous state exists in full and is not reassembled from parts.
The rollback takes minutes and does not require the person who built the change.
The instructions for it sit somewhere that someone looking for them the first time will find.
After the rollback, the same three proofs are used to check that the old version really is answering.
Conversations, records and deletion periods do not roll back; they belong to the time, not to the version.
The reason for the rollback is noted, otherwise the same change returns unchanged two weeks later.

A rollback is something different from a fallback mode. The rollback changes the version while the system is fully operational. Fallback mode kicks in when part of the chain does not answer at all, and deliberately reduces what the assistant still takes responsibility for. Both routes need a decision made in advance, and the same person should be allowed to trigger either. How both routes can be rehearsed before go-live is described in our article on acceptance testing.

When a maintenance window is needed after all

Not every change can be run with overlap. Moving data to different storage, switching to a model version with a different context length, or restructuring the search index are interventions where two versions cannot work cleanly at the same time. Saying so openly is more honest than promising a changeover nobody notices. What matters then is not whether there is a window but what it looks like.

A usable window sits in the low-demand period you know from your own evaluation rather than from gut feeling. It is announced in the chat window before it starts, it appears on the status page, and the route via the form stays open throughout. What must never happen is a chat window that keeps answering pleasantly while half the chain is missing. An assistant that points to the form in that situation costs less than one that guesses.

An announced window is not a disruption

The same changeover is experienced very differently: without notice it is an outage; with a message in the chat, an entry on the status page and an open form it is an appointment. The difference does not lie in the technology but in whether the visitor knows where they stand at that moment and which route to take instead.

What the identifier is worth in legal terms

Versioning is a working tool first of all, but it touches two duties directly. The first is the accountability principle of the General Data Protection Regulation. It requires not only that processing be lawful but that the controller be able to demonstrate it. If you cannot say which instruction and which tool permissions applied at a given time, you cannot give any dependable account of the processing during that period either.

The controller shall be responsible for, and be able to demonstrate compliance with, paragraph 1 ('accountability').

Regulation (EU) 2016/679 (GDPR), Article 5(2)

The second duty concerns disclosure. Under Article 50(1) of the AI Act, systems interacting directly with people must be designed so that those people learn they are talking to an AI. For an update that means something very concrete: a newly written system instruction can lose the sentence carrying that disclosure without anyone noticing. So a test case for the disclosure belongs in every version, not only the first. It costs two minutes. Our article on the disclosure duty sets out the obligation.

that the natural persons concerned are informed that they are interacting with an AI system

Regulation (EU) 2024/1689 (AI Act), Article 50(1) (extract)

A third situation deserves attention even though it arises less often: a change that gives the assistant access to new fields from a connected system is more than an update. It widens which data can enter a conversation at all. Changes like that do not belong in the same drawer as a new paragraph in the instruction text; they are reviewed beforehand, and where there is doubt, reviewed legally. Which permissions a tool receives and how tightly they are cut is covered in our article on tool control.

How we handle an update

We treat a change to the assistant like a change to an application, because that is what it is. The four layers are bundled into one version and given an identifier before anything is switched. What the assistant may say comes from your approved knowledge base; what it may query is set out in tool control with clearly bounded permissions. Both are versioned together, because a new instruction on an old corpus is a different thing from the same instruction on a new one.

Every version bundles instruction, knowledge base, tool permissions and model state under one identifier.
The preview runs against the unchanged test catalogue from acceptance, not against three spot checks.
The switch happens in steps, with a stop criterion fixed beforehand and the time recorded.
The new version counts as delivered once the script, the record and a distinguishing question confirm it.
A rollback to the previous version is possible in minutes and is rehearsed regularly.
If a maintenance window is unavoidable it is announced, and the route via the form stays open.

An honest qualification belongs with this: a versioned procedure lowers the risk of a failed update, it does not remove it. A version can pass every test case and still receive a question in its first week that nobody thought of. That is exactly why the staging, the rollback and the reading of the first real conversations are part of the procedure rather than decoration. If you want an assistant set up from scratch, the route is described in our article from idea to live assistant; for an assessment of your case, a conversation is enough.

Frequent questions

Does the chat have to be switched off for an update?

In most cases, no. The new version is first checked in a preview, then released to your own people, then to a small share of conversations, and only at the end to everyone. Meanwhile the existing version keeps answering, and a conversation in progress stays on the version it started with. A window only becomes necessary where two versions cannot work cleanly at the same time, for instance when moving data, switching to a model version with a different context length, or restructuring the search index. In that case the window belongs in the low-demand period, is announced beforehand and leaves the route via the form open.

What belongs in a version identifier?

Four layers and three details. The layers are the system instruction as a checksum or version number, the list of documents in the knowledge base with their date and section count, the tools together with their permissions, and the designation of the model version. The three details are the date, the responsible person and the reason for the change in one sentence. On top of that comes the identifier of the version to fall back to. The reason is the line most often missing and worth the most: without it, a change that was taken back returns unchanged a few weeks later, because nobody remembers why it did not work at the time.

How do you check a new version before it goes live?

In a preview that runs on the same code and is reachable internally only, using the unchanged test catalogue from acceptance. Three questions you think up on the spot only test whether the change can do what it was built for. The more important half is whether everything else has stayed the same. A measure that can be compared across versions helps, for example the share of answers carrying a source reference. The preview does have limits, though: load, real data from connected systems and the caches of live operation are usually absent there. What the preview cannot show, the staged switch will.

How do I know the new version is really the one answering?

From three independent proofs, not from a tool's success message. First, the file loaded in the browser names the new version identifier, fetched with the cache bypassed. Second, the record of the next conversation carries the same identifier. Third, a question agreed in advance, whose answer differs between the old and the new version, returns the new answer. The background: the script in the browser, the answering service and the search index of the knowledge base are three separate layers, any one of which can stay on the old state without an error being reported anywhere.

How fast does a rollback have to be, and what does not roll back with it?

It should be possible in minutes and work without the person who built the change, otherwise it is unavailable at the decisive moment. The precondition is that the previous state exists in full and does not have to be reassembled from parts. What does not roll back is the data: conversations created under the new version remain, as do records, and deletion periods continue to run independently of the version. After the rollback, the same three proofs are used to check that the old version really is answering. It is also worth rehearsing the route once a quarter on a quiet morning.

How often should an assistant be updated at all?

More often and in smaller steps than most businesses expect. Small changes are easier to check, their effect can be attributed, and a rollback costs little. A package of twenty bundled changes, by contrast, is hard to untangle once one of them causes trouble. A workable rhythm: changes to the knowledge base as soon as the occasion arises, each with a run against the test catalogue; changes to the system instruction collected and made less often, because they bear on every answer; the model version only deliberately and never in passing. More important than the interval is that every change gets its identifier.

Ask it yourself, Assistant of this website

Automated assistant

Settings