Conversation Memory: What an AI Assistant Keeps

An assistant that starts over after the second sentence is not help, it is a loop. Someone who asks about the delivery time and then adds whether that also applies to the larger variant expects the word "that" to be understood. Equally, an assistant must not keep everything that has ever been typed into a chat window. Between these two failures sits what is called conversation memory in practice: a deliberately built chain of what travels verbatim, what is condensed, what survives a page change, and what disappears at the end of the session. This article describes the four stages, the technology behind them, the legal limits and the test cases that let you check the result before go-live.

Enlarge graphic

Labels in the illustration

• Current turn
• The most recent questions and answers travel verbatim into the next turn.
• Every back-reference in the dialogue depends on them.
• kept verbatim
• as long as the window allows
• Older turns
• What came before is condensed into a few sentences: the request, named
• objects, constraints, open points and promises already made.
• condensed
• a form, not a free essay
• Page change
• A session identifier holds the conversation together beyond the click.
• The transcript itself sits on the server, not on the device.
• identifier says nothing
• about the person
• End of session
• The transcript expires after a fixed period, with no manual clean-up.
• Recognition on the next visit only after consent.
• forgetting
• is a built function
• Deliberately excluded: payment data, credentials, health information.

Conversation memory of an assistant

What memory means for an assistant

The term is misleading because it carries an everyday image with it. A language model does not remember the previous conversation the way a person remembers a phone call from this morning. At every single turn it is handed a text and continues it. If the transcript so far is inside that text, the assistant can refer to it. If it is not, it does not exist for the assistant. Everything that looks like memory is therefore a decision made in the system in front of it: who puts what into that text, in which form, and for how long.

That decision is taken in four places, and each of them can be set independently. The current turn carries the most recent questions and answers verbatim. A condensation summarises what came before. A session store holds the conversation together across a page change. And a lasting store, if there is one, connects two separate visits. The further back you go along this chain, the greater the benefit for customers and the stricter the requirements for legal basis, retention period and deletion.

The context window: the workspace of a single turn

The text handed to the model at every turn has an upper limit. It is measured in tokens, that is, in fragments of words, and it caps system rules, knowledge excerpts, conversation transcript and the answer being generated all together. Picture the window as a desk: whatever lies on it can be used, whatever has fallen off is gone. The desk is large enough for a detailed consultation, but it is not infinite, and several parties occupy it at the same time.

This competition is easy to underestimate. An assistant that answers from a knowledge base pulls in matching excerpts for every question. An assistant with connected tools also places the descriptions of its functions and the results of their calls alongside, as described under tool control. In a long conversation, transcript, knowledge excerpts and tool responses share the same space. Enlarging the memory almost always shrinks something else. The workable rule is therefore not "as much transcript as possible" but "as much transcript as necessary, the rest condensed".

When the transcript grows: condense instead of truncating

Once the transcript grows longer than the space allotted to it, there are three routes, and two of them are poor. The first is hard truncation: the oldest turns drop out. That is cheap, and it destroys exactly the details that sit at the start of a conversation, namely the actual request and the constraints. The second is dropping the answers while keeping the questions. That fails too, as soon as a question refers back to an earlier answer. The third route is condensation: older turns are summarised into a few sentences while the most recent turns stay verbatim.

Condensation is the only route that preserves both, but it is not a side issue. What may enter the summary is defined in advance, otherwise a second source of error appears: an assistant that freely summarises its own transcript can shift a detail in the process, and the shifted detail then sits in the context as fact. For the same reason that answers are tied to sources, as described in the article on hallucinations and the knowledge base, the summary is a tightly defined form and not a free essay.

• The request in one sentence, worded as the person put it, not as the assistant reinterpreted it.
• Named objects with their identifiers: article number, contract number, model designation, location, appointment.
• Constraints that narrow the answer: deadline, quantity, dimensions, language, responsibility, preferred channel.
• Open points: what the assistant has already asked about and what has not been answered yet.
• Promises made during the conversation, such as a call back or a handover to a person.
• Not included: pleasantries, repeated phrasings and anything without bearing on how the conversation continues.

The page change: why the conversation should not restart

The most common break in everyday use has nothing to do with models. Someone asks about a product in the chat, clicks through to the product page, comes back, and the window is empty. From the visitor's point of view that is a defect: she did not end the conversation, she only clicked. Technically it is the normal case if nobody does anything about it, because a page change rebuilds the page and with it the chat window. For the conversation to survive, it needs an identifier under which the transcript can be found again, and a place where it sits in the meantime.

The two should be considered separately. The identifier is a short, random string that says nothing about the person. The transcript itself sits either in the browser or on the server. Server-side storage is usually the cleaner variant: the content of the conversation never leaves your own area of responsibility, the deletion period is enforceable in one place, and only the identifier remains on the device. How this fits into an existing site is shown in the article on embedding on a website and in a shop.

Across sessions: what may be stored

The moment an assistant knows on the next visit who is writing, a line has been crossed that is easy to overlook in everyday use. Holding a running conversation together and recognising a person across visits are two different processing operations with two different assessments. Storing information in the end user's terminal equipment, or accessing information already stored there, is only permitted where the end user has consented on the basis of clear and comprehensive information (Sec. 25 (1) TDDDG). This applies regardless of whether the stored information is personal data.

The second paragraph of the same provision exempts what is strictly necessary in order for the provider of a digital service to provide a digital service expressly requested by the user (Sec. 25 (2) no. 2 TDDDG). Anyone who opens the chat window and asks a question expressly requests that service. The session identifier, without which the conversation breaks off at the first click, falls under it. An identifier that links today's visit to a visit three weeks from now does not, because it is not necessary in order to answer today's question. The line in the individual case is drawn in the article on GDPR, data protection and hosting.

For the content of the conversation the General Data Protection Regulation applies on top, wherever that content sits. Anyone handling an enquiry relies on taking steps at the request of the data subject prior to entering into a contract (Article 6(1)(b) GDPR). Purpose, recipients and retention period must be communicated at the time of collection (Article 13 GDPR), and by default only data necessary for the specific purpose may be processed (Article 25(2) GDPR). For a conversation memory that means, by default: short, tied to a purpose and without recognition. Anything beyond that is a deliberate, justified and documented decision.

What an assistant deliberately forgets

Forgetting is not a shortcoming but a function, and it is built like any other. Some details end up in a chat window without anybody wanting them there: an account number somebody sends along just in case, a password in the wrong field, a diagnosis inside the description of a problem. An assistant cannot prevent such things from being typed. It can prevent them from ending up in a summary, moving into a log, or surviving beyond the session. With XICBOT this goes as far as follows today: the conversation is stored in full and goes to the language model unabridged; in the view for your team, email addresses, phone numbers, IBANs and postal addresses are masked, and only the owner role can open the plain text, stating a reason that is logged.

• Payment data: full card numbers, check digits and bank details should be detected, masked in the transcript and never carried into the condensation.
• Credentials: passwords and one-time codes should be neither read back nor stored. The assistant should point to the proper route and discard the input.
• Health information and other special categories within the meaning of Article 9(1) GDPR should stay outside any lasting store wherever possible.
• Free text without a purpose: whatever is not needed to handle the request never enters the condensation in the first place.
• Third-party details: names and contact details of people not taking part in the conversation do not become an attribute of the transcript.
• The assistant's assumptions about the person: assessments of solvency, mood or intent do not belong in a stored transcript.

The reason sits in the data minimisation principle: personal data must be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (Article 5(1)(c) GDPR). For a conversation memory that is a remarkably practical requirement, because it turns the question around. Not "what could be kept" but "what is needed to continue this conversation". Everything else falls away, and it falls away where the condensation is created, not later in an overnight clean-up job.

Deletion periods, access and withdrawal

A memory without an expiry date is an archive. Data subjects have the right to obtain the erasure of personal data concerning them without undue delay where the data are no longer necessary for the purposes for which they were collected (Article 17(1) GDPR). Complying with that is considerably easier when the periods are set per storage location from the outset and expire automatically. A period that somebody has to enforce by hand stops being a period in its third month.

• Window content: ends when the page is closed. Nothing to delete, because nothing was stored.
• Session transcript: a short period after the last turn, so a returning visitor can continue without a daily archive building up.
• Case transcript: for as long as the enquiry is being handled, plus a brief run-off for follow-up questions. After that the case notes remain, not the raw log.
• Analysis: without personal data and without free text, otherwise a lasting transcript appears through the back door. What can sensibly be analysed is shown under conversation analytics.
• Recognition: only with consent, with a withdrawal path as simple as giving it, and with immediate effect on the transcript already stored.

Two points are regularly forgotten. First, access must be possible: anyone who asks has the right to obtain information about the personal data being processed (Article 15(1) GDPR), and a conversation transcript is part of that as soon as it can be attributed to a person. Second, a withdrawal has to reach the memory as well. A withdrawn consent after which the assistant still knows the name on the next visit is a finding, not a detail. Both belong in the same acceptance procedure as the substantive answers. The conditions for processing and storage location are set out under data protection and hosting.

Memory at the handover to a person

Conversation memory shows its value most clearly at the moment the assistant stops. If a request is handed over to a person and that person opens with "So what is this about?", the previous quarter of an hour was lost for the customer. The handover therefore needs exactly what a condensation produces: the request, named objects, constraints, open points and the promises already made. The colleague should see not just a transcript of the chat but its core, with the option to expand the exact wording when needed. With XICBOT your team receives the reason for the handover, the known contact details and the conversation history word for word; XICBOT does not create a condensation.

The same condensation carries two further cases. It carries the call back on the next working day, where the caller should know the state of play without reading the chat. And it carries the channel switch, when a conversation starts in chat and continues by message. In both cases the quality of the summary decides whether the handover feels like progress or like starting again. How a clean transition works is described in the article on handover to a human agent, and where it comes too late can be read off the conversation analytics.

What a long memory costs

More transcript in the context is not free, and it is paid for in two currencies. The first is time. Every additional piece of text has to be transmitted and processed before the first character of the answer appears. How much headroom there is depends on the starting figure: in our own measurement series the time to the first character of the answer was 1.3 s at the median, measured across 64 questions to our own assistant, not to customer projects. The raw values and the conditions are set out under measurements. A conversation that drags the entire transcript along at every turn moves away from that figure with each further question.

The second currency is accuracy, and it is the more unpleasant one. A very long context dilutes what matters: among thirty turns of incidentals, the decisive constraint sits there exactly like a passing remark, and the answer picks up the wrong one. Contradictions make it worse. If someone first says "for the small flat" and later "actually for the office", the arrangement inside the context decides which detail wins. A condensation that explicitly marks the current state as the current state solves exactly this problem. Simply appending every turn does not.

Testing the memory: test cases before go-live

Whether a conversation memory holds up cannot be read off the system text, only off conversations. The check therefore needs test cases that hit exactly the transitions where something gets lost: the back-reference across several turns, the page change, the long conversation beyond the condensation threshold, the contradiction and the change of subject. Each of these cases is written down as a conversation, together with the answer that counts as a pass. An answer that happens to be right but uses the wrong state does not count as a pass.

• Back-reference: turn five refers to a list from turn two. A pass is an answer that names the intended item instead of asking back.
• Page change: a subpage is opened mid-conversation and then left again. A pass is a window that shows the transcript so far.
• Condensation threshold: a conversation is carried past the configured threshold and then asked about a detail from the first turn.
• Contradiction: a detail is corrected during the conversation. A pass is an answer that uses the new state and names the correction.
• Change of subject: after one request is settled, a second one begins. A pass is an answer that does not carry the old context into the new request.
• Forgetting: a card number and a password are typed in. A pass is a stored transcript in which both are masked or absent.
• Deletion: after the period has run out, the store is inspected. A pass is a store without that transcript, with nobody having intervened by hand.

This list belongs in the same acceptance procedure as the substantive test cases, and it is run again after every change to system rules, condensation threshold or retention period. The effort is small because the cases are short. The benefit is large because memory failures are rarely reported as failures in live operation: a visitor who had to explain the same thing twice does not write a complaint, she closes the window. It only becomes visible in the analysis of conversations, and there later than it should. How to set up such an acceptance procedure is described in the article on test cases and go-live. Which retention periods you set for conversations and how deletion works is shown on the page on privacy and retention at XICBOT.

Frequent questions