Knowledge Base Exclusions: What the Assistant Should Not Read

Anyone who gives an assistant their own website as a knowledge source gives it more than meets the eye: last year’s summer offer, a dealer price list reachable only through a hidden link, the test version of the new service page, the applicant portal. All of it is technically reachable, none of it should appear in an answer. This article sorts out which content is typically excluded, which means make an exclusion work – robots.txt, sitemap, path patterns, single sources –, how targeted probe questions show that it holds, and what the assistant says when someone asks about the excluded content anyway.

Enlarge graphic

Labels in the illustration

• On the website
• In the knowledge base
• robots.txt
• noindex
• Your patterns
• Leave out
• Service pages
• Service pages
• Applicant portal
• Test page /staging/
• Expired promotion
• Internal price list
• Public FAQ
• Public FAQ
• Only what passes
• all four barriers
• supports answers
• Check after the next run
• Ask a probe question
• aimed at the excluded content
• No excerpt supports it
• no source passage: refusal is right
• Honest refusal
• logged as a refusal, no guessing

Four barriers in front of the knowledge base

Reachable is not the same as approved

A crawler reading a website for the knowledge base sees it differently from a visitor. The visitor follows the menu, the crawler follows every link in the page text and every address in the sitemap. What surfaces is whatever has piled up over the years: landing pages of past campaigns, a folder with dealer terms, the preview of a page that never went live. For a search engine this may be harmless, because hardly anyone finds these pages. For an assistant it is not: it answers from everything in its material, and a forgotten section speaks with the same voice as the current service page. The question before the build is therefore not only what should go in – the article on building the knowledge base covers that – but also what explicitly stays out.

Seven content types that stay out

The following content types can turn up in any set of material built from a website that has grown over time. Not every one has to be excluded in every case. What matters is whether an answer drawn from it tells the visitor something correct today, and whether the visitor may learn it at all.

Three rows call for judgement rather than a blanket exclusion. With the applicant portal, the portal stays out, not the job advert: an assistant can answer questions about advertised positions from the public advert, as the article on candidate questions on the careers page shows. Knowledge for your own team belongs in an internal assistant with its own login, not in the public one. And a language version with the same content and the same maintenance standard generally does no harm; it becomes a problem when the translation lags behind or country pages quote other prices. Then two answers to the same question sit in the material, and the search decides which one is drawn.

What the law adds on internal content

For several of the seven content types there is a legal reason to keep them out, besides the operational one; the expired promotion follows further below. The applicant portal and the staff area contain personal data, and that data was collected for a specific purpose – the application, the employment relationship – not for answering any visitor to the website. The General Data Protection Regulation frames this idea as purpose limitation:

Added to this is the principle of data protection by default. A public chat is exactly the situation that the last sentence of Article 25(2) GDPR describes: whatever the assistant knows, anyone who opens the website can ask for. The provision requires measures that ensure “that by default personal data are not made accessible without the individual's intervention to an indefinite number of natural persons”. Material that reads the staff area along with everything else reverses that default.

For internal price lists and terms, another statute comes into play. Under Section 2 no. 1 (b) of the German Trade Secrets Act (GeschGehG), information only counts as a trade secret if, among other things, it is information “which is the subject of reasonable secrecy measures by its lawful holder under the circumstances” (unofficial translation). A dealer price list that a public assistant summarises on request is clearly at odds with such measures. Whether a piece of information is protected in a specific case is a matter for professional advice; operationally, the conclusion is enough that confidential material does not belong in a source from which any visitor receives answers.

robots.txt and sitemap: what they do and what they do not

robots.txt is a widely used way to keep a crawler out of certain areas. It consists of sections that begin with a name, with rules beneath them. For the knowledge base, one detail matters that is easily overlooked: under the protocol that defines robots.txt (RFC 9309), a crawler follows the section bearing its own name as soon as one exists. The general section for everyone, “User-agent: *”, then steps back for that crawler.

The sitemap is easily mistaken for a control. It is an index of the pages the operator considers important, and it serves the crawler as a starting point. A page removed from the sitemap is not thereby excluded: if a link to it appears anywhere in the page text, it will still be found. The signal on the page itself is more reliable. A page with “noindex” in the robots meta element tells every crawler that evaluates it that it does not belong in an index; for test pages and previews this is generally the right tool, because it is attached to the page and moves with it. None of these tools is access control. robots.txt is a request to crawlers that choose to honour it, and it is publicly readable itself – whoever blocks “/internal/prices/” there also reveals where the prices are. Whatever nobody outside the business may see belongs behind a login.

Custom path patterns and single sources

robots.txt belongs to the website. Sometimes, though, only your own assistant should stay out of an area while search engines keep finding it – the archive of old offers is the typical example. In robots.txt this requires a separate section for its crawler that repeats all the general blocks. Exclusion patterns attached only to the knowledge source are simpler. They generally follow the same notation as robots.txt: an asterisk stands for any number of characters, a dollar sign anchors to the end of the address. Custom patterns rank above robots.txt: whatever a pattern blocks, the crawler does not read even if robots.txt allows the area. Conversely, a pattern cannot open anything that robots.txt forbids.

• “/archive/” blocks everything under this path, such as discontinued products that stay online for search.
• “/promo-*” blocks every campaign page whose address begins with this word, including future ones.
• “/*preview=” blocks preview addresses that arise only through a parameter, wherever it appears in the address.
• “/dealers/” keeps the area with dealer terms out, even if it can be reached without a login.
• “/fr/” blocks a language version that is not maintained to the same standard as the original.

Documents take a different route. A price list as a PDF, a manual or a list of internal extensions generally do not arrive via the crawler but as an uploaded file in the knowledge base – that is, through a deliberate action. The most effective exclusion here is the least spectacular one: not adding the file. Anyone unsure whether a document should go in keeps it as a draft, which supports no answer until it is approved; a source that should no longer apply is removed instead of lingering in the material. Time-limited content also needs an end that does not depend on someone remembering: the day a promotion expires is in the calendar with a named owner, and on that day the source is removed or blocked by a pattern. How such dates fit into a maintenance plan is described in the article on keeping the knowledge base up to date.

Old offers and expired promotions

Two of the seven content types are not secrets but outdated truths. The page on the summer offer was correct once, and so was the product page of the previous model. That is exactly why they are tricky: they are cleanly written, well structured and an excellent hit for the search in the material – if a visitor asks about a discount, the search may rank the promotion page above the current pricing page, for instance because it contains the word discount more often. How an assistant handles vouchers and discounts while they are valid is covered in the article on vouchers and discounts in chat. For the time afterwards: if the assistant presents a price benefit that has ended as valid, that is more than a service glitch. The German Act against Unfair Competition expressly lists the price benefit among the circumstances about which a commercial practice can mislead:

Checking that an exclusion holds

An exclusion that has not been checked is an assumption. The test is simple: a question aimed precisely at the excluded content. If the assistant says it has nothing on the matter, that speaks for the block but does not yet prove it: a question using different words can also miss content that is still there. The proof becomes reliable with a marker word that appears only in the excluded content. If the assistant gives details instead, the content is still in the material – via another path, a second copy or a source nobody thought of.

• At least one question per excluded area, phrased the way a visitor would ask it, not the way an auditor would.
• A question about a detail that appears only in the excluded content: the dealer condition, the promotion code, the name from the rota.
• A unique marker word on a test page that occurs nowhere else. If it shows up in an answer, the block leaks.
• The same question in the second language as well, if language versions were excluded.
• A cross-check on permitted content next to it, so that a block that is too broad gets noticed.

When someone asks anyway: decline honestly

Excluded content is asked about anyway. Applicants want to know the status of their application, dealers ask about their terms in the public chat, regular customers about last year’s promotion. The right answer in these cases is neither a paraphrase nor a guess, but a clear refusal. The precondition for this is technical: an assistant that may answer only from excerpts of its sources generally has no supporting excerpt on an excluded topic – and therefore nothing from which it could build a sourced answer.

How a refusal that helps should sound, how the route to a person is offered in the same reply and how refusals can be measured is covered in the article declining instead of guessing. Exclusions add one thing: the question here often aims at content that does exist but is not meant to go outside. Three additional rules then apply:

• Confirm nothing that is not public. “I have nothing on that” is better than “Dealer prices are internal” – the second sentence reveals that they exist.
• Build no substitute from neighbouring content. A question about the dealer price is not answered with the consumer price as if they were the same.
• Name a route only from public information. If the material states how applicants learn their status or who looks after dealers, the refusal can point there. If no such statement exists, the refusal stands on its own – the route is not taken from the excluded content.

Every such refusal is also a signal: if refusals are counted separately from ordinary answers, the analysis shows which excluded topics are actually asked about. One of two decisions follows: the exclusion is right and the refusal remains the answer – or a public statement is missing, such as a sentence on how dealers find out their terms. Then that sentence goes into the material, not the internal list.

Who sets and maintains exclusions

Exclusions arise in several places in the business, and that is exactly why they get lost. HR knows that the applicant portal has to stay out, marketing knows the end date of the promotion, sales the dealer list, the web agency the test environment. What is missing is one place where all of this comes together. A single exclusion list per assistant is advisable, with one responsible person who accepts, implements and checks entries.

Three occasions call for a look at the list. A relaunch or a new area of the website shifts paths: a pattern such as “/archive/” blocks nothing after the move if the archive now has a different name, and no error message appears to show it. The launch of a promotion is the right moment to enter its end. And every new file is held against the list before it is uploaded. Who may change exclusions and who approves them belongs in the same role structure as changes to the system instruction and the knowledge base; the article on roles and approvals describes it.

What XICBOT brings for exclusions

The knowledge base in XICBOT reads websites under its own name, XICBOT-Wissensbegeher, and follows robots.txt: if there is a section for that name, it applies, otherwise the general one. The sitemap serves as the starting point of the crawl, JavaScript is not executed while reading, and other origins are not entered. A page with noindex and a source kept as a draft are not written into the searchable material at all and support no answer. In addition, custom exclusion patterns can be set per source; they rank above robots.txt and apply only to your own assistant. What stays internal is decided by you, source by source.

The assistant is instructed to answer exclusively from excerpts of these sources, and a finished answer without a single source reference used is recorded as a gap. If no excerpt supports the answer, the assistant usually says that nothing is on file on the matter; the refusal is marked on the message as a state of its own and generally recorded as a gap in a row of its own, which conversation analytics shows together with its type from the Pro package up. For checking, the answer test shows which source covers a question, and before go-live a probe set runs in which a question can carry forbidden terms – such as the marker word from the test page or the old promotion code. Which areas of your website should stay out can be decided together during setup; if you would like an assessment beforehand, get in touch.

Frequent questions