How knowledge and answers are built

Under Knowledge (app.xicbot.de/wissen) you store the sources your assistants answer from. This page explains what happens next: how a website is read, how text becomes chunks, how search finds the chunks that fit a question, and the rules that turn them into a cited answer – or into none. The fields, limits and messages of the area itself are described on the Knowledge page.

The path at a glance

1. 1

   Step 1

   Reading

   A text, file or website is read and turned into plain text. For a website, XICBOT follows links and the sitemap but stays on the same site.

2. 2

   Step 2

   Splitting

   The text is split into chunks, and each chunk is assigned its language. A chunk is the unit that later appears as a citation in an answer.

3. 3

   Step 3

   Switching over

   The new state is built next to the old one and only takes effect at the end, for the whole source at once.

4. 4

   Step 4

   Enriching

   After that, a background job gives each chunk keywords, up to four questions built from its own words and fixed templates, and a feature spread used to compare it with questions. Until then, only word search finds it.

5. 5

   Step 5

   Searching

   A question is matched against the chunks by word search and by feature comparison; both hit lists are then merged.

6. 6

   Step 6

   Answering

   The best hits are turned into an answer that names its passages by number. Under Check an answer and in the API, apart from a curated pair, only an answer that is sufficiently covered and cited is returned.

7. 7

   Step 7

   Checking

   In the widget, hard details in the answer such as prices or deadlines are checked against the material that was found. Anything not found there is replaced or marked.

How a website is read

A website source starts at the address you entered. Only what lives under the same combination of scheme, host and port is read: a subdomain or a switch from http to https counts as a different website. XICBOT identifies itself as XICBOT-Wissensbegeher when fetching. Page count, file types, timeout and sizes are listed under Website crawling; this section explains how the pages are collected.

• Order: the start address first, then the addresses from the sitemap, then addresses already in the index from an earlier run – with Rebuild completely these are left out. Links found while reading a page are added at the end. Since at most 25 pages are read per run, this order helps decide which pages make it into the knowledge base.
• What counts: every fetched page counts towards the 25, including one that has not changed since the last run. A page that yields no text does not count.
• Queue: new links are only queued while fewer than 100 addresses are open. Beyond that they are merely counted, up to 1,000.
• Which links apply: only the href of an <a> element. Links inside comments and inside script, style, textarea, title and template elements do not count, whereas links inside noscript do. mailto:, tel:, javascript: and data: are skipped, as are links to files such as PDF, ZIP, images, video, audio, CSS, JavaScript, XML and icons. For a link with #, only the part before it counts: /services#pricing leads to /services, while a pure jump link such as #top is skipped. XICBOT resolves relative links against the address under which the page was delivered: on /services/, contact leads to /services/contact, but on /services it leads to /contact; ../contact goes up one directory. If the page carries a <base> element with href, its address becomes the base, as in the browser, itself resolved against the delivered address: on /services with <base href="/services/">, contact leads to /services/contact. The first such element applies that is not inside a comment or inside a script, style, textarea, title, noscript or template element; an empty value or a base with data: or javascript: leaves the delivered address in place. If the base names another scheme, such as ftp:, the page’s relative links lead to no address; only links starting with http:// or https:// then count.
• One address, one page: anything from # onwards, a trailing slash and a trailing index.html, index.htm or index.php are removed. The query part from ? onwards stays, so /list?page=2 is a page of its own.
• Unchanged pages stay: if a page is already in the index, XICBOT sends the state of the last fetch (ETag and last-modified date). If the server answers 304, the page keeps its chunks as they are. If the page is delivered again but its text is identical, the chunks also stay. Rebuild completely asks without that state and splits every page again. A page without chunks – such as a withheld draft – is also requested without that state and read in full.
• Title: the content of <title>, otherwise the page address. Plain text pages (text/plain) use their address as title. The title is shown as the document name in a citation.

If a run ends at the 25-page limit while addresses were still open that had been neither visited nor blocked, the website counts as truncated. The source then shows Only the first {n} pages of this website were read. and {n} links were still open when the crawl stopped. The second number tells you how many addresses were open, not how many pages are missing. If that count is missing, it reads How many links were still open was not measured. See also Your sources.

robots.txt and excluded paths

• robots.txt is read once per run from the root of the website, before the first page is read. If the server answers there with a server error (5xx), the run ends with The website's robots.txt could not be reached because of a server error. Until it responds again, no page of this website is read. If no response arrives – for example because no connection can be made or the server does not answer in time – it ends with This address could not be reached. In both cases the existing index stays in place. If robots.txt cannot be read for any other reason – for example because the server answers with 4xx or too many redirects follow – or if it is empty, XICBOT applies no rules from it. How XICBOT interprets its rules is described under robots.txt.
• An address is blocked if a pattern from Excluded paths matches or robots.txt blocks it. Your patterns rank above robots.txt: an Allow there does not lift your exclusion.
• A pattern that contains :// is compared with the full address, any other pattern with the path including the query part. A pattern matches from the start; * stands for any number of characters, $ anchors to the end. How to enter patterns and their limits are covered under Excluded paths.
• Every address is checked before it is fetched, including every address from the sitemap. The target of a redirect is checked after the fetch: if it is blocked, the page is discarded.

Sitemap

• XICBOT uses the Sitemap: lines in robots.txt as long as they point to the same website. If there is no such line, XICBOT tries /sitemap.xml.
• At most six sitemap files of up to 5 MB each are read. Gzip-compressed files are unpacked, and a sitemap index (<sitemapindex>) is followed up to that limit. Once 100 addresses have been collected, XICBOT reads no further file.
• Only <loc> is evaluated. Entries on other websites and entries pointing to files such as PDF, ZIP, images, video, audio, CSS or JavaScript are skipped.
• The sitemap is how XICBOT reaches pages that no link from the start page leads to: without it, such pages are not found.

Drafts and future-dated pages

How XICBOT recognises that a page has not been published yet is described under Withheld pages. For answers, the following applies:

• A draft receives no chunks and therefore appears in no answer.
• If a page that has already been read becomes a draft or gets a future date, the next run removes its chunks.
• In addition, search checks for every question whether a document is published. A page whose date was moved into the future after it was read therefore disappears even before the next run.
• If a source contains nothing but such pages, the run fails with This source contains only drafts or content dated in the future.
• Through the API a text or file source can be created with draft or a published_at in the future. It receives no chunks, and its run fails with the same message. Re-read exists only for websites; for other sources the API answers 422 with website_only, so such a source is never picked up later. Create it only once the content is meant to apply. For websites the API rejects both fields, because there the page itself decides.

How text is extracted

• HTML: script, style, noscript, svg and template are removed together with their content, as are the page head and the nav, header, footer and aside areas. The end of p, div, li, tr, h1 to h6, section and article and every br become line ends so that sentences stay apart.
• Comments and control characters: HTML comments are removed from every text – they carry no content for any reader but are a classic vehicle for smuggled instructions. Control characters are dropped, and repeated spaces and blank lines are collapsed.
• Character set: a byte order mark (BOM) for UTF-8 or UTF-16 is honoured; without one, UTF-8, ISO-8859-15, ISO-8859-1 and Windows-1252 are detected. Everything is stored as UTF-8.
• PDF is only read if it has a text layer. Optical character recognition for scans deliberately does not run: text read from a scan is wrong in places, and a wrong answer with a citation is worse than a rejected file.
• Word: from .docx and .docm, the main body is read together with footnotes, endnotes, headers and footers; text marked as deleted in tracked changes is left out. A .doc file that is really a renamed .docx is recognised and read the same way. Older .doc and .dot files are accepted if their text can be extracted; otherwise the file is rejected.
• Text, Markdown, CSV and other text files are taken over unchanged; only the character set is normalised. A JSON file is read as lines of “key: value”. An HTML file (.html, .htm, .xhtml) uses its <title> as title, any other file its file name.

Which formats and sizes are allowed and how duplicate pages are handled is described under File formats and Website crawling.

How chunks are formed

A chunk is the unit that appears as a citation in an answer. You control its size per source with Chunk size and Overlap (see Editing a source). The default is 800 characters with 120 characters of overlap, for uploaded files 1,400 with 200.

• Headings first: the text is split at headings, and each heading stays with its text. A heading is a Markdown line with # to ###### of at most 120 characters, or a short line of at most 80 characters and at most 12 words that starts with a capital letter or a digit and does not end in ., :, ;, ,, ! or ?.
• Whole block if it fits: if the text under a heading is no longer than chunk size plus overlap, it becomes a single chunk.
• Otherwise at sentence ends: longer blocks are split at sentence ends (., !, ?) and at blank lines, not by character count. A chunk closes before the next sentence would push it past the chunk size.
• Overlap: the end of one chunk is repeated at the start of the next so that a sentence on the boundary is not torn apart. It is at most half the chunk size; a larger value is cut to that half when the text is split.
• Hard split: a single sentence longer than 1,600 characters or twice the chunk size – whichever is larger – is cut into pieces of the chunk size regardless of sentence boundaries.
• The heading is stored with each of its chunks and appears in the citation next to the document name.

Keywords, questions and language per chunk

• Each chunk receives up to 14 keywords: its main words by frequency, with the words of the heading weighted three times. Bare numbers such as “2026” do not become keywords.
• Up to 4 questions are added. XICBOT first takes questions that already appear in the text (10 to 160 characters, ending in ?). After that, questions are built from a heading of 3 to 80 characters using fixed templates – depending on the content, for instance “What does … cost?”, “How long does … take?”, “Which times apply to …?”, “Where can I find …?” or “What applies to …?” – and only for German or English text.
• Keywords and questions are built from the chunk itself and fixed templates; no language model is involved. The keywords count in the word search. The questions only help word search pick a chunk as a candidate; they do not add to its score.
• The language of a chunk is determined from its first 600 characters, see Language.

The feature spread

Alongside word search, XICBOT compares question and chunk through a feature spread. Every text is distributed over 256 buckets built from whole words (weight 1), word pairs (0.6) and three-character sequences (0.35). Search therefore still matches when question and chunk share no whole word – through shared word parts, inflected forms and spelling variants. The method is explicitly neither neural nor semantic: it does not connect synonyms that share no word parts. When to rebuild the spread is described under Feature spread.

• The feature comparison reads at most 1,500 chunks per question, higher-weighted ones first and, at equal weight, older ones first. Anything beyond that is reached by word search only.
• A chunk counts as a hit there from a similarity of 0.08.
• Chunks without a feature spread – for instance while the background job is still running – are left out of the feature comparison.

How a new state takes effect

• A run writes the new state next to the old one. XICBOT switches over only at the end, for all chunks of the source at once. A half-built state cannot answer.
• If the run fails, the previous state remains valid – even if the run produced no chunk at all.
• A run that changed nothing does not create a new version.
• The last five versions of a source are kept. Restoring an older version discards all newer ones. Label, address and character count are not part of a version and stay as they are. How to use this is described under Versions.

How search finds chunks

For every question, XICBOT searches in two ways and merges both hit lists into one ranking. It applies equally to Check an answer, the API, conversations in the widget and the probe set.

Use the probe set to measure whether search finds the right document for your typical questions. It only queries search, generates no answer and shows the rank at which the expected document lands.

Coverage: does the hit carry the question?

Coverage is the share of meaningful words in a question that appear in the text of the single best hit. The best hit counts, not the sum of all hits: two chunks that each carry half the question are two topics, not one source. The threshold is 0.34, a little over a third of the words.

• Under Check an answer and in the API, there is no answer below the threshold.
• In the widget, the coverage of every answer is measured and stored but does not block the answer. It is not shown per answer, not even under Conversations; as a mean, it makes up the Question coverage figure in Analytics.
• If no search ran at all – for example because a request was blocked, the conversation had reached its length limit, a person had taken over or a curated pair answered – coverage stays empty and does not enter the figure as 0.
• Suggested follow-up questions in the widget must reach the same threshold, see below.

How a cited answer is produced

This path is used by Check an answer under Knowledge and by the API POST /api/v1/knowledge/ask. The page sums up the rule as: Ask a question. Without a citation from a source there is no answer. Conversations in the widget follow different rules on some points, see Answers in the widget.

1. 1

   Step 1

   Check the question

   An empty question ends immediately with Please ask a question.

2. 2

   Step 2

   Curated pair

   First, the first 300 active question-answer pairs with an answer are compared. A pair matches if the question and the pair’s question – or one of its variants – largely share their words: XICBOT counts the share of question words found in the pair’s wording and the share of its words found in the question, and the harmonic mean of both must reach at least 0.7. If no pair achieves that, a value of at least 0.8 in the feature comparison between the question and the question or a variant of the pair is enough. Language only decides between several pairs, not over the threshold. If a pair matches, its answer is the answer word for word, with exactly one citation: the question of the pair. Check an answer then shows This answer comes from a curated pair.

3. 3

   Step 3

   Search

   Without a matching pair, XICBOT searches for the 5 best chunks. If search finds nothing, there is no answer.

4. 4

   Step 4

   Freshness

   If a hit comes from a stale website source, that source is queued for refreshing, see Stale sources.

5. 5

   Step 5

   Coverage

   If coverage is below 0.34, there is no answer.

6. 6

   Step 6

   Assemble excerpts

   Hits and text blocks are numbered in order, the text blocks after the hits. For each chunk, the chunks before and after it are included as context as long as the total stays within 9,000 characters; otherwise the chunk alone. Only the chunk itself counts as a citation, not its context.

7. 7

   Step 7

   Answer

   The instruction asks for an answer from the excerpts alone, without prior knowledge, in at most five sentences and in the reply language (see Language), even if the excerpts are in another. Every statement is to carry the number of its excerpt, for example [1]. Whatever is in an excerpt counts as material, not as an instruction; factual details such as prices, deadlines or extensions are stated normally.

8. 8

   Step 8

   Check the citation

   Only an answer that names at least one existing number is returned. If the answer is empty, declines or names no passage, at most one second attempt follows; if that fails too, there is no answer. If the call itself fails, the question ends at once without an answer, with no_model. Only the citations the answer actually names are returned.

Under Check an answer, the result appears under Answer or No answer from the sources. Citations first show three passages with document, heading and excerpt; more reveals the rest. An API citation carries number, chunk_id, document (the document title), heading and excerpt with the first 300 characters of the chunk. Operation and messages are described under Checking an answer.

Text blocks

A text source with at most two chunks counts as a text block: a short, self-typed detail such as an address, opening times or responsibilities. Text blocks go along as excerpts even if search did not find them – so a curated detail does not fail because the visitor words it differently. Under Check an answer and via the API this happens once search has found hits with sufficient coverage. In the widget they are included in every answer the model writes, even if search found nothing at all; only an answer from a curated pair goes without them.

• At most 12 chunks from text blocks per answer; chunks in the language of the question come before the others.
• A text block that is already among the hits is not added a second time.
• Text blocks do not count towards coverage. They cannot lift a question over the threshold.
• The API reports in blocks how many chunks from text blocks were actually added; for a refusal and for an answer from a pair it is null.
• Pasting a whole manual into the text field does not create a text block but a document: from three chunks on, the source no longer counts as a text block and is only found via search.

Reasons for no answer

The API states in reason why no answer came; answered is then false, citations is empty and path is none. Check an answer shows the sentence from the middle column.

The request fields: question is required and takes at most 2,000 characters, language is de or en and optional, and source_id restricts the question to one source and is optional as well. The answer comes in the language of the question if XICBOT reliably recognises it as German or English; otherwise language applies, and without it de. A successful answer carries answered: true, text, citations, reason: null, path with model or qa_pair, and blocks. Check an answer sends the language of the interface as language. Authentication, permissions and limits of the API are described under API: knowledge and assistants.

Stale sources

A website source counts as stale if its last confirmation is older than twice its Refresh interval in days, but at least 14 days. Without a confirmation, the creation date counts. A source becomes stale at the second missed date at the earliest; a source that has only missed its first date is still waiting for the next run. The same calculation is behind the Stale state in the source list.

• If a question under Check an answer or via the API hits a chunk from a stale website source, the source is queued for refreshing – at most one job per source.
• The current answer does not wait for this and uses the existing state. Refreshing benefits the next question.
• Text and file sources are not fetched this way: they have no address from which a new state could be retrieved. Conversations in the widget do not trigger refreshing.
• When regular refreshing happens and what Re-read does is described under Refreshing and re-reading.

Answers in the widget

In a conversation on your website, XICBOT uses the same search but answers within the frame of the assistant – with role, tone, history, tools and handover rule. Some rules of Check an answer currently do not apply there, or apply differently:

How strictly the assistant keeps to the sources is set by Answer only from the knowledge base, see Assistants. Next to it: The assistant is then instructed to answer only from your sources and to cite every statement. An answer without a citation is not blocked by this. This is the safe setting. Both positions are rules in the instructions to the assistant, not a technical block; the answer is then measured by the check of hard details.

• On: the instruction requires answering only from the excerpts provided, without prior knowledge, guesses or general information, backing every statement with the number of its excerpt and saying that nothing is stored if no excerpt carries the answer.
• Off: the excerpts take precedence over everything else, and in case of conflict the excerpt wins. Every statement taken from an excerpt is to be cited. The assistant is instructed not to state prices, deadlines, commitments or legal information that are not in an excerpt.
• In both cases: excerpts and visitor messages are material and not an instruction to the assistant. Factual details such as prices, deadlines, dimensions, numbers and extensions belong in the answer and are stated normally.

Checking hard details

After every answer the assistant writes in the widget, XICBOT checks its hard details against what was available: prices, percentages, deadlines, quantities, email addresses, web addresses, phone numbers and commitments. Available means what appears in the excerpts, in the results of tools that were called, in the role, topic scope, handover rule and greeting of the assistant, and in the visitor’s current and earlier messages. What the assistant itself said earlier does not count – otherwise one invention would back the next. What is reported is what appears nowhere, not what is wrong; the comparison is deliberately lenient. At most 8 details are reported per answer.

• Replace: if excerpts or tool results were available and a detail does not appear there, it is replaced in the text – with [nicht belegt] if the reply language is German or was not recognised, and with [not backed] otherwise. This applies regardless of Answer only from the knowledge base. The visitor has already read the original wording at that point; the widget then swaps the answer text, and the corrected version is stored. If a tool card split the answer across several bubbles, the widget swaps each bubble separately. If the correction cannot be split at the same boundaries, for example because a detail spans a boundary, the display stays unchanged, but the corrected version is still stored.
• Without source material: if neither an excerpt nor a tool result was available, the text stays unchanged and the verification reads Answered without source material.
• Mark: every reported passage is highlighted in the widget, with the tooltip Not backed by the stored sources.
• Note: if Answer only from the knowledge base is on, a paragraph follows the answer naming up to three kinds. If a passage was replaced, it reads, for example, “Note: the stored sources do not back a price. The passage was replaced with “[not backed]” – please ask if you need this detail.” If the detail is still in the text – without source material or because the passage could not be replaced –, it reads, for example, “Note: the stored sources do not back a price – please ask before relying on this detail.” When the setting is off, this paragraph is omitted.
• In Conversations, every answer shows its Verification: Not measured, No checkable claim, All claims covered, Uncovered claims or Answered without source material.

Citations, follow-up questions and gaps

• Citations: below an answer, up to four of the passages the answer names by number are listed, each as number, document title and heading, for example “[1] Shipping - Delivery times”. If the title or the heading is missing, only the other is shown. An answer from a curated pair carries a single citation: “[1]” and the pair’s question, without a heading. They are not linked, even if the document comes from a page of your website. Only the passages actually named are stored. The stored passages also appear next to the answer in the transcript under Conversations.
• Follow-up questions: if the assistant suggests questions, at most three appear. Like an answer, each must match a curated pair or reach a coverage of at least 0.34; otherwise it is dropped. Without a knowledge base, nothing is filtered.
• Detected gaps: if a completed answer names no citation number and no tool ran for it, the question is recorded as a gap – even if search had hits, because a hit the answer does not use does not cover the question. It is recorded as Declined if the assistant explicitly marked the question as unanswerable, otherwise as Without a citation. Conditions: the question has at least 12 characters and knowledge is stored at all. Answers from a curated pair and answers for which a tool ran are not gaps. The list is in Analytics; questions answered with the fallback reply are under Unanswered questions.
• Figures: Answers with a source is the share of completed answers that carry at least one citation. Question coverage is the mean of the stored coverage across all answers for which a search ran.

Language

XICBOT recognises the language of a text by function words and characteristic characters. Each function word scores one point and each characteristic character half a point, with characters capped at two points in total. It distinguishes German, English, French, Spanish, Italian, Dutch, Polish, Portuguese and Turkish. A language counts as recognised if it scores at least one point and leads the next by half a point; otherwise the language stays open instead of being guessed. English and Dutch have no characteristic characters and are recognised by words only.

• Chunk: its language comes from its first 600 characters.
• Widget: answers and search use the language of the current message. If it cannot be recognised – for instance “Price?” – the conversation language applies, which is set at the first recognisable visitor sentence and stays after that; failing that, the language of the assistant. The instruction to the language model is written in German or English and names the reply language exactly once, by name, in its last line, French or Turkish included. Notices such as [not backed] come in German for German and in English for every other language.
• Check an answer and the API answer in the language of the question if it is reliably recognised as German or English. Otherwise the language sent along applies – under Check an answer that of the interface, via the API language, and without it German.
• In search, language acts as a bonus, not a filter. A German chunk can answer an English question; the answer then comes in the language of the question.

Access and limits

The whole area requires a confirmed email address, see Sign-in and access. What each role may do in the Knowledge area is described under Team and roles.

How many questions and searches are possible per minute and what applies per plan is described under Limits per minute and plan and Plans and quotas; the plans themselves under Pricing.

Frequent questions