Maintain knowledge

Knowledge is where you decide what your assistants answer from: pasted texts, uploaded files, websites and curated question and answer pairs. This page covers every field in the area with its limits, defaults and messages, and explains what saving, re-reading, restoring and deleting actually do. How sources turn into passages and passages into cited answers is explained in Knowledge and answers.

Where to find it

Open it from the menu under Work › Knowledge (app.xicbot.de/wissen). In an assistant's setup path, the step Connect knowledge between Create and Test links here. The page opens with the sentence Add texts, files or a website. The assistant answers only from these sources and cites the passage for every statement.

• New source and Check an answer sit side by side at the top: on the left you add a source, on the right you put a question to your knowledge base.
• Below them come Question and answer pairs, Probe set with Weighting, Questions answered with the fallback answer and finally the list Your sources.
• The side column shows Your sources as a jump list: for each source the label and, below it, the address, the file name for files, or otherwise the type. Search the list filters by label and address or file name; if nothing matches, you see No entry matches this search term. Clicking an entry opens that source in the list.

Source types

The Type of source field offers four types. Its help text reads Paste text directly, upload a file, or have a website fetched from its start address. With “Questions and answers” you maintain the pairs individually below; there is one such collection per account. The type is fixed when the source is created and cannot be changed later. Once the Questions and answers collection exists, this type is locked in the field, and a note below it names the collection new pairs go into, with a link to Question and answer pairs.

A text or a file has no origin the service could read again later. To renew such content, add a new source and then delete the old one.

Adding a source

Add source submits the form; while the service is working, the button reads Processing …. Afterwards the fields are cleared and the new source sits at the top of Your sources. If the service rejects the input, you see That did not work. Please check your entries. and below the affected field a sentence built from the field name and the rule, for example Label is too long., File is too large (10 MB at most). or Start address is not a valid address. If the service rejects the start address itself because its scheme is not http or https, it does not point to the public internet, its host name cannot be resolved or it cannot be read, the reason appears at the address field, for example Only addresses starting with http:// or https:// can be fetched., This target must not be fetched, for example an internal address. or The host name of this address cannot be resolved. Please check the spelling. The message then reads Please check the marked fields. If the type File is chosen without a file, the field shows File is required. and the message reads Please check the marked fields. If the service rejects one of the More settings, the section opens and the sentence appears at the field.

File formats

The file extension decides how a file is read. There is no list of permitted extensions: anything without one of the extensions below is read as plain text. If the first 8 KB of a file contain a null byte, as with an image or an archive, processing fails and the source shows This file contains no readable text, for example an image or an archive. Upload text, Markdown, CSV, HTML, JSON, PDF or Word. This check does not apply to pdf, docx, docm, doc and dot; UTF-16 text with a byte order mark counts as text. Except for HTML, the file name becomes the document title.

The Your sources list

The list shows up to 200 sources, newest first. Each row gives the label, type and status, plus Documents and Passages, and for a collection of pairs their number under Question and answer pairs. Website rows add Refresh interval in days, shown as House interval (7 days) when you have not set your own, and the address; file rows show the file name there. Without any source you see No source added yet. with the sentence Without a source the assistant's answer stays empty. Add a text, a file or a website above. If loading fails, The list could not be loaded. appears with a Load again button.

Website rows show freshness on a separate line: Current or Stale, each with Last confirmed and a date, or Never confirmed. A website counts as confirmed when a run has read it successfully. Stale applies once twice the refresh interval has passed since the last confirmation, with 14 days as the minimum.

If a run did not read the whole website, the source shows Only the first 25 pages of this website were read. or, when no page limit is known, This website was not read completely. After that comes the number of open links, for example 12 links were still open when the crawl stopped., or How many links were still open was not measured.

When indexing fails

A failed source shows not indexed, with the reason and Failed on plus date and time below it. Its existing content stays in place and keeps answering; a failed run deletes nothing. A website only counts as failed when no page yielded any text; the reason shown is then the one the start address failed with.

Editing a source

Edit on a source expands its form inside the list, Close collapses it again. Save stores the settings, reports Saved. and does not re-read the source. If the audit log entry cannot be written, the change does not take effect and the page reports Not saved: the entry in the audit log could not be written. Please try again.

Excluded paths

The hint at the field reads One pattern per line, * as wildcard, at most 50 patterns of 300 characters each. Applies in addition to the source's robots.txt. The patterns take precedence over robots.txt: whatever a pattern excludes is not read even if robots.txt allows it. Changed patterns take effect from the next run; patterns entered under More settings when adding the source already apply to the first one.

• A pattern matches from the start of the path, that is from the first / after the host name, including the query after ?. A pattern that starts with neither / nor * and contains no :// therefore matches nothing.
• * stands for any number of characters; a $ at the end anchors the pattern to the end of the address.
• If a pattern contains ://, it is compared with the full address.
• Examples: /intern/ excludes everything below that directory, *.pdf$ any address ending in .pdf, /suche? any search page with a query.

How a website is read

A run starts at the Start address, adds the addresses from the website's sitemap, then the addresses already indexed for this source, and finally follows links to pages on the same website. The run ends when no address is left or 25 pages have yielded text; a page the server reports as unchanged counts too. Links are only queued while fewer than 100 addresses are waiting to be fetched; further links are left aside.

• Only HTML and plain text (text/plain) are taken in.
• Each page has 20 seconds to respond. Up to five redirects are followed, and only within the same website.
• A page may be at most 3 MB. Every document, including those from texts and files, is cut off after 2,000,000 characters.
• robots.txt is respected. A dedicated group whose name forms the beginning of XICBOT-Wissensbegeher, such as XICBOT, applies instead of the general group for *; if several match, the longest name applies. Upper and lower case do not matter here. When rules conflict, the longest one wins; at equal length, the allowing one wins. If robots.txt answers with a server error (status 500 or higher), the run ends with The website's robots.txt could not be reached because of a server error. Until it responds again, no page of this website is read. and the existing content stays in place. If no connection can be made at all, it ends with This address could not be reached. If the server answers the robots.txt request with a 4xx status, for example because the file does not exist, no rules apply.
• Pages with noindex or none in the robots meta tag or in a meta tag named XICBOT-Wissensbegeher, and pages with a publication date in the future (article:published_time, datePublished), count as unpublished: they are not split into chunks and do not feed into any answer. A page with a future date is added no earlier than the first run after that date.
• Pages with identical text are taken in only once. A page without a title, and a plain-text page, carry their address as the title.

Refreshing and re-reading

A schedule queues website sources whose interval has run out for a refresh in the background: a source is due when more days have passed since its last confirmation than Refresh interval in days says; without your own value, the House interval (7 days) applies. Failed and outdated sources are included. A source that already has an open job is skipped. The service's schedules are described under Schedules.

During a refresh a page is only split again if it has changed: if the server reports it unchanged, or its text is the same as before, its passages stay as they are. A run that changed nothing creates no new version. Pages the run no longer finds drop out; a run that finds nothing at all deletes nothing.

Re-read in a website's expanded form queues a run straight away. The source then shows processing, and the form adds The source is being indexed in the background. The status updates on its own. With Rebuild completely ticked, the run also re-splits unchanged pages, which is needed after changing the chunk size. If an intake or refresh job is already open for the source, Re-read takes over that job. With Rebuild completely ticked, a job that is still waiting picks up the rebuild; if it is already running without a rebuild, the service declines and the form shows A build is running or queued for this source. Please try again afterwards. The button is disabled while the source shows processing. If the open job ends at exactly that moment, Re-read queues a new run with a rebuild instead.

If a question under Check an answer hits an outdated website source, the service also queues a refresh for it. The answer itself still comes from the existing content.

Feature spread

Above the list you see the state of the feature spread, for example Feature spread: 180/200 sections · 20 pending. The first number counts the passages already in the semantic part of the search. Build spread queues a job for the whole knowledge base; while the request is being sent, the button reads building. If such a job is already open, that job is used.

Restoring versions

The expanded form of a source lists its builds under Versions. The explanation reads Every build writes a new version. The previous one keeps answering in full until the new one is complete. and The last 5 versions are kept. Each version is shown with its number, content and time, for example Version 3 with 12 documents, 140 chunks; the one in use is marked answering. If there is none yet, you see No version has been built for this source yet.

Restore on an older version asks Newer versions are discarded. Restore? with the buttons Restore and End. After you confirm, that version answers again, the newer ones are deleted, and the page reports something like Version 3 is answering again. Discarded: 2. Any answer shown under Check an answer is cleared, because its citations may belong to a discarded version.

Documents of a source

Below the versions, the expanded form shows the Documents list: the pages and files of the source by title, each with its address, character count and status. Above it, it reads The pages and files of this source, by title. Drafts and documents with a future date are listed here but are not used in answers. For a website this shows which pages were taken in and where an excluded path pays off. Every role can see the list.

• The address of a page is a link that opens in a new window. Text and file sources have no address.
• At most 200 documents are shown. If the source has more, it says Showing the first 200 of … documents. above the list.
• Without a document it says This source has no document yet. If loading fails, The list could not be loaded. appears with Load again.
• After Restore of a version and when a run of the source ends, the list reloads.

Deleting a source

Delete on a source asks Delete this source and all of its passages? with Delete and Cancel. After you confirm, the page reports The source has been deleted. and any answer shown under Check an answer is cleared.

• Along with the source go its documents and passages, all its versions, the question and answer pairs that belong to it, and the probe lines that point to one of its documents.
• The deletion is recorded in the audit log in the same step as the deletion itself. If the record cannot be written, the source is kept and you see Not saved: the entry in the audit log could not be written. Please try again.
• There is no separate retention period per source. What else is kept and for how long is described in Retention and deletion.

Question and answer pairs

A pair is a ready-made answer to a specific question. Above the form it says If a visitor question matches a pair or one of its variants, the curated answer is returned verbatim, without a model call.

• Add pair stores the pair in the oldest source of the type Questions and answers. If there is none, a new collection is created. A second one cannot be created; accounts that have several from earlier keep them, and new pairs go into the oldest. Its label follows the language of your sign-in, not the interface language currently chosen: Questions and answers for English, Fragen und Antworten otherwise. An existing collection keeps its label; you can change it via Edit under Label. The new collection counts against your plan's limit for sources: once it is reached, no pair is created and the page reports, for example, The … plan limit is reached: … of …. See Billing for the overview. The collection is only created together with its Knowledge source created audit log entry.
• On an existing pair you change the question, the answer, Variants, one per line and Active, and store them with Save; there is no separate message for this.
• Delete asks Delete this question and answer pair? with Delete and Cancel. Without any pair you see No pairs yet.
• Adding, saving and deleting a pair only take effect together with their audit log entry. If the entry cannot be written, no new pair is created, an existing pair stays as it was, and the page reports Not saved: the entry in the audit log could not be written. Please try again.
• Matching covers the first 300 active pairs that have an answer, in list order, first by words and then by feature spread. If there are more active pairs with an answer, the section warns: Only the first 300 active pairs with an answer, in list order, are compared when answering. The remaining … are not used. If a pair matches, its answer is the answer, word for word, and Check an answer adds This answer comes from a curated pair.

Checking an answer

Here you ask your knowledge base a question the way a visitor would. The hint reads Ask a question. Without a citation from a source there is no answer. The Your question field is required and takes at most 2000 characters. Ask sends it; meanwhile the button reads Answering …. The answer comes in the language of the question, German or English; if the language cannot be determined reliably or is a different one, the language of the interface applies.

The result appears under Answer or, if the sources yield nothing, under No answer from the sources. The Citations follow below; more reveals further ones. If there is no answer, you see one of these sentences:

Check an answer is open to every role and creates nothing. To try the complete assistant with its role, tone and handover rule, use the test dialog, see Test and acceptance.

Probe set

The page describes it like this: The probe set measures whether search finds the right document. Each line names a question and the document it should hit; what is measured is the position among the first 10 results. The run asks no model, only your own search.

• Add probe line adds the line. The probe set holds at most 100 lines; once it is full the button is disabled, and adding a line through the API fails with The probe set is full. Delete a line before adding a new one.
• Each line shows Expected with the document, followed by Position 3, not found or, before the first measurement, never measured.
• Measure searches each question once; meanwhile the button reads Measuring …. The top of the section then shows Retrieval quality, the mean of 1 divided by the position across all lines, with a line that was not found counting as 0, and Found as found lines out of all lines, for example 8 / 10. Before the first measurement it says Not measured.
• In addition, a schedule measures the probe set by itself in the background, each time after the refresh run that queues the due website sources for re-reading. Without probe lines it measures nothing.
• Delete on a line asks Delete this probe line?. Without any lines you see No probe lines yet.
• Adding and deleting a probe line only take effect together with their audit log entry. If the entry cannot be written, the probe set stays unchanged and the page reports Not saved: the entry in the audit log could not be written. Please try again.

Weighting

The weighting sets how search scores its hits. The explanation reads The title factor sets how much a hit in a section heading counts, the keyword weight how much a keyword derived during the build counts. Empty fields mean default. Tuning measures the probe set with every candidate and changes nothing until you apply a value.

• Below each field you see its range and default. Save weighting stores both values and reports Weighting saved. Next to it the values in force are shown as In effect with title factor and keyword weight.
• Tune needs at least one probe line and measures the oldest 20 probe lines with several candidates: first the title factors 1.0, 1.4, 1.8, 2.2, 2.6 and 3.0, then, with the best of those, the keyword weights 0, 1, 2, 3, 5 and 8. Each candidate is then listed with Retrieval quality and Found, the current one marked In effect, with a line above such as Measured on 20 of 35 probe lines. Tuning itself saves nothing. While it measures, the button reads Measuring ….
• Apply best value is only available when the best candidate genuinely beats the current setting; on a tie the button stays disabled and the line above the list says No candidate beats the current setting. The button measures again against the current knowledge base and stores the best value only if it still comes out ahead; the page then reports Best value applied. Meanwhile the button reads Saving ….
• Save weighting and Apply best value only change the weights together with an audit log entry. If the entry cannot be written, the previous values stay in force and the page reports Not saved: the entry in the audit log could not be written. Please try again.

Questions answered with the fallback answer

This list shows visitor questions from the last 30 days that the assistant could only meet with its fallback answer because reply generation failed. The latest 50 such cases are evaluated; the questions are anonymised, cut to 300 characters and grouped with similar ones. The page explains These are questions the assistant could only answer with the fallback answer because reply generation failed. Similar questions are grouped, and what weighs most comes first: conversations affected, handed-over ones counted twice. One click turns a group into a pair; you add the answer above. Questions with no supporting source are listed in the analytics.

• Each group states how often the question was asked, how many conversations were affected and how many of them were handed over, for example 5 times asked · 3 conversations affected, 1 of them handed over, plus the variants in the group.
• Create pair creates a pair from the group's question and variants, without an answer and therefore inactive; the row then shows Saved. You add the answer above under Question and answer pairs and tick Active. The adopted variants appear there in the field Variants, one per line; trim whatever does not fit the question before you activate the pair.
• Without entries you see The assistant gave no fallback answer in the last 30 days.
• The list requires conversation analytics in your plan. Without it, the section says Your … plan does not include this feature. See Billing for the overview. with the name of your plan.
• If loading fails, The list could not be loaded. appears with a Load again button. If the service does not recognise your device's time zone, the sentence reads The service does not recognise the time zone of this device. Without it the period cannot be determined, so the list is not shown. Trying again will not change this. instead, and there is no Load again button.

Keeping knowledge current

The assistant answers from whatever your sources contain right now. A detail that is out of date is passed on just like a correct one. These habits keep your knowledge reliable:

• After a change on your website, choose Re-read instead of waiting for the next refresh. While the run is going, the list updates every 5 seconds; once it has finished, it shows indexed, the document and passage counts, and Last confirmed with a date.
• The content of a text or file source cannot be changed in place: Edit only covers the label and settings. Add the new version as a new source, then delete the old one. No schedule refreshes text or file sources either; only websites are queued.
• Delete sources nobody maintains any more. The assistant searches all sources at once for every question, so a forgotten source can end up in an answer at any time.
• Keep prices, deadlines and commitments in exactly one source. If two versions exist, either of them can make it into an answer. The check of hard details does not catch this: it flags details that appear in none of the passages at hand, not details that are out of date.
• Go through Questions answered with the fallback answer and, in analytics, the Detected gaps at regular intervals, and add whatever your sources lack.
• After larger changes to your knowledge, run the acceptance check again.

Rate and plan limits

How often an action can be triggered per minute is counted per signed-in account:

Beyond that, the page reports Too many attempts. Please wait a moment. When you add a source, the service also checks your plan's limits for sources, documents and passages; when you re-read, the limits for documents and passages. If one is reached, a message with the plan and the numbers appears, for example The … plan limit is reached: … of …. See Billing for the overview., or Your plan does not cover this. See Billing for the overview. The limits per plan are listed under Plans and quotas and on the pricing page.

Through the API

Sources, pairs, search, test questions and the probe set can also be used through the API, for example POST /api/v1/knowledge/ask for a cited answer. Endpoints, fields and responses are described in API: knowledge and assistants, the limits in API limits.

Frequent questions