Background jobs

Some work takes longer than a single request: reading a website, recalculating knowledge, syncing a shop, delivering emails and webhooks, running an assistant's check questions. XICBOT accepts such work as a background job, answers right away and does the job afterwards. This page covers which jobs exist, in what order they get their turn, what happens when one fails and how long a job is kept.

What runs in the background

Every job has a type (task), a lane and a priority. The lane follows from the type; the priority is set by whatever creates the job. A lower number means an earlier turn.

When the daily refresh run, the hourly shop sync and the weekly report start is covered in Schedules.

Lanes and order

The index lane carries the long runs: reading sources, recalculating, setup, acceptance runs and shop sync. The default lane carries everything else, namely emails, webhooks and the weekly report. The split is meant to keep an email or a webhook from waiting minutes behind a source being read.

On each lane XICBOT takes the waiting job with the lowest priority number and, with equal numbers, the older one. A job sitting out its wait after a failed attempt is skipped until that wait is over.

Job states

Retries after a failure

A job has up to three attempts. An attempt counts as soon as XICBOT picks the job up. If it fails and attempts are left, the job goes back to pending with a wait. If the third attempt fails, the job stays on failed for good. The error text remains on the job, cut to 500 characters. A job that has failed for good raises a critical notice with the source queue for operations; see Notices.

What counts as a failure

• Reading a source: if the source is not ready after the run, the attempt has failed. Every failed attempt triggers the webhook event knowledge.failed; a successful one triggers knowledge.indexed with the number of documents and sections. If the build fails after fetching, the source shows not indexed with the reason ingest_failed, and the existing content keeps answering. If the source has been deleted in the meantime, the job counts as done without being carried out; its result carries skipped with source_deleted.
• Acceptance: if the check run has been deleted along with its assistant, the job counts as done without being carried out (skipped with run_deleted). If the job fails for good, is recovered as failed or expires, the check run shows stopped with the reason job_failed; see Test dialog and acceptance.
• Webhook: only a response with a 2xx status counts as delivered. XICBOT does not follow redirects (3xx); they count as a failure, as does a response that takes longer than 8 seconds. If the webhook has been deleted or switched off in the meantime, or the plan no longer includes the feature for keys and webhooks, the job counts as done without delivery.
• Switching off a webhook: after 20 failed delivery attempts in a row XICBOT switches the webhook off; every retry counts. A successful delivery resets the counter.
• Email: an email already recorded as sent does not go out again on a further attempt. A handover email is no longer sent if the conversation has been deleted in the meantime.
• Shop sync: a failed fetch from the shop does not make the job fail: it ends on done, and its result carries ok: false with the reason under error. The existing catalogue stays, the error is shown on the connection, and the next scheduled sync tries again.

Webhook signatures, headers and events are covered in API keys and webhooks, connecting a shop in Catalogue, basket and shop connection.

Stuck and stale jobs

A job that has been marked as running for more than an hour, or whose worker process is no longer running, counts as orphaned. XICBOT resets it when the background service starts and during the hourly maintenance: to pending while attempts are left, otherwise to failed. The error text gives the reason; for the one-hour limit it reads „seit mehr als 3600 Sekunden ohne Fortschritt“ (more than 3600 seconds without progress).

A job that is still waiting or running 7 days after it was created expires during maintenance. It ends on failed with the error expired and is not carried out further; its end is taken as 7 days after creation. If an email was still waiting to be sent with it, that email counts as failed with the note „Versandauftrag nach 7 Tagen abgelaufen“ (send job expired after 7 days).

Duplicate jobs

In several places XICBOT creates no second job while one of the same kind is open, that is waiting or running:

• Re-read on a source that already has an open intake or refresh job, including the first intake of a new website, takes over that job; the response carries already_open: true. If you tick Rebuild completely while a job without a rebuild is still waiting, the job picks up the rebuild. If it is already running, the API answers 409 with refresh_running and the app reports “A build is running or queued for this source. Please try again afterwards.” When an open job is taken over, the recalculation is still queued in addition.
• The daily refresh run and the refresh during an answer skip a source that already has an open intake or refresh job.
• POST /api/v1/knowledge/embed takes over an open recalculation job of the account, even one for a different source; a source_id you pass then has no effect.
• A manual sync takes over an open sync; the app then reports “A sync is already waiting.”, otherwise “The sync has been queued.” If no sync is open, a lock applies until 15 minutes after the last manual request: response 429 with sync_too_soon, retry_after (seconds) and the Retry-After header. The app then reports “A manual sync is possible at most every 15 minutes. The regular sync continues independently.”
• The scheduled hourly sync skips an account that already has an open sync.
• The weekly schedule skips an account that already has a weekly report job this week or has already been sent the report.
• An acceptance run for an assistant that already has an open run returns that run, provided it was created less than an hour ago.
• While a source has an open intake or refresh job, no earlier version of that source can be restored: response 409 with refresh_running.

Tracking jobs through the API

When a request creates a job, the API answers with 202 and a job object holding id, status and task: when creating a website source, for POST /api/v1/knowledge/sources/{id}/refresh and for POST /api/v1/knowledge/embed. The manual sync via POST /api/v1/catalog/shop/sync returns the ID as job_id, together with already_pending; it requires the Administration role. GET /api/v1/knowledge/jobs/{id} returns the state of each of these jobs, including the sync.

How long jobs are kept

• Once a job is done or has failed for good, XICBOT cuts its data down to references such as source, run, account, email template, delivery, webhook and event. Recipient, email content and the content of a webhook message are dropped; if anything was dropped, the job carries the marker reduced.
• Done and failed jobs are deleted 14 days after they end. Until then type, times and error text remain.
• When you delete a conversation, the related email and webhook jobs go with it. Open webhook deliveries of the events lead.created and appointment.requested stay, because they report a case.
• Your account's jobs are part of the data export. Under Legal and data, the Stored data table counts them in the Background jobs row. They are deleted together with the account.

Retention periods for conversations and other data are in Retention and deletion, export and account deletion in Privacy and legal.

Frequent questions