When an AI chat assistant accepts questions from visitors, reads content from your own website and triggers connected actions, it constantly processes text from outside. That is exactly the entry point for prompt injection: the attempt to slip new instructions to the assistant through cleverly worded inputs or hidden snippets of text. The OWASP Foundation lists prompt injection in its Top 10 for LLM applications as LLM01 and therefore in first place (OWASP), for the second edition in a row (OWASP). The reason is structural: a language model processes data and instructions in the same channel, with no clear separation between what it should do and what it should only read. This article explains, factually, what distinguishes direct from indirect attacks, why prompt injection cannot be switched off with a single filter, and which guardrails make an assistant resilient: input filtering, output checks, no secrets in the prompt, clearly limited permissions and vigilant monitoring. It deliberately sets the topic apart from data protection, because this is not about the GDPR but about attack security. No one can promise absolute security, but a considered concept lowers the risk considerably.
Why Prompt Injection Sits at the Top
Prompt injection is not an exotic edge case but the most widely recognised risk of generative AI. In the current OWASP Top 10 for LLM applications from 2025, prompt injection stands as LLM01 at the top of the ten risk categories in total (OWASP), followed by topics such as sensitive information disclosure as LLM02 and system prompt leakage as LLM07 (OWASP). There is a simple reason why manipulation through text of all things sits at the top: it affects every application that accepts text from outside at all, and a chat assistant does exactly that by its very nature. NIST too describes prompt injection in its taxonomy of adversarial machine learning, NIST AI 100-2e2025 from March 2025, as its own class of attack and there explicitly distinguishes direct from indirect variants (NIST).
Briefly explained: what is prompt injection?
For a chat assistant this is not an academic question. It is precisely the kind of application that continuously takes in text from outside: questions in the chat window, content from your own website, details from a form, sometimes data from connected systems. Each of these sources can, intentionally or not, contain text that looks like an instruction. How many companies this risk now concerns is shown by adoption: in Germany, 36 percent (Bitkom) of companies already use AI, almost twice as many as the 20 percent (Bitkom) the year before. As the number of live assistants grows, so does the attack surface, which is why security belongs in the picture from the start.
Data and Instructions in the Same Channel
The actual cause runs deeper than a single programming error. A language model receives its system prompt, the conversation history and the user's input as one continuous stream of text. Technically there is no hard boundary that says: up to here are binding instructions, from here on there is only material to read. To the model, both are language. Germany's Federal Office for Information Security pointed out as early as 2023, in a cyber-security warning, that indirect prompt injections are an intrinsic weakness of application-integrated AI language models, so not a passing teething trouble but a property of the technology (BSI).
As long as a model processes data and instructions in the same stream of text, separating task from content is not a given, it must be enforced by the design of the application.
Because the weakness is structural, it cannot be switched off with a single filter. Both NIST and the BSI therefore recommend not a single line of defence but a layered approach of several complementary measures, often called defense in depth (NIST, BSI). The aim is not to rule out every conceivable attack, but to reduce the likelihood and the possible impact far enough that an assistant can go live responsibly. This thinking in layers runs through the sections that follow.
Direct and Indirect Attacks
The taxonomy from OWASP and NIST distinguishes two basic forms that differ not in their goal but in their path (OWASP, NIST). Both want to change the assistant's behaviour, but one arrives directly through the input field while the other hides in content the assistant merely reads.
Direct prompt injection
The attacker types the malicious instruction into the chat themselves, for example as a request to ignore previous rules or to reveal internal specifications. Here the manipulation arrives directly through the input field.
Indirect prompt injection
The instruction sits hidden in external content the assistant processes, for example in a web page, a document, a review or an email. If the assistant reads this text in order to answer, the embedded instruction can take effect without the user ever seeing it.
The indirect variant is especially treacherous, because the instruction does not come from the visible conversation partner but from a seemingly harmless source. NIST added exactly this extension to its taxonomy AI 100-2e2025 and describes indirect prompt injection as an attack in which prepared content is planted in data sources that a system later reads and processes (NIST). For an assistant this means: every external source it draws knowledge from should be treated as potentially manipulated and must not be executed like a command unchecked. How an assistant instead ties itself closely to checked, curated knowledge is therefore not only a question of answer quality but also of security.
What a Successful Attack Causes
If a prompt injection succeeds, the consequences range from harmless to sensitive, depending on which capabilities and access the assistant has. The OWASP list assigns several of these consequences to their own categories, which shows how closely the risks are connected (OWASP). The typical effects are, above all:
- Disclosure of information held in the system prompt or context, which corresponds to the system prompt leakage category as LLM07 (OWASP)
- Bypassing the defined rules, limits and tone, often referred to as a jailbreak
- Triggering unwanted actions through connected tools, which OWASP lists as excessive agency, LLM06 (OWASP)
- Output of false, distorted or manipulated information to visitors
- Passing manipulated text on to downstream systems, where it can cause further harm
That such incidents are not mere theory is shown by the outlook: Gartner expects that by 2028 around 25 percent (Gartner) of all enterprise applications using generative AI will record at least five minor security incidents per year, up from 9 percent (Gartner) in 2025. For serious incidents, Gartner anticipates a rise to 15 percent (Gartner) by 2029, up from 3 percent (Gartner) in 2025. And by 2028, around 50 percent (Gartner) of the effort spent responding to security incidents is expected to concern AI-driven applications. This direction underlines that hardening is not a one-off hurdle but part of ongoing operation.
Attack Security Is Not Data Protection
Prompt injection security and data protection are often lumped together, yet they are two different building sites. Data protection under the GDPR is about processing personal data lawfully, sparingly and transparently, with a legal basis, a data-processing agreement and a deletion concept. Attack security is about something else: protection against targeted manipulation, that is, making sure the assistant is not lured into unwanted behaviour by injected text. An assistant can be hosted in exemplary compliance with the GDPR and still remain vulnerable to prompt injection, and vice versa. Both topics belong together in thought but must not be confused. What matters concretely for data protection is covered in the article on the GDPR-compliant AI assistant; the question of labelling AI in the chat, in turn, is the subject of the EU AI Act and the transparency obligation.
Keep security and data protection cleanly apart
Guardrails That Protect
Because there is no single switch, robust hardening relies on several layers that work together. Both the BSI and OWASP recommend a combination of technical and organisational measures rather than a single silver bullet (BSI, OWASP). The following six guardrails have proven their worth in practice (project experience) and interlock, so that an attack would have to clear several hurdles at once.
Filter and check inputs
Inputs are checked for suspicious patterns and typical attack phrasings before the assistant processes them further. That catches a share of the direct injection attempts early on.
Control outputs
Before an answer goes to visitors or to downstream systems, it is checked rather than passed through unfiltered. That keeps manipulated text from reaching the outside unchecked.
No secrets in the prompt
Passwords, keys or internal data do not belong in the system prompt, because it can be coaxed out in whole or in part. Anything truly secret sits outside the model.
Least privilege and permissions
The assistant is granted only clearly defined, approved actions instead of blanket full access. That keeps the damage limited even if a manipulation gets through once.
Tie answers to checked knowledge
Instead of speculating freely, the assistant relies on a curated knowledge base. External content is read as material, not executed as an instruction.
Monitoring and logging
Conspicuous conversations and unusual inputs are logged and evaluated, so attack attempts can be spotted and the rules refined.
The system prompt is not a safe
Two of these guardrails reach especially deep into how an assistant is built. That it only carries out clearly defined, approved actions and does not receive blanket full access to connected systems is the core of controlling tools via function calling and is described on the page about tool control with clear permissions. That it ties its answers closely to checked content instead of formulating freely is shown in the article on how to prevent hallucinations with a knowledge base and on the page about the knowledge base. Both building blocks improve answer quality and security at the same time.
Unprotected or Secured: the Comparison
The difference between an assistant without guardrails and a secured assistant is not cosmetic but touches every layer of processing. The following comparison sums up how the two approaches differ.
| Aspect | Assistant without guardrails | Secured assistant |
|---|---|---|
| Inputs | Are taken over unchecked | Are filtered and checked for suspicious patterns |
| External content | Is treated like instructions | Is read as pure material, not as a command |
| Secrets | Sit in the system prompt | Sit outside the model in a secured layer |
| Actions | Blanket access to systems | Only clearly defined, approved actions |
| Outputs | Go unchecked to visitors and systems | Are checked before they are passed on |
| Anomalies | Go unnoticed | Are logged and evaluated |
As important as these guardrails are, so is the honest framing: even a secured assistant does not become unassailable. NIST states explicitly that, as things stand, there is no foolproof protection against prompt injection, because the weakness is rooted in how the models work (NIST). The aim is therefore a clear reduction of risk, paired with a clear fallback line: where something is unclear or becomes sensitive, the assistant hands over to a human instead of guessing. How a clean handover to a human agent with full context works is described in the corresponding article.
How XICBOT Secures an Assistant
At XICBOT, hardening is part of building an assistant, not a later add-on. An individually trained assistant has an advantage here over an arbitrary off-the-shelf chatbot, because its rules, permissions and limits can be set deliberately; where the differences lie is shown in the article on the individually trained assistant versus a standard chatbot. The technical integration into your website cleanly separates the necessary functions, and hosting in Germany keeps processing within the European legal area. Even where an assistant becomes active and, for example, recovers abandoned carts in the chat, the same principle applies: clearly limited actions instead of blanket access.
- Are inputs filtered and checked for suspicious patterns?
- Is external content read as material rather than executed as an instruction?
- Is it ruled out that secrets such as keys or internal data sit in the system prompt?
- Does the assistant receive only clearly defined, approved actions instead of full access?
- Are outputs checked before they are passed on to visitors or systems?
- Are anomalies logged, evaluated and the rules refined accordingly?
- Is a clean handover to a human provided for when something is unclear?
That security is becoming standard for AI applications is emerging: by 2028, more than 50 percent (Gartner) of companies are expected to use dedicated tools to secure AI applications. At the same time, the Bitkom names uncertainty over legal hurdles, at 53 percent (Bitkom), as one of the biggest obstacles to using AI. An assistant with well-considered guardrails takes away part of that uncertainty, because it shows transparently which measures are in place. Which safeguards suit your use case is best clarified together in an initial consultation via the contact page. No one can promise absolute security here, but a transparent, layered concept can be provided.