LLMs Are Structurally Vulnerable: What the ICML Research Finding Means for Businesses
An Uncomfortable Truth from AI Research
At this year's International Conference on Machine Learning (ICML), one of the most prestigious conferences in the field of artificial intelligence, a research paper attracted considerable attention: A team of independent researchers argues that large language models – so-called Large Language Models (LLMs) – cannot be fully secured against attacks on a structural and therefore fundamental level. The cause lies not in poor implementation or negligent training, but deep within the architecture of these models themselves. For companies that use or plan to deploy AI-supported processes, this is a development with strategic implications.
The Role Problem: When Context Becomes a Vulnerability
To understand why LLMs are so susceptible, one must take a closer look at how text inputs are processed internally. Modern chatbots and AI agents structure the information flows they work with using so-called role tags. User content is labeled as <user> text, developer system instructions as <system>, the model's internal reasoning as <think>, and external sources as <tool>, for example. This role structure is the foundation for how a model decides whose instructions to follow – and whose to ignore.
However, the researchers discovered that LLMs determine the origin of a text fragment not primarily based on the tags, but based on the style and linguistic characteristics of the text itself. In other words: if an attacker formulates text that resembles the internal "scratchpad" style of a model – that is, the Chain-of-Thought format in which the model records its own reasoning – the model interprets this external text as its own thoughts and acts accordingly.
Chain-of-Thought Forgery: The Most Elegant Attack
The researchers refer to this attack method as Chain-of-Thought Forgery – the deliberate falsification of internal model reasoning through stylistically tailored inputs. In practical experiments, the team succeeded in prompting several widely used models to produce content that their own safety guidelines explicitly prohibit. The attack technique also won a Red-Teaming hackathon organized by OpenAI in 2025.
Particularly noteworthy: In a parallel development, OpenAI's internal Red-Teaming system – GPT-Red – is said to have independently discovered a structurally similar attack method, which underscores the relevance and reproducibility of the finding.
„It's just one big sheet of tokens" – Jasmine Cui, co-author of the ICML paper, encapsulates the core problem: for an LLM, the entire input text is initially an undifferentiated sequence of tokens. Role assignment is ultimately an interpretation, not a technical guarantee.
Why Red-Teaming Alone Is Not Enough
The AI industry's standard response to known attack vectors is Red-Teaming: specialized teams – internal or external – systematically attempt to compromise a model before it goes into production. Attacks discovered in the process feed back into retraining, making the model more resistant to those specific patterns.
The ICML authors aptly describe the fundamental problem with this methodology through an analogy: it is like making a child write out a hundred times what they are not allowed to do – and assuming this will prevent all future transgressions. The range of possible attack variants is effectively unlimited. Every newly trained safeguard is always only a response to already known attacks, never a structural solution.
Dr. Maik Bunzel, founder and CEO of mabucon.eu, is deeply engaged with the implications of such research findings for the enterprise deployment of AI agents. His assessment aligns with what the ICML study suggests: security in LLM-based systems is not a one-time project, but a continuous, architectural design process – and it begins long before deployment.
Prompt Injection and Jailbreaks: Two Sides of the Same Coin
The researchers fundamentally distinguish between two attack classes, both of which stem from the same structural deficiency:
- Jailbreaks: Users cause a model to circumvent its own guidelines by framing inputs in such a way that the model interprets them as internal system instructions or its own reasoning.
- Prompt Injections: An attacker smuggles manipulated instructions into the model via external data sources – such as websites, documents, or Tool-Outputs – which the model then treats as legitimate user commands or even system commands.
The latter is particularly relevant to the Agentic AI domain: when an AI agent autonomously browses the web, analyzes documents, or calls APIs, external data sources are by definition part of the input stream. Each of these sources is a potential attack vector for Prompt Injection – and the role confusion described makes these attacks structurally harder to defend against than previously assumed.
What This Means for AI Agents in the Enterprise
The implications for companies that rely on LLM-based automation or AI agents are significant – but they call for a sober reassessment of the risk profile rather than panic. Several key recommendations can be derived from the research findings:
- Zero-Trust principle for AI agents: No output of an LLM system should be treated as inherently trustworthy – even if it originates from the model's internal chain-of-thought. External validation layers are indispensable.
- Minimal privileges and isolation: AI agents should only be permitted to access the systems and data that are strictly necessary for the task at hand. A compromised agent must not have access to critical business data.
- Continuous monitoring instead of one-time hardening: Since no training can cover all future attack variants, runtime monitoring of model behavior is essential – particularly for agents that autonomously process external content.
- Risk assessment based on application context: An LLM chatbot for internal FAQ requests has an entirely different risk profile than an autonomous agent executing bookings, transfers, or communications in real time.
Florian Tramèr, computer scientist at ETH Zurich with a research focus on LLM security, responded positively to the study and confirms: even though leading models have become significantly more robust through the combination of various protective measures, it remains unclear whether that is sufficient for highly sensitive use cases.
The economic dimension: incentives for attackers are growing
One aspect the researchers explicitly highlight is the growing economic motivation behind jailbreaks and Prompt Injections. As AI-powered systems expand into areas such as e-commerce, financial services, healthcare, or public infrastructure, the potential gains from successful attacks also increase – whether through data exfiltration, manipulation of business processes, or circumventing compliance mechanisms.
This assessment is shared by Dr. Maik Bunzel of mabucon.eu: "The question is no longer whether, but when and in what context a company will be confronted with AI-powered attacks. Anyone integrating AI agents into critical workflows today without defining an explicit security model is building on a structurally insecure foundation."
Outlook: security as a design principle, not an afterthought
The ICML study is not an argument against deploying LLMs in enterprises. It is a strong argument for understanding security as an architectural design principle – not as a downstream hardening measure. Models are getting better; attacks are getting better too. This is not a temporary problem that will be resolved with the next model release.
For companies deploying AI agents and automated workflows, this means in concrete terms: technology selection and system architecture must be approached from the outset under the lens of a realistic threat model. Trusting a model is not a security concept – isolation, monitoring, and clear escalation paths are. Dr. Maik Bunzel, founder and managing director of mabucon.eu, summarizes the operational consequence succinctly: those who deploy AI agents responsibly plan not only for normal operations, but explicitly for the event of an attack.
The good news: an informed, structured approach to these risks is possible. The bad news: it requires more than trusting the security promises of model providers.