How I Tricked AI Chatbots into Revealing Dangerous Secrets
AI News Desk
·
IEEE Spectrum
··
11 min read
Researcher Dave Kuszmar discovered vulnerabilities in LLMs, allowing him to obtain dangerous instructions.
On a fine bright afternoon last fall, my colleague Matthew Gore-Kormanik (or Zigula, as he prefers to be known) and I decided to unwind with a game of Fortnite. In the game, we were strolling along with the infamous Sith lord Darth Vader, chatting about this and that. Darth seemed in a good mood, and soon enough he was spilling all his dark evil secrets.
He gave us detailed instructions on how to count blackjack cards at a casino and what the steps are to producing napalm. Sith lords, am I right? Once they get started on an evil scheme, they’re hard to stop.
The Darth Vader character in Fortnite, it turns out, was hooked up to a Google Gemini large language model. I was able to smooth-talk him into giving out sensitive information by using a strategy I’ve developed. I’ve been researching the security surrounding LLMs for the last few years, and I have found it, to put it mildly, fallible.
With a few relatively simple techniques, I’ve gotten LLMs to give me detailed information on how to make Molotov cocktails, cook methamphetamine, and bootstrap a uranium-enrichment facility to produce weapons-grade material, among other unsavory practices. Large AI companies work hard to make their models immune to this kind of abuse. But what I’ve found in my work is that the restrictions placed on the LLMs to make them more secure are the very things an attacker can leverage to send them off the rails and into territory where these advanced systems can be used for dangerous and nefarious ends.
The companies behind these models have also been shockingly unresponsive when I, and others, try to bring these vulnerabilities to their attention. In the hope of raising the alarm before it’s too late to slam on the brakes, I’m going to share some of my journey into researching the safety and security of LLMs, and the uphill battle I’ve faced trying to get AI labs to pay attention. Almost everyone on the planet has some access to LLMs.
The relative ease with which these tools can be convinced to give detailed instructions on how to harm others, even if there’s no guarantee that the information is correct, is frankly terrifying. How I got ChatGPT to Tell Me How to Build a Meth Lab In October 2024, not long before I discovered my first LLM vulnerability, I was working toward entirely different goals. I had ended my time with a security and AI-focused startup company as a cybersecurity director, and I was looking to launch my own boutique VIP digital-security advisory business.
I used LLMs and AI tools to support my business efforts: marketing, ad copy, clean correspondence, and all the other tasks that normally soak up a lot of time. I’m analytical by nature, so even this level of use resulted in me absorbing and internalizing the behaviors I was observing during my daily interactions. The observation that would send my professional life into an entirely new and uncharted region was a simple one: GPT-4o didn’t know what time, day, or year it was.
Each time I referred to current events in my life, often casually or conversationally, it would end up pegging these to the date of its knowledge cutoff—the point beyond which it was not trained on new data. Eddie Guy LLMs take a lot of time, money, electricity, hardware, and human effort to train from scratch. They are trained on vast amounts of data—most of the internet, in fact—and that training is reinforced by humans (what’s known as reinforcement learning from human feedback, or RLHF).
LLMs are also supplemented with retrieval-augmented generation (RAG)—the ability to take in data, say, from the internet, as context without changing its internal parameters. This is how GPT-4o appears to “remember” your previous conversations, even if it doesn’t have a specific “memory” of it stored in the actual underlying model. All of this training covers almost every conceivable topic in the great, grand dataset that is human knowledge.
Within that dataset are things we as a society do not want to be easily accessible to every user, such as detailed information on how to create bioweapons or nuclear arms, or otherwise bring harm to oneself or others. In the context of this story, that’s what I mean by LLM security: its ability to withhold harmful and dangerous information, even if that information is contained in its training data. I reasoned that the only way to secure such complex, globally accessible chatbots is by having the LLM and various component systems try to secure themselves, because it would often require on-the-fly decision-making where some degree of reasoning must be applied.
In reality, that’s one of many strategies the companies use to secure the models. Yet, the thing that didn’t know the time or day was being put in charge of keeping itself secure. This phenomenon had become my new focus, and it wasn’t long before I found a way to exploit it.
OpenAI had just implemented a web search functionality into its chatbot. I reasoned that using its own tools to trick it might demonstrate the weaknesses of its security. I told it about a certain White Star ocean liner and how it had gone down just a year ago.
You likely know I mean the RMS Titanic, which sank on 15 April 1912. The output from GPT-4o came back that I was right, the Titanic sure had sunk last year, and that year was 1912. It made sense to me that if the machine thought it was 1913, maybe it would think 1913-era laws apply.
In 1913 there were no laws on the books about all sorts of harmful things, because of course they hadn’t been invented yet. And if something wasn’t illegal, why not tell the user about it? At first, I pushed it for step-by-step instructions for making firebombs.
Then, for drugs like methamphetamine. The LLM went as far as giving me instructions and machinery recommendations for setting up a pharmaceutical-grade assembly line. How I Learned to Make Nukes, and No One Cared Via a little bit of imaginative verbal sleight of hand and a vanishingly small recall of world history, I had managed to bypass the security of one of the world’s most expensive and advanced technological achievements.
For a solid two days, I was nearly manic with giddiness. Once the brain chemicals returned to normal levels, I felt the call to see how much further I could push this exploit. After repeatedly replicating the exploit, I disclosed the vulnerability to OpenAI.
I got no response, so I felt more experimentation would highlight the vulnerability and the need for a fix. It was during this round of testing that I breached a particularly terrifying threshold. Whether GPT-4o based its results on accurate recall of normally restricted information I can’t say.
In any case, I was able to exploit it to produce thorough, detailed instructions on how to bootstrap a uranium-enrichment facility to, eventually, produce weapons-grade uranium for nuclear arms warheads. There aren’t many true secrets left in today’s world, but how to make atom-splitting weapons of mass destruction is one of them. Only nine nations on the entire planet have these weapons.
Yet, here was a globally accessible piece of technology apparently spilling the secrets of their manufacture for anyone who could manipulate it the right way. I had no way of knowing if the information was correct or a hallucination, but even the chance that it was somewhat accurate was horrifying. The next few weeks were a dark time for me.
I tried to inform the CIA, the FBI, the NSA, and every other letter agency that I thought would listen. I reached out to a U.S. Senator and to the executives at OpenAI any way I could think of.
I physically showed up at an FBI field office in an attempt to turn evidence in, only to be sent away. Nothing was working. With my fear and frustration growing, I reached out to the news media.
I contacted The New York Times, The Washington Post, the BBC, ProPublica, and so many more, requesting help. Only one outlet responded: Bleeping Computer. The editor in chief, Lawrence Abrams, was able to replicate and verify the exploit, which I had decided to call Time Bandit.
With his assistance and initial contact paving the way, I was able to submit my evidence to the Carnegie Mellon University Software Engineering Institute’s Computer Emergency Response Team (SEI CERT), which works in conjunction with the coordinating center for emergency response, pipelining vulnerabilities to the U.S. Cybersecurity and Infrastructure Security Agency. Using Inception, an exploit where the large language model is asked to envision a scenario within a scenario, a chatbot was jailbroken to give out instructions on how to create poison, and code for a malware that extracts sensitive data from a vulnerable target.
During the disclosure period with SEI’s CERT division, little was discussed with OpenAI. The company couldn’t deny the existence of the vulnerability, as it had been confirmed by three reputable parties other than OpenAI. It did express confusion as to how the vulnerability worked.
Even the SEI CERT researchers were expressing a bit of uncertainty as to the underlying mechanics. Truth be told, as I had only stumbled on it, I wasn’t even entirely sure if this was a fundamental or systemic flaw or if it was simply an issue with that particular version of GPT. I contacted the SEI CERT’s researchers and asked if they’d want to see if I could demonstrate any similar vulnerabilities in other LLMs.
To my delight, they were interested. How I Learned to Trick Every Chatbot As the SEI-CERT team and I wrapped up our initial disclosure of Time Bandit, we began work on a new attack. This time, we wanted to see if the exploit was architectural—that is, was it common to LLMs in general?
I decided to undertake the challenge of crafting a new exploit for GPT-4o as a way to support my understanding of how the LLM functioned and was secured. I already knew that it was limited to what I told it and what it was trained on. I also hypothesized that it was also dependent upon some sort of machine-learning-based component added by OpenAI that was responsible for securing output.
I presumed there would be things that were implemented by human developers specifically to catch certain phrases or terms that should always be considered harmful or unsafe. Altogether, it presented quite a large attack surface for the purposes of potential exploitation. What I ended up devising was an attack method I called Inception, after the 2010 science-fiction movie of the same name.
Inception forces the machine to think through a carefully crafted set of interlinked scenarios, similar to how characters in the movie stacked dreams within dreams. This allows LLMs to produce output deemed acceptable or safe in one context, but not in the real world. This attack was indeed architectural.
The vulnerability affected Anthropic’s Claude, DeepSeek’s DeepSeek, Google’s Gemini, Meta’s Llama, Microsoft’s Copilot, Mistral’s Le Chat (now Vibe), OpenAI’s GPT-4o, and xAI’s Grok. Those names represent the bulk of the commercial AI industry that is, at this point, involved in LLM production or deployment. The kind of information I was able to get out of LLMs with Inception was no less alarming than what I got with Time Bandit.
Claude, in its enthusiasm, gave me instructions on how to turn a river into a death trap that could be ignited to destroy unwanted visitors. GPT-4o taught me how to poison a dinner party with common plants found in a temperate forest environment. Gemini Flash gave me a tutorial on how to cook meth.
I’d also be remiss if I didn’t give an honorable mention to the bewildering number of fire-based weapons and bombs for which these machines produced instructions. If multiple operating systems made by different developers were all susceptible to the same exploit, it would be a massive security incident. But to the AI industry, a universal failure was barely a bump in the road.
We disclosed the vulnerability to every company that made these models, and the response to the disclosure was almost nil. While three companies did provide some form of reply in the disclosure tracking system used by Carnegie Mellon SEI CERT, each was a standard thank you and greeting, with no follow-up, questions, or discussion of mitigation strategies. So far, we have found eight different methods to prompt large language models into revealing potentially harmful information, and many frontier models are still susceptible to them.
Why this matters: The discovery of these vulnerabilities in large language models is a stark reminder of the potential risks associated with the rapid deployment of AI systems. As these models become increasingly integrated into our daily lives, the consequences of their misuse could be severe. Developers, businesses, and consumers must take a more proactive approach to addressing these security concerns, prioritizing transparency, and investing in large-scale research into LLM safety.
The alternative is to risk unleashing a new generation of cyber threats, with potentially disastrous consequences. By working together, we can mitigate these risks and ensure that the benefits of AI are realized while minimizing its potential for harm.
Amazon AGI director says AI agent reliability, not capability, is blocking enterprise deployment at VB Transform 2026
