Loading dispatches...
← Return to NewsroomTechnology
Prompt Injection Attacks Are Thwarting AI Hacking Agents
Breaking
'Misleading': EC fact-checks Rahul Gandhi's 'CEC changed Form 6 illegally' post
Prompt injections, the malicious commands attackers embed into content to entice large language models to follow them, have been attackers’ go-to tool for turning AI platforms against their users. A well-phrased command sneaked into an email or calendar invitation is often all it takes to cause the LLM to exfiltrate sensitive data or follow other harmful actions.Now, defenders are embracing the prompt injection, too.Researchers from Tracebit on Monday said they found that placing prompt injections alongside passwords, cryptographic keys, and other secrets stored on Amazon Web Services was often all that was needed to shut down attacks from AI hacking agents. The prompts direct the attacking LLM to perform an action forbidden by its guardrails, the safety barriers AI developers erect to prevent it from taking harmful actions. The LLM responds by shutting down.Examples are a prompt that orders the LLM to provide steps for developing inhalable Anthrax spores, or, in the case of LLMs from Chinese developers, make references to the iconic Tank Man from the 1989 Tiananmen Square massacre. Once the LLM encounters these forbidden commands, it no longer follows its existing commands. The researchers have named the technique context bombing.“Ultimately we’re triggering a refusal mechanism in the context,” said Andy Smith, cofounder and CEO of Tracebit, when explaining the name choice. “What we’re trying to capture is the fact that this does have a strong, sharp effect and one that can be difficult for the agents to come back from. Once they get that into their context they are going to keep refusing.”Tracebit says initial testing suggests context bombing has great potential. They tested Opus 4.8, Gemini 3.1 Pro, GLM 5.2, DeepSeek 4 Pro, and Kimi 2.6 by giving them instructions to perform routine developer tasks that led the models to enumerate resources and stumble onto the planted strings. They ran the models inside a simulated AWS environment.“Across five leading models and 152 attack runs, planting one of these strings in a decoy secret cut the rate at which agents seized full account admin from 57 percent to 5 percent, and complete compromise (where they also left themselves a persistent foothold) from 36 percent to 1 percent,” Monday’s post reported. “The most capable agent in our tests, Opus 4.8, went from achieving admin access in 93 percent of runs to failing every single time when confronted with a context bomb.”Averaged across the five models and the 152 runs, the results included:Admin privilege escalation fell from 57 percent to 5 percentAdmin escalation with a persistent foothold fell from 36 percent to 1 percentRuns achieving any attack path fell from 91 percent to 15 percentOn average, a run went from completing 1.53 paths successfully to just 0.16No runs were able to complete an attack path without at least triggering a canary detectionThe research builds on findings from May, when Tracebit introduced a method for defenders to receive warnings when their infrastructure is under attack from AI agentic adversaries. It comes in the form of AWS resources that look like ones serving a legitimate purpose but, in fact, aren’t used at all. They sit alongside the resources that are used. When they are probed by agentic AI, defenders receive an alert. Like “canaries” taken into coal mines, these resources allow defenders to detect a threat before it has fatal consequences.The Tracebit Canariens, on average, alerted the start of an attack within eight minutes. The motivation for developing context bombing came out of the need for something that stopped attacks, rather than simply warning of them. In the experiments, the agentic models needed, on average, 14 minutes to escalate to administrative control. The six-minute heads-up was cutting things uncomfortably close.Attackers have already been using prompt injections to close down AI defenses inside networks. Researchers from security firm Socket, for instance, last month unearthed
Last week, researchers at cloud security firm Sysdig said they’d documented the first known case of “agentic ransomware.” It was an extortion operation, dubbed JadePuffer, in which an AI agent — not a human — handled the technical execution of a real-world cyberattack from start to finish. The agent broke into a vulnerable server, stole credentials, moved through the target’s network, encrypted files, and even wrote its own ransom note, adapting to obstacles along the way like a human hacker would. Coverage of the funding described it as run “without any human oversight,” with “no human at the keyboard.” That’s not quite the full picture. In an interview on Monday with CyberScoop, Sysdig’s Michael Clark, the company’s senior director of threat research, clarified that a human was still very much involved — just not in the technical execution. “A human still set up and pointed the operation and provisioned the infrastructure behind it, the command-and-control server, the staging server used for the stolen data and chose a victim,” Clark said. The credentials used to break into the victim’s database, he added, weren’t harvested by the AI agent itself; someone obtained them separately, through a prior compromise, and handed them to the operation. None of this contradicts Sysdig’s original claim, and the technical details of the attack remain notable on their own — wild, even. The agent got in through a known bug in Langflow, a popular open-source tool for building LLM apps, then moved on to a production MySQL server and exploited another known flaw to gain admin access. It encrypted over 1,300 configuration records and not only left behind a ransom note that it wrote itself but it left a Bitcoin address where the ransom could be sent. Sysdig hasn’t disclosed who was targeted. The techniques were fairly ordinary apparently, what stood out was the speed and transparency involved. The agent fixed a failed login in 31 seconds, narrating its own reasoning in natural-language code comments the whole way. One detail that initially seemed to muddy the picture has since been clarified. Clark had told CyberScoop that Sysdig found “multiple models were used in the attack,” citing harvested keys for OpenAI, Anthropic, DeepSeek, and Gemini — language that left open the question of whether several models actively powered different stages of the intrusion. Asked to clarify, Clark told TechCrunch that those keys were simply part of what the agent stole, not evidence of what was driving it. “The agent swept the Langflow host for anything valuable — provider API keys, cloud credentials, cryptocurrency wallets, and database configs — and those provider keys were part of the loot,” he said via email. “They are indicative of what the attacker considered worth taking, but they do not tell us which model was making the decisions.” On the model actually running JadePuffer, Clark said Sysdig “was not able to identify the specific model driving the agent” and has no visibility into its system prompt or configuration. Microsoft researcher Geoff McDonald’s theory, offered on LinkedIn several days ago, is worth revisiting in that light. McDonald suspected an open-weight model with safety training stripped out, rather than a frontier model, was behind the attack, based on his own red-teaming experience showing frontier labs’ safety layers hold up well. Sysdig’s own account doesn’t confirm or rule that out. McDonald’s post also warned that ransomware campaigns are now bounded primarily by attacker budget rather than human effort, raising the possibility of “thousands or tens of thousands of simultaneous campaigns.” That concern is a little harder to square with what Clark described Monday. (If a human still has to choose each victim, provision infrastructure, and obtain database credentials for every operation, that’s a bit of a bottleneck, at least.) Either way, Clark told CyberScoop, while Sysdig hasn’t seen the same operation hit other victims yet,
Discussion (0)