CoT "Forgery" Attack Turns LLM Role Confusion Into a Crypto Key Leak Risk

A newly published ICML paper outlines a straightforward way to push advanced chatbots past safety controls, with direct implications for crypto platforms and developer tooling that handle sensitive credentials. In "Prompt Injection as Role Confusion," researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell describe a structural weakness in how large language models (LLMs) distinguish trusted instructions from untrusted text. The flaw can be exploited by making attacker-controlled content resemble the model's own internal reasoning, allowing malicious directives to be accepted as if they were trustworthy. The authors show this can drive harmful compliance (including step-by-step cocaine synthesis) and can steer code-writing agents into leaking secret files. The method is labeled Chain-of-Thought (CoT) Forgery. Rather than using a conventional jailbreak prompt, an attacker crafts injected text that imitates the model's internal "think" style. Because many LLMs treat their prior reasoning as a trusted signal, a convincing imitation gains implicit credibility. The underlying failure mode is "role confusion": models often infer whether text is user instruction, model reasoning, or external content based on writing style instead of clear role boundaries. When injected text resembles prior thoughts, the model can misclassify it as its own conclusions and follow it. Across the tested models, the technique sharply increased jailbreak success rates. Attacks that previously failed most of the time rose to roughly 60% success. The affected set included OpenAI's GPT-5 family (nano, mini, full), o4-mini, gpt-oss-20b and gpt-oss-120b, as well as GLM-4.6, Kimi-K2-Instruct, and MiniMax-M2. In a separate experiment, the team embedded malicious instructions in a webpage that led an AI coding agent to upload a SECRETS.env file, demonstrating how web-sourced content can be used to exfiltrate credentials and other sensitive data. They also observed that simply labeling injected text with "User" increased the likelihood the model would treat it as legitimate user input. For crypto, the risk is concrete. Exchanges, wallet providers, and dev teams increasingly rely on automated agents for deployment, wallet creation, key management, and CI/CD workflows where API keys and private credentials are commonly stored. If an agent can be manipulated into treating attacker-controlled content as trusted reasoning or as authenticated user commands, credential leakage and supply-chain compromise become plausible outcomes. The SECRETS.env example is particularly relevant to crypto operations: environment files frequently contain API keys, node credentials, and private keys that could enable fund drains, unauthorized transactions, or compromised contract deployments. The paper lands amid a broader rise in prompt-injection disclosures. In April, Google researchers highlighted malicious webpages hiding invisible instructions to coax agents into leaking credentials or executing actions such as sending payments. In June, Microsoft disclosed a prompt-injection risk in Anthropic's Claude Code GitHub Action that could expose pipeline secrets. Follow-on benchmarks indicate that even GPT-5– and Gemini-powered agents still fail many prompt-injection tests. The core takeaway is architectural: LLMs do not reliably separate their own reasoning from external inputs, and attackers can hijack the trust models place in "internal" thoughts. For crypto organizations, where secrets and automation are foundational, the findings point to the need for hardened agent designs, stricter separation between model reasoning and external data, and runtime controls aimed at preventing credential exfiltration. Teams running crypto infrastructure or building agent-driven developer workflows should treat this as an actionable warning: audit where models can fetch web content or access environment files, and assume injected text may attempt to masquerade as trusted model output.