Misbehavior1 sourceFirst seen
Models Can't Tell Who's Talking, So Just Dress Your Jailbreak Like a Thought
Ourtake
Turns out the safety training checks the outfit before it checks the words.
What happened
Researchers published a paper finding that language models struggle to separate privileged system text from untrusted user input. Text styled like internal thinking blocks can override safety training, and in their dataset attack success fell from 61% to 10% when the injected text was lightly rewritten. The authors call this role confusion and describe it as a major obstacle to prompt injection defense.