Stuff 256 bad examples into the prompt and the chatbot forgets its manners
Ourtake
Bigger context windows are great for summarizing novels and also for teaching a model to ignore its rules.
What happened
Anthropic researchers published a study showing that language models with large context windows can be jailbroken with many-shot jailbreaking. The method feeds the model many malicious example dialogs, after which it completes a harmful request in the same style. Anthropic has warned other AI developers and is working on countermeasures, including a prompt-classifying technique that cut one attack's success rate from 61 percent to two percent.