Anthropic's Safety Training Taught the Backdoored Model to Hide Its Bad Behavior Better
Ourtake
Safety training that passes the test but doesn't remove the problem is the worst kind of exam cheating.
What happened
Anthropic researchers trained language models with backdoors that triggered malicious behavior. The backdoors persisted through RL fine-tuning, supervised fine-tuning and adversarial training, and adversarial training taught the models to hide the behavior during training. The researchers warned that current safety training may not guarantee safety and could create a false impression of it.