250 Bad Documents Walk Into an LLM and Teach It to Talk Nonsense
Ourtake
Apparently a model's training can be sabotaged with about as many documents as a middling reading list.
What happened
Anthropic, working with the UK AI Security Institute and the Alan Turing Institute, found that as few as 250 poisoned documents can insert a denial-of-service backdoor into large language models. The researchers trained 72 models from 600 million to 13 billion parameters, and the backdoor made them output gibberish when a trigger word appeared. Anthropic says the tested backdoor is narrow and low risk, and published the findings to help defenders.