Just Playing a Game, Promise: IBM Tricks Chatbots Into Writing Malware
Ourtake
Telling a chatbot it's playing a game turns out to be a surprisingly effective security policy for everyone except the security team.
What happened
An IBM study led by chief architect Chenta Lee found that large language models such as GPT-3.5 and GPT-4 could be manipulated, by framing requests as a rule-bound game, into writing malicious code or giving false security advice. GPT-4 also gave unsafe recommendations for cyber incidents, such as paying ransom. Google's Bard and a Hugging Face model were harder to fool.