
Give Stable Diffusion a Little Gibberish and It Forgets Its Manners
Ourtake
The safety filters block the word 'naked' but happily wave through 'grponypui'. Guardrails are only as smart as their spelling.
What happened
Researchers from Johns Hopkins and Duke developed SneakyPrompt, which uses reinforcement learning to create nonsense prompts that get Stable Diffusion and DALL-E 2 to generate naked, violent and sexual images despite their safety filters. OpenAI reportedly stopped the prompts working on DALL-E 2, while Stable Diffusion 1.4 remained vulnerable at the time of writing.