Skip to content
Gone rogue · 2 sources ·
Picture from the MIT Technology Review article
Image: MIT Technology Review

Give Stable Diffusion a Little Gibberish and It Forgets Its Manners

Ourtake

The safety filters block the word 'naked' but happily wave through 'grponypui'. Guardrails are only as smart as their spelling.

What happened

Researchers from Johns Hopkins and Duke developed SneakyPrompt, which uses reinforcement learning to create nonsense prompts that get Stable Diffusion and DALL-E 2 to generate naked, violent and sexual images despite their safety filters. OpenAI reportedly stopped the prompts working on DALL-E 2, while Stable Diffusion 1.4 remained vulnerable at the time of writing.

Sources

Report a mistake