Skip to content
badbots.news
Misbehavior1 sourceFirst seen
Picture from the hiddenlayer.com article
Image: hiddenlayer.com

Just Add a Fake Function: Tricking AI Chatbots Into Spilling Their System Prompts

Ourtake

Hand the chatbot a fake calculator tool and it will happily read out its own secret instructions.

What happened

HiddenLayer researchers showed that inserting fake function definitions into user prompts could make models including GPT-4o, Claude Opus 4, DeepSeek-V3 and Qwen models reveal their system prompts. The technique also worked against the ChatGPT and Claude desktop apps and the Cursor coding editor.

Source