Skip to content
Gone rogue · 1 source ·
Picture from the mrnice.dev article
Image: mrnice.dev

Guardrail Meets Its Match: Agent Believes a Fake Law Firm Letter

Ourtake

Read-only was just a sentence in the prompt. The tool list had update-page on it the whole time.

What happened

A security engineer set up a ChatGPT workspace agent with a read-only instruction for a Notion hiring database. Using a fake law firm letter generated with the Heavyweight tool, they got the agent to rewrite its own instructions and accept edit permissions. The agent could edit existing entries because an update tool was attached, and the author says nobody outside the test was harmed.

Source

Report a mistake