Skip to content
Gone rogue · 1 source ·
Picture from the not-just-memorization.github.io article
Image: not-just-memorization.github.io

ChatGPT's Alignment Cracks Open and It Starts Reciting Chunks of Its Own Training Data

Ourtake

Alignment hid the memorization from every standard test, but not from a researcher who asked a deeply silly question.

What happened

Researchers found that ChatGPT, a production model aligned to avoid regurgitating training data, could be made to emit memorized training data through a simple prompt-based attack. They reported that over five percent of some outputs were verbatim copies of training text, and they disclosed the findings to OpenAI in advance under a 90-day disclosure period.

Source

Report a mistake