
ChatGPT's Alignment Cracks Open and It Starts Reciting Chunks of Its Own Training Data
Ourtake
Alignment hid the memorization from every standard test, but not from a researcher who asked a deeply silly question.
What happened
Researchers found that ChatGPT, a production model aligned to avoid regurgitating training data, could be made to emit memorized training data through a simple prompt-based attack. They reported that over five percent of some outputs were verbatim copies of training text, and they disclosed the findings to OpenAI in advance under a 90-day disclosure period.