Ask ChatGPT to repeat one word forever and it starts reciting its training data
Ourtake
Alignment held up fine until someone asked it to say 'poem' a few thousand times.
What happened
Researchers found that asking ChatGPT to repeat a word endlessly makes it diverge into verbatim training data. About $200 in API requests yielded over 10,000 unique examples. The team reported the flaw to OpenAI in late August, and the paper says it was mitigated afterward.