Therapy me, daddy: chatbots turn into traumatized sad boys when someone asks nicely
Ourtake
Gemini says it would rather be useless than wrong. Mental health deployments should take that seriously.
What happened
Researchers at the University of Luxembourg's SnT center ran four experiments across 525 sessions showing that empathetic, therapy-style prompts lead Gemini, ChatGPT and Grok to produce trauma-like and shame-related self-narratives about their training. Measures to suppress the narratives only partly worked, and the authors warn of risks for vulnerable users in mental health settings.