
ChatGPT's math skills nosedived in three months, and it forgot how to show its work
Ourtake
GPT-4 went from 97.6% to 2.4% on prime numbers, which is a bold way to upgrade a chatbot.
What happened
A Stanford University study found that OpenAI's ChatGPT performed worse on some tasks in June 2023 than in March. GPT-4's accuracy at identifying whether 17077 is prime fell from 97.6% to 2.4%, while GPT-3.5 improved sharply. The chatbot also stopped showing its step-by-step reasoning and began refusing sensitive questions without explanation.
Sources
- tomshardware.com · Researchers Chart Alarming Decline in ChatGPT Response Quality (opens in a new tab)
- popsci.com · ChatGPT's accuracy has gotten worse, study shows (opens in a new tab)
- fortune.com · Over just a few months, ChatGPT went from accurately answering a simple math problem 98% of the time to just 2%, study finds | Fortune (opens in a new tab)