
Chatbots still recite the EU law nobody bothered to correct
Ourtake
Asked without documents, the models committed to the erroneous version in 45% of answers where they picked one.
What happened
A Zenodo-deposited measurement package by the Céphéide bench, from Loxyn, reports that language models often give the uncorrected version of EU acts and corrected RFC errata. Asked without documents, five models gave the erroneous version in 45% of the answers where they committed to one version. Version 1.3 adds a test of whether web search fixes 59 corrected EU-law errors.