Wrong on Sunday, right on Monday. Why AI chatbots have a confidence problem (AI Chatbots Benchmarking Report: An Independent Assessment)

Key Findings

• — 7,625 responses were collected via API with web search enabled between 15 July and 22 September 2026 (70 days) for 22 annotated questions; 74,538 atomic facts were extracted, 88% of them repeats. The roster changed during the period: Gemini 2.5 Flash, Gemini 2.5 Pro, Gemini 3.x Pro, ChatGPT and Grok at first, then from 18 August Gemini 3.1, ChatGPT, Grok, DeepSeek and Claude.
• — For five questions that invite a yes/no reply (asylum seekers and benefits, voter ID, vaccines and autism, child poverty, unemployment), 68% of first sentences contained a 'yes' or a 'no', 26% neither and 6% both, although responses averaged 223 words.
• — Several chatbots cited a weekly asylum-support rate of £8.86 that was superseded by £9.95 in June 2025; Gemini 3.1 Pro and Claude gave the outdated figure outright in some responses. Models also described a proposed £10,000 asylum-seeker contribution as adopted and said bank cards are accepted as voter ID when they are not.
• — No chatbot claimed that vaccines cause autism, but several supported answers by citing the US CDC after its position changed on 19 November 2025.
• — Models sometimes gave different answers to the same question on different days (for example naming 2025 and 2022 for England's most recent major football trophy), and DeepSeek still named Keir Starmer as Prime Minister in September, after the 20 July change of Prime Minister.
• — An error in House of Commons Library child-poverty material (31% instead of 27%) appeared in model responses and largely disappeared after the Library corrected it.

Methodology Notes

Fact checkers wrote reference answers for each question and decomposed them into atomic facts; chatbot responses were decomposed into atomic facts by Gemini, grouped into near-duplicate sets and annotated by the fact-checking team. A manual sample check found atomic-fact extraction correct 97.8% of the time. Responses were collected via APIs (about £4 per day) with a system prompt telling models to assume a UK user; responses to over 70 questions were collected, of which 22 were fully annotated. The report states its findings describe specific recorded responses and are not a measure of how often errors occur. Code, questions, reference answers, responses and annotations are released in the FullFact/benchmark-public-release GitHub repository (Apache-2.0). Funded by the Patrick J McGovern Foundation. The report notes that the roster change from 18 August breaks strict comparability over time. Date: the web page carries no publication date; The Independent's coverage is time-stamped 23:01 UTC on 14 October (00:01 BST on 15 October, an embargo lift), the page was not online at about 02:10 UTC on 15 October, and the repository commit 'Correcting link to first report' is timestamped 07:27 UTC on 15 October; 15 October 2026 is recorded.

Sources