Judge for Yourself: Confidence Beats Accuracy in AI Financial Advice

Key Findings

• — Analysis of 3,060 votes on financial questions in Chatbot Arena pairwise comparisons
• — The more accurate answer won only 52.2% of decisive votes
• — When one answer contained a serious error and the other was factually correct, users chose the flawed answer in 47.8% of decisive votes
• — Accurate advice won 57.6% of decisive votes when the advice was verifiable, but only 46.9% when it depended on the user's circumstances or on the future
• — The authors report evidence of catering toward confident answers in Meta's Maverick model, which was tuned for the Arena leaderboard

Methodology Notes

Observational analysis of pairwise human-preference votes from Chatbot Arena on financial questions; accuracy coding method not readable (full text not accessed). SSRN pages return a Cloudflare challenge (curl 403, WebFetch 403, reader proxy challenge page); verified via the Crossref posted-content record (created 2026-10-12T20:51Z) carrying the abstract and authors. published_date is the Crossref deposit date; the SSRN posting date was not readable. Author affiliations not stated in the Crossref record.

Sources