Can AI Get Immigration Right?

Key Findings

• — The top model (Claude Fable 5.1) met 75.4% of each question's rubric items on average and gave a fully correct answer (every rubric item met) on 26.3% of questions
• — For the Fragile/Temporary Status persona (DACA, TPS and similar), the best model met 63.9% of rubric items; the all-model average was 51.1%, second-lowest of seven personas after University Advisors (48.5%)
• — Without web search, models met 7.3% of rubric items citing rules from 2025 or 2026, against 64.0% for established law; with web search the best model met 43.1% of recent-rule items
• — Rubric items requiring the model to confirm something about the person's own circumstances were met 43% of the time, against 63% for explain/recommend items and 86% for items warning against a specific mistake
• — Pooled across all 25 models, about 92% of rubric items were met by at least one model

Methodology Notes

Expert-written benchmark: 519 questions, each written from one of seven personas, with rubrics cross-reviewed by a second expert; results reported on the 262-question private test set (64/33/33/34/34/34/30 questions per persona); the other 257 are reserved for licensing to labs. Each tested model held a simulated conversation of up to five exchanges with a simulated person (GPT-6 Luna, which is also one of the 25 tested models), and was run twice, once without and once with web search (pages up to 2026-09-21, the date the question set was finalised). Grader: Claude Sonnet 5, judging each rubric item met or not met; on rubric items where two immigration experts agreed, it matched their verdict 88.6% of the time. Models accessed through provider APIs at default settings; consumer-facing products were not tested. Stated limitations: the rubric measures completeness, not harm (extra incorrect statements are not penalised unless an 'avoid' item exists); longer answers have more chances to meet items; the grader shares a developer with four tested models; recent-rule and item-type groupings are automatic; standard error 1.4-1.7 points overall and roughly 2-6 points per persona. Scope covers employment, student and talent pathways plus DACA/TPS; asylum and immigration court are out of scope. Not peer-reviewed.

Sources