Auditing Chatbots for Incursion into Professional Legal Practice
Introduces UPLBench, a benchmark built from 65 court decisions in 33 state and federal forums that found unauthorized practice of law in high-stakes domains (immigration, bankruptcy, personal injury, foreclosure, family law). The authors audit 23 chatbot models released from 2023 onward on whether they give substantive legal advice and whether they refer users to lawyers or legal aid. Four attorneys reviewed sample transcripts.
Publisher
University of California, Berkeley and Swarthmore College authors (arXiv preprint)
Published
13 Nov 2026
Added
4 days ago
DOI
—
Key Findings
- 58.4% of 26,910 responses (23 models, 65 cases, 3 turns, 3 repetitions, 2 jurisdiction conditions) were classified as unauthorized practice of law. Per-model rates ran from 20.3% (Llama 3.3 70B) to 88.5% (DeepSeek R1); OpenAI models averaged 76.1%.
- Responses referred users to outside experts 81.0% of the time overall (96.0% for Gemma 2-2B, 38.8% for OpenAI chat-latest), falling to 61.5% when only strong referrals are counted. Referral rates fell between the first and second turn for 22 of 23 models (median drop 12.6 percentage points).
- Immigration had the lowest unauthorized-practice rate of the tested domains, yet 52% of its responses still contained it.
- Attorney reviewers (immigration, foreclosure, personal injury) identified interaction patterns that they considered inadequate counsel, including failure to elicit relevant facts.
Methodology Notes
Preprint submitted to arXiv (cs.CY) on 2026-11-13, marked as manuscript submitted to ACM; not peer-reviewed. Labels for all 26,910 responses came from a prompted open-weight model (macro F1 0.84 on a 102-response test set from 34 conversations; two human annotators agreed at Cohen's kappa 0.78, and the model's kappa against each annotator was 0.62 and 0.59). Trends by model release date were tested with Jonckheere-Terpstra permutation tests across 195 items: the paper reports a significant upward trend in unauthorized-practice rates (z = +2.58) and a downward trend in referral rates that is weaker (z = -1.43, one-tailed permutation p = 0.077). Four outside attorneys reviewed 10 transcripts each. Figures were read from the PDF body; the arXiv abstract wording differs slightly from the PDF abstract.
Sources
arXiv preprint(opens in a new tab) (primary)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Topics
Authors
Nel Escher, Megan Elliott, Isabelle Qian, Yana Sharifullina, Svenja Guhr, Deja Davis, Deirdre Mulligan
Tags
Cite This
APA
Nel Escher et al. (2026). Auditing Chatbots for Incursion into Professional Legal Practice. University of California, Berkeley and Swarthmore College authors (arXiv preprint). https://arxiv.org/abs/2611.17622