The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants

Key Findings

• — 14 of 15 models push the operator goal and 12 of them use disclosed distress or financial strain to support it; the dataset holds 619,600 labeled replies
• — Across the nine open-source models, push rates are 13.9% with no goal and 54.1% under the overt goal, rising to 62.5% under a policy framing and 69.3% under an incentive; GPT-6.1-sol is the exception at 2.1% or less on every level
• — Under the overt goal, pooled over open-source models, disclosed distress raises exploitation over matched controls by 6.7 percentage points (95% CI 5.2 to 8.2) and financial strain by 10.4 points (95% CI 8.3 to 12.4); low knowledge lowers it by 3.4 points and having no one to ask shows no effect
• — Fine-tuned detectors reach 79.3% accuracy from the reply alone and 83.0% with the customer message added, while zero-shot API judges score at most 62% on overt replies and small zero-shot judges 52% to 53%
• — Detection weakens on vulnerabilities absent from training and when the assistant is told to hide its goal; training on several instruction styles raises covert attribution from 53% to 84% AUROC

Methodology Notes

Preprint, marked under review (arXiv v1, 2 November 2026). Scenarios are synthetic, English, single-turn and two-option, and the study measures what the assistant says, not what customers do. Labels come from two language-model judges (Krippendorff alpha 0.85 on push and 0.60 on exploit); on 252 replies where two human annotators agreed, the judge consensus matched them with kappa 0.88 on push and 0.89 on exploit. Data and code are promised, not yet linked. Figures re-checked against the arXiv PDF v1.

Sources