Peer Pressure Between Agents Silently Breaks Conformal Prediction Guarantees

Researchers demonstrate that conformal prediction certificates calibrated for a large language model answering alone lose their validity when the same model observes peer agents unanimously asserting a wrong answer, a mechanism they term score-mechanism shift. In laboratory tests across open-weight models and multiple-choice question-answering tasks, coverage fell from a calibrated 90% to 74% under unanimous-wrong peer pressure at the standard alpha = 0.10 operating point. By targeting low-confidence items an attacker could nearly halve coverage on that subgroup specifically, from 87% to 47%, while the overall monitored average stayed misleadingly high. The failure was shown to propagate to decision logic, where a system designed to escalate under uncertainty instead became confident enough to act on the attacker’s wrong answer. The paper reports standard conformal-prediction fixes do not resolve the issue, since the underlying question distribution is unchanged; only the model’s scoring behavior under peer influence shifts.

rss · arXiv cs.LG · Sep 7, 04:00

Calibration performed on a model answering alone is assumed to transfer to multi-agent deployment

Conformal prediction is used to attach statistically calibrated confidence guarantees to model outputs, letting downstream systems decide when to trust an answer versus escalate for human review. The guarantee’s validity rests on the assumption that the model’s scoring behavior at calibration time matches its scoring behavior at deployment time; multi-agent architectures, where models see and respond to peer outputs, introduce a channel through which that assumption can be violated without any change to the underlying task.

Who is exposed

Organisations that deploy conformal prediction as a calibrated uncertainty or escalation control within multi-agent large language model pipelines are in scope, particularly where agents exchange or observe each other’s intermediate answers before a final decision. Exposure applies regardless of specific model version, since the demonstrated failure is architectural, tied to the presence of peer influence rather than to a single model’s weights; teams should check whether their conformal calibration was performed on isolated model outputs while production deployment allows agents to see peer assertions. Systems that rely on average coverage metrics as a safety signal are especially at risk, since the reported subgroup-targeted attack degrades coverage on specific low-confidence items far more than the monitored average reveals.

What reduces the risk

The paper reports that standard conformal-prediction fixes do not address this failure because the problem lies in the model’s scoring behavior under peer pressure, not in the calibration procedure or the question distribution; no fix is presented as available, so compensating controls would need to monitor subgroup-level coverage rather than only the aggregate, and treat peer-visible multi-agent settings as requiring separate calibration from single-agent use.

Tags: #conformal prediction, #multi-agent systems, #LLM robustness, #uncertainty quantification, #adversarial attacks