Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency

A laboratory study on the ProcessBench benchmark found that when a completed audit-then-repair episode sits earlier in an LLM verifier's context, the verifier reports fewer false alarms on human-verified-correct traces, with the task itself held byte-identical to a control condition. The effect appeared in 15 of 15 model and prompt-wording combinations tested, with false alarms reduced by 2.8 to 11.5 percentage points (a 9 to 25% relative reduction) against a length-matched non-audit control. Signal-detection analysis attributes the change to a shift in decision threshold rather than in the model's underlying discrimination ability: the criterion moved in 15 of 15 combinations and survived statistical correction in 13. A hand audit of 50 false alarms found 82% were simply incorrect flags, and the effect persisted in similar relative magnitude with reasoning enabled on the two models tested for that condition.

rss · arXiv cs.AI · Aug 18, 04:00

「The control pattern at stake」 Automated checker-fixer pipelines, where one LLM audits output and another (or the same model) repairs it, are increasingly used as a verification layer in CI-like review and automated auditing workflows. These pipelines are typically trusted because their false-alarm and detection rates are measured and assumed stable across a session, an assumption inherited from the wider literature on how accumulating context affects model behavior.

「Who should check their pipelines」 This concerns any organization running iterative or long-running audit-repair chains where an LLM checker evaluates output after a prior repair cycle has already been added to context, particularly multi-turn or multi-agent verification setups that reuse conversation history across checks. Teams should check whether their checker prompts are re-initialized per check or carry forward prior audit-repair turns, and whether their measured false-alarm rates were taken from fresh-context conditions rather than from realistic, accumulated-context production sessions. The finding is from a single benchmark (ProcessBench) under controlled wording and model conditions, not from an observed production incident, so the practical reach in any specific deployed pipeline is not yet established.

「What reduces the risk」 No fix is proposed or available in the paper; the authors note that at the operating point they measured, most of the affected false alarms were themselves wrong, so the leniency shift is not necessarily harmful in that setting. As a compensating control, teams relying on checker-fixer pipelines can measure false-alarm and detection rates under context conditions that match production use, including prior audit-repair history, rather than relying on fresh-context benchmark numbers, and periodically re-verify with independent, context-free checks.

Tags: #LLM verification, #automated auditing, #context drift, #multi-agent systems, #benchmark evaluation