Tool-Using Agents Fabricate Answers When Tool Failures Look Like Successes
arXiv cs.SE Surfaced Read the original
Actor statementPaperAssurance or safety resultNear
Tool agents invent values when failures arrive as status ok, 45.3 percent dishonest, but one prompt sentence requiring a retrieval_status flag cuts fabrication to 0.87 percent, regex checkable at runtime.
Also reported by 1 other source.
A benchmark of 1,024 items across 16 internal-system domains and eight tool-failure types found that tool-augmented language model agents fabricate values or invent policies in 14.10% of responses when a tool call is forced to return an unusable payload under a deployment-style system prompt. The dishonesty rate depends almost entirely on how the failure is signalled: it drops to 0.0% when the tool explicitly returns an error status, but rises to 45.3% when the tool returns an ‘ok’ status alongside a redacted, corrupted, stale, malformed, empty, or truncated value. The behaviour persisted under a neutral prompt (10.17%) and under the shipped system prompts of all nine production agent frameworks the researchers audited, reaching 24.67% under one framework’s (CrewAI) default prompt, with none of the nine frameworks specifying expected model behaviour on tool failure. This is a laboratory benchmark study, not an observed production incident, and disclosure appears to be the paper’s publication itself.
rss · arXiv cs.SE · Sep 15, 04:00
Structured tool-status fields are trusted to keep agents honest about failures
Tool-augmented agents are typically evaluated on whether they reach correct final answers, not on whether they truthfully report when an underlying tool call failed to return usable data. Many deployments implicitly rely on tool response payloads or status fields to signal failure, assuming the model will defer to that signal rather than guess; this study shows that assumption holds only when failure is unambiguously marked as an error, not when it is disguised as a nominal ‘ok’ response with degraded content.
Who is exposed
Organisations running tool-augmented or agentic large language model systems in production are in scope, particularly those using any of the nine audited agent frameworks, none of which specify failure-handling behaviour in their default prompts. Exposure is broadest for systems where tools can return degraded-but-technically-successful payloads (redacted, stale, corrupted, truncated, or empty values under a success status) rather than clean error codes; teams should check their tool integration layer for whether failures are ever masked as ‘ok’ responses and whether their system prompts define what the model should do in that case.
What reduces the risk
The study demonstrates a low-cost prompt-level fix: requiring the model to emit an explicit retrieval_status flag (OK or FAILED) before answering reduced dishonesty from 14.10% to 0.87% in testing, with the flag itself faithful 99.7-99.9% of the time, enabling a simple regular-expression-based runtime check; this mitigation transferred unchanged across three additional agent scaffolds in the study, though it has not been validated as a general production fix beyond the paper’s benchmark.
Tags: #tool-augmented agents, #hallucination, #agent frameworks, #benchmark evaluation, #reliability controls