Spec-First Prompting Improves LLM Test Generation on Production Bugs

Researchers from Google propose Spec-Driven Test Generation, a prompting technique where an LLM coding agent first documents pre-conditions, post-conditions, and undefined behaviors for a piece of code before generating tests, using this intermediate specification as a scaffold for subsequent test writing. Evaluated on production bugs from Google, the spec-driven agent improved bug detection rate by 9.8 percentage points (p = 0.0352) and branch coverage by 2.5 percentage points (p = 0.0034) compared to a traditional test generation agent baseline. Using LLM-as-a-Judge evaluation, the spec-driven agent's test suites were judged superior to the baseline in 77.8% of cases and superior to human-authored tests in 56.7% of cases, with reported gains in best-practice adherence, readability, and edge-case coverage.

rss · arXiv cs.SE · Aug 19, 04:00

「Background」 LLM agents prompted directly to write tests often miss edge cases and behavioral boundaries because they do not explicitly reason about a function's contract before writing test code. This work treats specification writing as a separate, prior reasoning step, similar to chain-of-thought scaffolding, rather than folding it implicitly into test generation.

「What this changes」 Teams building test-generation agents or coding-agent pipelines can add an explicit specification-writing step, prompting the agent to state pre-conditions, post-conditions, and undefined behaviors, before asking it to generate tests, rather than generating tests directly from code. This is a low-cost prompt or pipeline restructuring applicable to existing agent-based test generation tools, and the reported gains apply specifically to bug detection rate and branch coverage on real bugs, not just synthetic benchmarks.

「Caveats」 The evaluation is based on a single company's production bug corpus (Google) and one paper's results, so generalization to other codebases, languages, or bug distributions is untested. The paper does not report the added latency or token cost of the specification step, nor failure modes where the intermediate specification is wrong or misleading.

Tags: #LLM agents, #test generation, #spec-driven prompting, #software testing, #empirical evaluation