New Benchmark Measures Security Drift in LLM Code Editing
arXiv cs.AI Surfaced Read the original
Researchers introduce WeSCE, a benchmark of 400 executable programs derived from real-world code, designed to measure how LLM-driven code edits change security posture when tasks specify only functional goals such as feature addition, feature removal, bug fixing, and refactoring. The work proposes a continuous risk representation that aggregates heterogeneous vulnerability signals and defines drift measures covering changes in overall risk, worst-case severity, and vulnerability distribution. The abstract describes the benchmark's construction and metrics but does not report specific drift rates or failure percentages for any particular model. No affected model versions, deployment prerequisites, or disclosure timeline apply, since this is a measurement benchmark rather than a vulnerability disclosure.
rss · arXiv cs.AI · Aug 18, 04:00
「Why functional-only code editing is trusted by default」 Teams routinely delegate bug fixes, feature changes, and refactoring to LLMs using prompts that state only what the code should do, not how it should remain secure while doing it. This practice relies on the assumption that an LLM's edits preserve existing security properties even when no security requirement is stated, since maintenance tasks are treated as low-risk relative to greenfield generation. That assumption has been largely untested because prior benchmarks for LLM code generation focus on functional correctness or on security of newly generated code, rather than on whether security quietly regresses across successive edits to existing programs.
「Who This Concerns」 This concerns organisations that use LLMs for routine code maintenance, such as bug fixes, refactoring, or feature changes, without explicitly specifying security requirements in the task prompt or review process. Teams relying on LLM-assisted coding pipelines without a dedicated security review step for AI-generated diffs are the intended audience for this benchmark, since it targets exactly the weak-constraint pattern common in everyday coding requests. Exposure is broad in principle, since the scenario tested (functional-only task specification) is a default mode of use, but the benchmark itself does not measure any specific production system or model deployment.
「Mitigation」 There is no fix to apply, since this is a measurement tool rather than a vulnerability in a specific product. Organisations can use WeSCE to evaluate their own code-editing models or workflows for security drift, and can compensate by adding explicit security requirements to code-editing prompts and by maintaining independent security review for LLM-generated changes rather than relying on functional correctness alone.
Tags: #LLM code generation, #security benchmark, #vulnerability measurement, #software supply chain, #AI-assisted coding