Study Compares Copilot, Cursor, Windsurf on Full-Stack App Generation

Researchers ran a comparative empirical study of three agentic IDEs, GitHub Copilot, Cursor, and Windsurf, tasking each with generating five full-stack web applications from scratch. The agents performed reliably on established patterns such as CRUD operations and authentication, but produced significantly more errors when asked to implement less common distributed architectures such as a task queue. The authors conclude that agentic IDEs cannot replace developers outright but shift the developer's role toward orchestrating LLM-based agents through natural-language instructions and iterative refinement. Differences between the three tools existed but were narrow, with each showing its own peculiarities rather than one clearly outperforming the others.

rss · arXiv cs.SE · Aug 24, 04:00

「Context」 Agentic IDEs embed LLM-based agents that can plan, write, and iterate on code with reduced human intervention, and are increasingly marketed as capable of building entire applications rather than just autocompleting snippets. Most existing evaluations of these tools focus on isolated coding tasks or benchmarks rather than end-to-end generation of complete, deployable applications, which is the gap this study targets.

「Practical implications」 Teams evaluating agentic IDEs for scaffolding new services get a concrete signal about where to trust generated code with light review versus where to budget for heavier scrutiny: common patterns like CRUD and auth appear largely reliable, while less standard distributed architectures such as task queues warrant closer inspection or manual implementation. This supports treating agentic IDEs as accelerators for boilerplate-heavy work while keeping engineers responsible for architectural decisions and orchestration on non-standard designs, rather than expecting full autonomous delivery of a production system.

「Limits」 The study covers only five generated applications across three tools, so the sample size is small relative to the range of real-world architectures and coding conventions teams use. The abstract does not specify the scoring methodology, error definitions, or reproducibility details, which limits how confidently the specific error rates can be generalized beyond the tested scenarios.

Tags: #agentic IDEs, #LLM code generation, #empirical software engineering, #developer tools, #AI coding assistants