From 228 items, 3 important content pieces were selected
Reliability & Assurance
Practice
- New Benchmark Finds Agents Save ERP Records Correctly in as Few as 3% of Runs ⭐️ 7.0/10
- Pareto Atlas Maps Which LLM Inference Optimizations Actually Win ⭐️ 8.0/10
Reliability & Assurance
Paper Proposes Label-Efficient Statistical Test for Model Update Regressions ⭐️ 7.0/10
A newly posted arXiv paper introduces DISCERN, a sequential two-tier statistical protocol for certifying whether a model update (retraining, fine-tuning, quantization, or vendor swap) is at least as good as the model it replaces, using mostly unlabeled traffic. The method exploits the fact that risk differences between two models only show up where they disagree, so a zero-label tier can certify benign updates directly from disagreement rates, while an audited tier spends labeling budget only on sampled disagreement cases, backed by an anytime-valid statistical guarantee. The authors report results across more than 14,000 replayed audit streams over 785 update pairs, including LoRA (low-rank adaptation) fine-tunes of language models up to 1.4 billion parameters, with a measured miscoverage rate of 0.0002 against a nominal 5%, a detection power of 0.986, zero false alarms, and 56% of benign updates certified with zero labels. These are results from a single arXiv preprint, evaluated in the authors’ own replay experiments; there is no independent replication or production deployment history disclosed.
rss · arXiv cs.LG · Sep 17, 04:00
Is informal spot-checking enough to trust that a model update didn’t quietly get worse? Production teams routinely update models through retraining, fine-tuning, quantization, or swapping vendors, and typically rely on ad hoc regression testing, sampled manual review, or aggregate offline metrics to decide whether to promote an update. These informal practices are trusted mainly because full relabeling of production traffic is expensive, but they offer no formal statistical guarantee about the rate of undetected regressions or how much labeling effort is actually needed to catch them.
Who is exposed This is relevant to organizations that promote model updates into production based on informal or partial validation rather than a statistically certified process, including teams doing frequent fine-tuning, quantization, or vendor model swaps. Exposure is best assessed by checking whether current update-validation practice can quantify its false-alarm and miss rates and how much labeled data it requires; teams already using rigorous paired testing with formal guarantees are less affected. The method itself was only tested on replayed audit streams and language-model fine-tunes up to 1.4 billion parameters, so applicability to other model types or scales is not yet demonstrated.
What reduces the risk No deployed fix is implied since this is a proposed method rather than a patch to an existing system; organizations concerned about undetected update regressions could evaluate adopting a paired disagreement-based auditing protocol like DISCERN, or at minimum benchmark existing informal validation against a formal statistical standard, once the method is independently reviewed or reproduced.
Tags: #model updates, #statistical auditing, #regression testing, #post-market monitoring, #LLM fine-tuning
Practice
New Benchmark Finds Agents Save ERP Records Correctly in as Few as 3% of Runs ⭐️ 7.0/10
Researchers introduce ERPBench, a benchmark that evaluates screenshot-only computer-use agents on a live, reproducible Enterprise Resource Planning (ERP) system by checking task outcomes against ground-truth values in the underlying database, rather than relying on screen-based success signals. The benchmark also ships a production-grade harness that gates agent actions behind human approval for safe deployment, though ERPBench itself runs agents autonomously for evaluation. Across six closed- and open-source agents, the study finds that strong performance on general graphical user interface (GUI) benchmarks does not transfer to enterprise reliability: some agents successfully reach the correct form and save a record in up to 85% of runs, but write the actually correct value in as few as 3% of those runs. The paper also catalogs failure modes specific to enterprise workflows, such as dense interfaces and coordinated multi-step interactions where an ERP system needs several linked actions to record one business event.
rss · arXiv cs.AI · Sep 17, 04:00
Existing enterprise agent benchmarks rely on proxies rather than real systems Computer-use agents that act via screenshots and simulated clicks are increasingly evaluated on general desktop and web tasks, but ERP systems that run finance, procurement, inventory, and customer operations have largely been tested only on proprietary platforms or simplified approximations. Because ERP errors alter persistent business records rather than producing visible on-screen failures, a benchmark that only checks whether an agent appears to finish a task can miss silent data corruption entirely.
What a team would do differently Teams building or evaluating agents for ERP-style automation (finance, procurement, inventory, customer records) should not treat ‘form saved’ or ‘task marked complete’ signals as evidence of correctness; ERPBench’s methodology shows these can diverge sharply from what actually lands in the database. Anyone considering autonomous agent deployment against systems of record should adopt state-grounded evaluation, checking outcomes against ground-truth database values, and should default to human-approval gating for write actions until agents demonstrate high database-verified correctness, not just high save rates. This is most relevant for organizations piloting agent-driven back-office automation where undetected wrong values carry regulatory or financial risk.
Caveats The results come from one benchmark built on a specific live ERP system with six evaluated agents; correctness rates (as low as 3%) and save rates (up to 85%) may not generalize across all ERP platforms, task types, or future agent versions. This is a newly introduced benchmark rather than an evaluation of an already-deployed production system, so it establishes a measurement method and a current snapshot rather than a definitive ceiling on agent capability.
Tags: #computer-use agents, #enterprise software, #benchmark evaluation, #agent reliability, #ERP systems
Pareto Atlas Maps Which LLM Inference Optimizations Actually Win ⭐️ 8.0/10
Researchers built a calibrated Pareto atlas covering cost, quality, and latency for large language model (LLM) inference, measuring 54 configurations of Qwen2.5-7B-Instruct on vLLM 0.12 across L4, A100, and H100 GPUs, then used those anchor points to calibrate a simulator with cross-campaign drift below 1.5 percent. On a calibrated 36-configuration grid, 18 reached the Pareto frontier, and combined optimization methods reached it more often than single methods (9 of 15 combinations versus 9 of 21 single methods). A separate quality evaluation on 200 GSM8K math questions found FP8 (8-bit floating point) weights retained 99.4 percent of baseline accuracy at 0.61-0.65x baseline latency across all three GPUs and appeared in three of four regime winners, while AWQ 4-bit quantization cut per-token latency to 0.34x baseline on L4 but lost 5.9 percent accuracy, narrowly missing a 95 percent quality floor. Most strikingly, a naive FP8 key-value (KV) cache maintained normal throughput but answered zero of 200 questions correctly, and n-gram speculative decoding measured 0.90-0.98x baseline with no real benefit on this stack. H100 won for tight latency constraints while A100 won for throughput and cost, reaching $0.106 per million tokens.
rss · arXiv cs.AI · Sep 17, 04:00
Do published inference speedups transfer across models and hardware? LLM inference optimization papers typically report speedups on different models, GPUs, prompts, and quality metrics, which makes it hard to know which techniques compose well or which ones dominate under a given deployment constraint like a latency budget or a cost target. Because exhaustively testing every combination of quantization method, batch size, and GPU is impractical, this work builds anchor measurements and calibrates a simulator to extrapolate across the full configuration space.
What a team would do differently Teams running Qwen2.5-7B-class models on vLLM can use FP8 weight quantization as a near-default choice, since it kept 99.4 percent of baseline accuracy while cutting latency to roughly 0.61-0.65x across L4, A100, and H100, and it appeared in most regime-winning configurations. Teams should treat FP8 KV cache as unsafe without dedicated accuracy testing, since it preserved normal throughput numbers while producing zero correct answers on GSM8K, meaning throughput dashboards alone would not have caught the failure. GPU selection can now be constraint-driven rather than default-to-largest: pick H100 when latency is the binding constraint and A100 when the target is throughput per dollar (measured here at $0.106 per million tokens). Teams considering n-gram speculative decoding on a similar stack should not expect a latency win, since it measured 0.90-0.98x baseline with no real benefit in this setup.
Caveats All measurements are scoped to one model (Qwen2.5-7B-Instruct), one serving stack (vLLM 0.12), and three specific GPUs (L4, A100, H100), so absolute numbers and winner rankings may not transfer to other model sizes, architectures, or serving frameworks. Sparse attention was evaluated only in simulation, not measured directly, and the quality evaluation used a single benchmark (200 GSM8K questions, five-shot), so accuracy conclusions are specific to that task type and may not generalize to other domains like open-ended generation or code.
Tags: #LLM inference optimization, #quantization, #benchmarking, #GPU cost-performance, #vLLM