Building a 128-GPU Cluster from Retired Hardware for LLaMA-70B Inference

Researchers built a 128-GPU cluster (DumpsterCluster) entirely from second-hand V100 components and operated it in production for one year, serving LLaMA-70B inference. The cluster cost roughly $22K to build, versus about $600K for an 8-GPU B200 system, and pipeline-parallel optimizations allowed it to achieve competitive LLaMA-70B throughput. The catch is energy: older GPUs consume significantly more energy per token, so total cost of ownership only favors the retired-hardware approach in regions with cheap electricity. Under grid-average carbon intensity, the second-hand system produces about 4x higher total carbon emissions per token for 8B models and over 40x higher for 70B models compared to current-generation hardware.

rss · arXiv cs.LG · Aug 19, 04:00

「Background」 As AI datacenters cycle out GPUs for newer generations, large quantities of still-functional accelerators reach secondary markets at steep discounts. Whether these retired chips can be assembled into a cluster capable of serving modern large language models, and under what economic and environmental conditions that makes sense, has been an open practical question rather than a benchmarked one.

「What This Changes」 Teams evaluating low-cost inference infrastructure now have a concrete data point: a physically built and year-long-operated 128-GPU retired-hardware cluster serving LLaMA-70B, with real acquisition cost ($22K vs $600K) and throughput results, not just a theoretical TCO model. This supports a strategy of deploying repurposed GPU clusters specifically in regions with cheap and preferably clean electricity, for inference workloads where higher per-token energy draw is acceptable. It does not support deploying such clusters in regions with expensive or carbon-intensive grids, where the paper's own numbers show total carbon emissions per token can run 4x to over 40x higher than current-generation hardware.

「Caveats」 The result is specific to V100-class hardware, a 128-GPU scale, and LLaMA-70B pipeline-parallel serving; the economics reverse in regions without inexpensive electricity, and the carbon accounting depends heavily on grid carbon intensity assumptions, so the approach is not a general substitute for current-generation inference hardware.

Tags: #LLM inference, #hardware economics, #GPU clusters, #sustainability, #systems research