License-Aware Distillation Recipe for CPU-Deployable Safety Classifiers

The paper presents a reproducible, license-aware knowledge-distillation recipe for building small safety classifiers from a strong open guard model. A teacher model labels roughly 97,000 prompts drawn from 24 public datasets into seven safety categories aligned to a public hazard taxonomy, and a fleet of small student models spanning lexical, shallow, encoder, and generative architectures is trained to reproduce that signal. The training corpus is partitioned at the license boundary so a deployable model and a research model differ only in training data, making the accuracy cost of licensing restrictions measurable. Every model is evaluated against an independent gold benchmark of 6,361 rows across four slices, including a harmless-prompt slice used to measure over-defense. The distilled students match the teacher on adversarial text within overlapping confidence intervals and reduce false alarms on harmless prompts: the smallest generative student reaches 3.8% false alarms versus 4.8% for the 8-billion-parameter teacher, and the encoder student classifies in roughly 24 ms per request on CPU. Per-class rebalancing is reported as the single decisive ingredient of the recipe; the authors do not claim superiority over the distilled guards on the clean reference slice, where the teacher-derived guards remain ahead.

rss · arXiv cs.AI · Aug 25, 04:00

「Why this matters」 Open guard models used to filter unsafe LLM outputs typically have 1 to 9 billion parameters, target GPU inference, and take seconds per request on CPU, making them costly to deploy at scale or on commodity hardware. Knowledge distillation trains a smaller student model to mimic a larger teacher's outputs, and license-aware partitioning here means separating training data by usage rights so that a model built only from freely redistributable data can be compared directly against one built from all available data.

「What a team could do differently」 Teams building content moderation or safety-filtering layers for LLM applications can use this recipe to train a CPU-only classifier that runs in tens of milliseconds per request, avoiding GPU dependency for the safety layer specifically. The license-boundary partitioning gives a concrete, measured estimate of the accuracy tradeoff from restricting training data to permissively licensed sources, which is useful for teams that need a legally clean deployable model but want to know what they give up. The reported reduction in false alarms on harmless prompts is relevant to teams whose current guard models over-block benign traffic, since it suggests smaller distilled students can be tuned to be less trigger-happy than their larger teacher while matching adversarial detection.

「Limits」 The abstract does not report full benchmark accuracy across all four evaluation slices or compare against other existing guard models beyond the one teacher used, so the general competitiveness of this recipe versus alternative small guard models is unconfirmed. Results are specific to the seven-category hazard taxonomy, the 24 source datasets, and the particular teacher model used to generate labels; performance may not transfer to other taxonomies or domains without repeating the distillation process.

Tags: #LLM safety, #knowledge distillation, #CPU inference, #model licensing, #content moderation