Adversarial Prompts Recover 'Unlearned' Data From LLMs

Researchers evaluated prompt-based and fine-tuning-based unlearning methods on the TOFU benchmark using Llama-3.2-3B-Instruct, then subjected the strongest-performing methods to adversarial probing across eight attack suites using a new Attack Success Rate (ASR) metric, defined as the fraction of adversarial responses whose leakage score exceeds 0.2. Several fine-tuning-based methods achieved Forget Quality scores above 0.91 under standard clean-query evaluation, yet targeted 'forgotten' information remained recoverable with ASRs between 72.8% and 84.3%, close to the 87.5% ASR measured on the unprotected base model. Clean multilingual reformulations were an exception, yielding only 2.95% measured leakage. A manual audit found the binary ASR judgments agreed with human factual assessments in seven of ten cases, suggesting the metric is a useful but imperfect signal. The work is limited to one model and one benchmark, and has not been reported as disclosed to any vendor since it concerns research methodology rather than a specific deployed product.

rss · arXiv cs.CL · Aug 25, 04:00

「Why unlearning benchmarks were trusted」 Machine unlearning is used to remove the influence of specific training data from a model, often to support data deletion or right-to-be-forgotten obligations without full retraining. Standard evaluation has relied on clean, non-adversarial queries and aggregate metrics like Forget Quality, on the assumption that passing these benchmarks indicates the information is genuinely inaccessible.

「Who should check their assumptions」 This is most relevant to organizations that use or plan to use machine unlearning as a compliance mechanism for data deletion or right-to-be-forgotten requests, particularly where they rely on Forget Quality or similar clean-query metrics as evidence of removal. Exposure is currently narrow in evidentiary terms: the demonstration used a single 3B-parameter model (Llama-3.2-3B-Instruct) and a single benchmark (TOFU), so it is not yet established how broadly the gap generalizes across model families, scales, or unlearning techniques. Teams should check whether their unlearning validation includes adversarial or strategic-prompting tests, or whether it relies solely on standard clean-query benchmarks.

「What reduces the risk」 There is no fix for the underlying gap demonstrated here; the paper's main proposal is to add adversarial stress-testing, such as the eight attack suites and ASR metric used in the study, as a complementary evaluation step alongside standard clean-query benchmarks before treating unlearning as verified. Clean multilingual reformulation showed markedly lower measured leakage in this lab setting and may warrant further investigation as a partial compensating approach, though it was tested on the same limited model and benchmark.

Tags: #machine unlearning, #adversarial robustness, #data privacy compliance, #LLM evaluation, #benchmark validity