Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoning
Poojitha Thota, Shirin Nilizadeh
Read on arXiv →
Key claim
Effective detection and recovery from fine-tuning data poisoning.
In plain English
Imagine you're building a large language model that summarizes text for users. You want it to provide accurate and unbiased summaries, but what if someone maliciously alters the training data during fine-tuning? This could lead to the model producing biased or harmful summaries, even if it seems to perform well on standard tests. This is a real risk, especially when fine-tuning on small, task-specific datasets where a few bad examples can have a big impact. This issue is known as training-time data poisoning, and it can be hard to detect because the model might still meet evaluation metrics while behaving poorly in practice.
To address this, the authors propose a new approach that helps identify and fix these poisoning issues after the model has been trained. They found that poisoned data can leave behind noticeable patterns that can be detected, even when you don't have access to the original training data. Their method uses influence-function analysis and checks for semantic consistency to spot these problems. They also introduce new types of attacks that can distort facts and introduce biases without raising alarms in traditional evaluation methods.
In their experiments, they show that their detection methods can achieve 85-92% precision across various models and datasets. Additionally, they found that they could restore much of the model's original performance after identifying and addressing the poisoning, with minimal loss in utility. This means that even if a model is compromised during fine-tuning, there are effective ways to recover it without needing to retrain from scratch, which is a significant improvement over previous methods.
The paper introduces a unified defense framework against data poisoning in fine-tuning, which is a significant extension of existing methods.
The claims are supported by experiments across multiple architectures and datasets, demonstrating solid detection precision and recovery methods.
Deep reliability assessment
The methodology supports the claim that, in controlled open-source fine-tuning settings for summarization, the evaluated poisoning attacks leave detectable influence or sensitivity artifacts that can be partially remediated without full retraining. The broader claim that this is a practical unified supply-chain defense is more stretched, especially for closed API models, unseen adaptive attacks, and real-world poisoned checkpoints where ground truth and attack objectives are unknown.
Reproducibility
No code repository is mentioned in the provided paper text. The experiments use public datasets and open-source models, but the paper also states that they do not release tools that automate poisoning at scale.
Key figure
The key framework diagram depicts a two-branch defense pipeline: white-box poisoned dataset detection using influence analysis plus semantic filtering followed by gradient-ascent unlearning, and black-box poisoned model detection using adversarial perturbation sensitivity auditing.