Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson
Read on arXiv →
Key claim
Requential coding significantly improves model compression and generalization.
In plain English
Compression is crucial for effective intelligence, but existing methods often fail to capture the simplicity of learned functions. Current parameter-based compression techniques do not account for the actual information stored in models, leading to inefficient code lengths. Requential coding addresses this by allowing a teacher model to select training samples based on the student's distribution, resulting in much shorter codes that reflect the model's learning. This approach not only enhances compression but also provides insights into model behavior and data structure, which can be valuable for builders looking to optimize their models.
Introduces a new coding method that significantly reduces code length and improves generalization guarantees.
Presents theoretical results and comparisons to existing methods, though empirical validation may be limited.
Deep reliability assessment
The methodology supports the claim that requential coding can achieve shorter code lengths independent of parameter count and data entropy, but the generalization guarantees may be overclaimed without broader empirical validation across diverse datasets.
Reproducibility
Yes, the paper provides a GitHub repository for the code: https://github.com/shikaiqiu/requential-coding.
Key figure
Figure 1 illustrates the process of requential coding, showing how a student model samples training data and a teacher model selects samples, with the code length being determined by the KL divergence between the teacher and student models.