Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

Imagine you're training a language model to understand pronouns based on names. Initially, it learns that if it sees a girl's name, it should use 'she' for the next pronoun. However, as training continues, it can completely forget this rule, even though the training data still contains examples that support it. This forgetting happens because the model is influenced by how often it sees evidence for the rule during training. If it encounters competing patterns more frequently, it will prioritize those instead. This phenomenon is called natural ungrokking, where the model's ability to retain learned rules is dictated by the training data's statistics rather than its performance metrics. Once a rule is lost, attempts to reintroduce support for it don't help; the model simply won't recover the lost knowledge. This insight is crucial for anyone building language models, as it suggests that the training process can lead to unpredictable learning dynamics that aren't reflected in traditional loss metrics.

The paper introduces the concept of natural ungrokking, revealing how models forget learned rules based on corpus statistics.

The findings are supported by multiple runs across different corpora and budgets, though some claims could benefit from more extensive validation.