Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
Andres Nava, Matthieu Wyart
Key claim
Hierarchical concept geometry emerges from spectral structure of word statistics.
This paper presents a theory that explains how the relationship between general and specific concepts is geometrically represented in language models. The key finding is that the structure of word embeddings reflects a hierarchical organization that mirrors taxonomic relationships, which can be observed in both word2vec and Gemma 2B embeddings.
The paper introduces a new theoretical framework for understanding hypernymy in language representations.
The methodology is solid and supported by empirical validation across multiple datasets.
Deep reliability assessment
The methodology supports the claim that hierarchical concept geometry in language models emerges from co-occurrence statistics rather than being a result of hierarchy-specific mechanisms. However, the generalizability of these findings to all language models and contexts may be overstated.
Reproducibility
Yes, the authors mentioned that code for reproducing the experiments and figures will be released in a public repository.
Discussion questions
- What if the assumption that words closer on the WordNet graph co-occur more frequently is not universally applicable?
- How can builders leverage the findings on hierarchical geometry in practical applications like semantic search or recommendation systems?
- If future experiments show that hierarchical geometry does not hold in certain contexts or with different datasets, how would that impact the conclusions drawn in this paper?
Key figure
Figure 1 illustrates the WordNet hierarchy for a taxonomy of organisms, showing how mean co-occurrence statistics relate to hierarchy distance.