Forecasting With LLMs: Improved Generalization Through Feature Steering
Humzah Merchant, Bradford Levy
Read on arXiv →
Key claim
Enhancing time-awareness reduces look-ahead bias in LLMs.
In plain English
Imagine you're trying to predict future events based on past data, like forecasting sales or weather. The challenge is that many models can get stuck in their own biases, often looking too far ahead and missing the patterns that actually matter from the past. This is called look-ahead bias, and it can lead to poor predictions when the future doesn't match the model's expectations. Current methods often struggle to balance between using historical data effectively and making accurate future predictions, which is a common failure mode in forecasting tasks.
What this paper does is explore how large language models (LLMs) can be applied to forecasting by examining their internal workings. The authors found that by focusing on specific features that represent time-awareness, they could reduce the look-ahead bias significantly. They used sparse autoencoders to analyze how these models process information over time, revealing that some features are more beneficial for making grounded predictions than others. By amplifying the right features, they were able to steer the models toward better reasoning based on historical data.
In practical terms, this means that if you're building a forecasting system, you can leverage these insights to enhance your model's performance. By understanding and manipulating the temporal features within LLMs, you can create a more reliable forecasting tool that doesn't just guess based on future trends but is firmly rooted in historical context.
The paper introduces a novel approach to using LLMs for forecasting by focusing on interpretable temporal features.
The findings are supported by analyses of internal states and interventions, indicating solid experimental validation.
Deep reliability assessment
The methodology supports a causal claim for the specific intervention tested: amplifying SAE-identified time-awareness features can reduce look-ahead-biased language in selected forecasting prompts while preserving performance on MMLU-style reasoning checks. It is more tentative on whether this improves true forecasting generalization, since the evaluation mostly measures reduced future leakage rather than better out-of-sample predictive accuracy.
Reproducibility
No code repository or dataset release is mentioned in the provided paper text. The paper describes using prediction-market data plus M&A and pharmaceutical forecasting settings, but the excerpts do not indicate whether these datasets, prompts, SAE feature IDs, or steering code are publicly available.
Key figure
Figure 1 shows that amplifying SAE-derived time-awareness features reduces future-looking mentions per 1K tokens on M&A and pharma forecasting prompts while keeping utility high on forecasting and MMLU-style reasoning tasks.