Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking
Large Language Models (LLMs) demonstrate partial forecasting competence across social, political, and economic events. Yet, their predictive ability varies sharply with domain structure and prompt framing. We investigate how forecasting performance varies with different model families on real-world questions about events that happened beyond the model cutoff date. We analyze how context, question type, and external knowledge affect accuracy and calibration, and how adding factual news context modifies belief formation and failure modes. Our results show that forecasting ability is highly variable as it depends on what, and how, we ask.
69270769243b2216fb75cc4d
Chinmay Karkar
69270769243b2216fb75cc4e
Paras Chopra
Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking
Large Language Models (LLMs) demonstrate partial forecasting competence across social, political, and economic events. Yet, their predictive ability varies sharply with domain structure and prompt framing. We investigate how forecasting performance varies with different model families on real-world questions about events that happened beyond the model cutoff date. We analyze how context, question type, and external knowledge affect accuracy and calibration, and how adding factual news context modifies belief formation and failure modes. Our results show that forecasting ability is highly variable as it depends on what, and how, we ask.
Forecasting performance of Large Language Models varies significantly across different domains and question types, influenced by context and external knowledge.
Large Language Models forecasting competence domain structure prompt framing model families real-world questions model cutoff date context question type external knowledge accuracy calibration belief formation failure modes