Every three months, participants in the Metaculus Forecasting Cup compete for a prize pool of
;
around $5,000 by attempting to predict the future. The platform poses high-stakes geopolitical questions such as, “Will Thailand experience a military coup before September 2025?” or “Will Israel strike the Iranian military again before September 2025?”
Unlike simple yes-or-no guesses, forecasters assign probabilities to potential outcomes, often weeks or months in advance—and with striking accuracy. Metaculus users correctly anticipated the date of Russia’s invasion of Ukraine two weeks early and assigned a 90% chance to Roe v. Wade being overturned nearly two months before it happened.
Still, one result from the recently concluded Summer Cup caught everyone off guard: an artificial intelligence system placed in the top 10. “It’s actually kind of mind-blowing,” said Toby Shevlane, CEO of Mantic, the UK-based startup behind the AI. When the competition began in June, participants estimated the leading AI would perform at only 40% of the level of top human forecasters. Instead, Mantic scored above 80%.
“Forecasting—it’s everywhere,” notes Nathan Manzotti, who has worked in AI and analytics for several U.S. government agencies, including the Department of Defense. Practically every federal agency engages in some form of forecasting.
The value is clear, explains Anthony Vassalo, co-director of the Forecasting Initiative at RAND, a U.S. government think tank. Forecasting not only helps leaders anticipate events but also influences their decisions. By simulating how policies may alter outcomes, forecasters help decision-makers avoid surprises and change undesirable scenarios before they unfold
Yet geopolitical forecasting is notoriously difficult. Top human predictions can require days of work and tens of thousands of dollars for just one question. For organizations like RAND, which must monitor multiple issues across the globe, building and regularly updating forecasts through humans alone would take months.
Traditionally, machine learning has excelled in areas rich with structured data, such as weather modeling and quantitative finance. Geopolitical forecasting, however, is messier. “You have a lot of complex, interdependent factors where human judgment can often be both more affordable and accessible,” says Deger Turan, CEO of Metaculus.123
But large language models are changing that dynamic. Like human forecasters, they sift through unstructured information, simulate judgment, and improve with practice—by making predictions, testing them against reality, and refining their methods at scale.
“Our main insight was that predicting the future is inherently verifiable—that’s how humans learn,” says Ben Turtel, CEO of LightningRod, which builds AI forecasters that have also competed successfully on Metaculus. His company recently trained a model on 100,000 past forecasting questions.
The improvements are showing up on the scoreboard. In June, the top AI built by Metaculus on OpenAI’s o1 reasoning model ranked 25th in the competition. This time, Mantic’s system broke into the top 10, finishing eighth out of 549 contestants—the first AI to do so in the cup’s history.
Even so, some caution is warranted. Ben Wilson, an engineer at Metaculus, points out that the tournament featured only 60 questions and that many of the 600 participants were amateurs who made few predictions, lowering their scores..ad-overlay{position:fixed;inset:0;display:flex;align-items:center;justify-content:center;background:rgba(2,6,23,0.6);z-index:9999;opacity:0;pointer-events:none;transition:opacity .25s}
.ad-overlay.show{opacity:1;pointer-events:auto}
.ad-card{background:#fff;border-radius:12px;max-width:820px;width:92%;box-shadow:0 12px 40px rgba(2,6,23,0.5);overflow:hidden;position:relative}
.ad-media{display:flex;gap:0}
.ad-media img{width:100%;height:auto;display:block}
.ad-body{padding:16px;text-align:center}
.ad-body h2{margin:0 0 8px;font-size:18px}
.ad-body p{margin:0 0 12px;color:#334155}
.ad-actions{display:flex;gap:10px;justify-content:center;padding-bottom:16px}
.btn{padding:10px 16px;border-radius:8px;text-decoration:none;font-weight:700;border:none;cursor:pointer}
.btn-primary{background:#06b6d4;color:#012;}
.btn-ghost{background:transparent;border:1px solid #cbd5e1;color:#0f172a}
.ad-close{position:absolute;right:10px;top:10px;background:transparent;border:none;font-size:20px;cursor:pointer}
.dont-show{display:flex;gap:8px;align-items:center;justify-content:center;padding-bottom:12px;color:#64748b}
@media(min-width:700px){.ad-media{flex-direction:row}.ad-media img{max-width:420px}}
On top of that, AIs enjoy a structural advantaghttps://otieu.com/4/9900344
“coverage”—how early forecasts are made, how many questions are tackled, and how often predictions are updated. An AI, constantly scanning and adjusting to new information, can outperform humans on coverage even if its accuracy is slightly lower
Comments
Post a Comment