We are grateful to the entire INFER community – the forecasters, experts, teams, stakeholders, and partners – who make it possible to carry out our mission to advance the forecasting capability and analytical rigor behind U.S. policymaking. We want to share with you the strides INFER has made towards this mission in 2023:
1. We expanded our support for stakeholders with data-driven insights from ‘issue decomposition’ workshops and improved reporting elements.
One of INFER’s key achievements last year was establishing a process for ‘issue decomposition,’ our approach to working with government stakeholders to break down a strategic issue into the forecastable signals that quantifiably track the outcome of that issue. This year, we continued the evolution of our issue decomposition methodology by developing stakeholder workshops. INFER facilitated these workshops with experts from both inside and outside the U.S. Government intelligence and defense sectors to better understand and track Iran's nuclear weapon capabilities and regional influence on violent non-state actors. We also decomposed potential disruptions to semiconductor supply chains and provided reports for analysts to monitor warning signals.
To help our stakeholders monitor these issue decompositions, INFER introduced improvements to our periodic snapshot reports summarizing the crowdsourced forecasts. We began using AI Large Language Model (LLM) tools to develop high-level summaries of user rationales that would help analysts quickly scan important themes from the qualitative data (see the Iran nuclear report from October). We also explored new data visualizations, including a heatmap to more clearly depict how specific forecast signals were directionally impacting the future of a strategic issue (see the Iran nuclear report from November).
2. We formed partnerships with Google, Cosmic Bazaar, and the Phoenix Challenge to bring new perspectives to INFER and our stakeholders.
In an effort to expand and diversify the external perspective INFER provides for government stakeholders, INFER formed several new partnerships. We worked with strategists in the AI space to collect forecasts as part of the Mission: AI Advancement challenge that examined the current AI-boom, the future of generative AI, and its impact on society. The challenge concluded with a forecasting meetup for select INFER forecasters and forecasters from Google to exchange perspectives and forecasting best practices.
We partnered with Cosmic Bazaar, the UK Government’s forecasting program, and cross-posted a series of questions to forecast global trends in biofuels, AI, and the future of the internet. From this effort, we gained comparative insight from the two communities on topics of interest to both governments.
Additionally, we worked closely with a presenter at the Phoenix Challenge, a Department of Defense-funded seminar for senior officials from governments, think tanks, and other policy experts. Together, we posed several forecasting questions about the impact of AI on mass media and misinformation to the INFER community and to Phoenix Challenge attendees. This revealed striking differences in how each group forecasted about the immediate risks of AI and fueled further discussions at the Phoenix Challenge event.
3. We increased the caliber of forecasting activity and collaboration across our global forecasting community.
INFER continued to invest in a cohort of paid Pro Forecasters and a group of Team Leads that have helped elevate the quality and engagement of the forecasting community. This season, we selected 85 Pro Forecasters from 26 countries who brought multifaceted expertise from government, academia, and industry. Their combined knowledge shaped diverse perspectives and facilitated collaboration and growth. Together, our Pro team improved their relative brier score by 0.447 net brier points, demonstrating a bump in accuracy over the course of the season.
Our Team Leads recruited and organized teams of forecasters, who comprised 30% of INFER's active forecasting population. Their teams competed in a six-month Future Bowl tournament, generating hundreds of forecasts on critical issues, such as nuclear tests conducted by North Korea, economic sanctions on Russia, and global AI innovation and competition.
Collectively, INFER’s community of forecasters represented 85 countries across the globe and forecasted on 115 questions about critical world events. The community contributed 30% more forecasts this season compared to last season and enjoyed collaborating and exchanging perspectives – 54% of forecasters were on a team. Forecasters shared more than 9,000 upvotes on commentary that sparked conversation and new insights, and made hundreds of forecasts on the new mobile app.
Individually, we saw forecasters engage more with the community and forecast topics. On average, each forecaster made 36% more forecasts compared to last year. More than 60% of forecasters provided at least one update to their original forecasts, a critical metric we follow to ensure our current aggregate forecasts are timely. This season also brought a number of first-time forecasters to INFER, who comprised about 17% of our community.
4. We enhanced our forecasting platform and user experience through new capabilities like our mobile app, forecast survey integration, and use of AI tools.
To better support our stakeholders and improve the forecaster experience, we introduced several new platform features:
- We launched the INFER mobile app with push notifications, upvoting, and alerts to enable on-the-go forecasting.
- We released Request for Response capability, which allows conference or meeting attendees to submit forecasts via survey without logging in to the forecasting platform. Their forecasts are automatically integrated with INFER’s crowd.
- We introduced AI-generated rationale summaries to provide quick overviews of forecaster arguments on all platform questions.
- We created crowd forecast change alerts to enable users to request notifications about significant shifts in the crowd forecast on questions they have forecasted.
- We expanded team capabilities to allow forecasters to form and invite teams, and for invited users to accept or reject invitations without the help of an Admin.
- We produced several other user features to improve our site usability, including the use of emojis in comments, auto-saving a last draft of forecast rationales, searching for questions that a user has previously forecasted on, and targeted site announcements.
As we embark on a third season of forecasting in 2024, INFER aims to further grow our community, nurture forecaster skills, and build more strategic relationships with specific entities within the U.S. Government. We look forward to delivering unique insights that support and inform intelligence analysis and policymaking well into the future. We hope you’ll continue joining INFER in fulfilling this mission and forecasting with us into next season.