Frequently Asked Questions
Updated April 2022
What is INFER?
INFER, short for INtegrated Forecasting and Estimates of Risk, is a forecasting program designed to generate valuable signals and early warning about the future of critical science and technology trends and events for U.S. Government policymakers. INFER empowers scientists, researchers, analysts, and hobbyists from inside and outside the U.S. Government to have a direct impact on policy and decision-making. The public portion of INFER is one of multiple forecasting sites to be operated as part of this program
INFER is run by the non-profit Applied Research Laboratory for Intelligence and Security (ARLIS) at the University of Maryland and Cultivate Labs. Funding for this program has been provided by a grant from Open Philanthropy.
The domain of this site is INFER-pub.com. What is INFER-pub?
The INFER program uses different forecasting platforms for participants inside and outside the U.S. Government. INFER-pub is what we lovingly call INFER-public, the forecasting platform open to the public. INFER-pub.com is one of multiple forecasting platforms targeting different populations inside and outside the U.S. Government.
What is ARLIS?
The Applied Research Laboratory for Intelligence and Security (ARLIS) at the University of Maryland. ARLIS is one of 14 U.S. Department of Defense University Affiliated Research Centers (UARCs). ARLIS builds robust analysis in the "human domain,” focusing on three research areas: Human and Sociotechnical Systems; AI, Automation, and Augmentation; and Advanced Computing and Emerging Technologies.
Who do I contact if I need help?
For general questions about the project, click the "Contact Support" button below or send an email to [email protected].
What am I being asked to do as a participant?
We are asking participants to estimate the probability that various events related to science and technology trends will occur and to offer a rationale for their forecasts. INFER is designed to generate real-time crowd forecasts, so we ask participants to log in regularly (several times per month) to make and update their forecasts. We will also release new sets of questions periodically and encourage participants to check in when alerted of new questions. We also encourage you to support the project in other ways, such as suggesting forecasting questions or topic areas to the INFER team, inviting friends and colleagues, and posting about INFER on social media.
Can I invite my friends or colleagues to become forecasters?
Yes, of course! We encourage anyone with an interest in forecasting or science and technology to apply to join the INFER project community. No specific background in any of the topic areas is necessary.
Making and Updating Forecasts
How does forecasting work?
To make a forecast on INFER, simply enter the probability (i.e., 0-100%) you think an event will occur, along with a rationale for your forecast. Here is a helpful article with tips on how to approach your first forecast.
How do you generate the crowd forecast?
The crowd forecast is an aggregation of all individual forecasts at any given time. Initially, the crowd forecast will simply be the mean of all individual forecasts. Over time we will transition to more sophisticated aggregation methods, such as putting more weight on the forecasts of individuals with the best track record.
Who determines what the correct answer is?
The background section for each question provides information about how we judge the correct answer. In many cases, we simply cite an authoritative data source, such as a well-known database. In other cases, we provide details about how we would evaluate borderline cases. But ultimately, the INFER Team has sole discretion to decide what the correct answer is.
How often should I submit forecasts and how much time should I spend each time I submit a new forecast?
Each participant should submit and update forecasts several times per month, though we encourage more active engagement. Participants should expect to spend several minutes every week on the site reading through new questions and updating forecasts. Updating is highly encouraged: the most accurate forecasters tend to update their predictions more often than the average forecaster.
What should I include in my forecasting rationales?
Rationales are short explanations for your forecasts. Share considerations that significantly affected your forecast, as well as links to information about those considerations that other forecasters might find useful. The more forecasters share their rationales, the more useful the data is to policymakers.
Can I withdraw my forecast from a question?
No, there is no way to delete a forecast or withdraw from a question. However, you can submit another forecast to reflect how you've changed your mind. If you forecast more than once in one day, only your last forecast of the day will count toward your accuracy score.
I have a specific question regarding a forecast question, how can I get more specifics before I make a forecast?
If you need any clarifications, first read the Information box underneath the question. If you still need clarification, you can request a clarification from within the question page, or send an email to [email protected].
What is INFER’s position if I encounter a question where I believe I need access to paid data to make an informed forecast?
All forecast questions on INFER can be answered using open-source information to come up with your best-estimate probability forecast. Where possible, INFER will provide historical or other relevant data within a question’s background section. If additional data is not readily available to you (e.g. it is behind a paywall), we ask that you make your best judgment using what is available. All questions released on INFER have been determined by our team to be important for generating signals for our research priorities. However, you may choose not to forecast in a question in which you feel you cannot make an informed judgment.
What does it mean to resolve a forecast question?
When we know the correct answer to a question (e.g., when an event occurs), we can "resolve" the question. An INFER site Administrator will resolve a question, which then allows the site to measure and provide accuracy scores.
How are my forecasts scored for accuracy?
The INFER site uses a measure for accuracy that is also used by weather forecasters, called the Brier Score (the lower your score, the better). This article explains how we calculate the Brier Score.
To measure how accurate you are compared to the crowd, INFER uses an Accuracy Score (or Net Brier Points). This score is based on the Brier Score, and is generally what shows up on the site leaderboards. This article explains how we calculate Net Brier Points.
How can I know my accuracy score?
You can find your cumulative Accuracy Score (or Net Brier Points) listed in My Profile or on any of the site’s leaderboards. Please note that INFER forecasters will not have Accuracy Scores until a forecast question has been resolved.
INFER's Question Topics
Can I suggest a question or topic area for the site?
Yes! Within the All Questions view, click the tab labeled "The Question Lab" to suggest a forecasting question and follow the instructions. From there, you will also be able to see questions submitted by others and leave comments and upvotes. The INFER Team makes selections on which questions to publish based on what is relevant to the overall research project.
Privacy & Etiquette
What are INFER’s policies for participant privacy?
INFER is committed to protecting participant privacy consistent with the goals of the site as outlined in the Consent Form.
Do you have a content moderation or etiquette policy?
Our Etiquette Policy can be summarized as “Be kind, respect others, and stay on topic.” Inflammatory comments, personal attacks, vulgar or offensive labels, marketing efforts (“spam”), repeated, overt political advocacy, and aggressive behavior or harassment of any kind are all prohibited. If you believe that a participant’s comment violates this policy, you may flag it by clicking “Flag” below their comment or email us at [email protected]. The INFER team resolves violations of the Etiquette Policy on a case-by-case basis, and we reserve the right to disqualify offenders from rewards or other opportunities, and deactivate the accounts of offenders. This policy applies to other forums that are INFER-related (e.g., email, meetups, webinars, or INFER Slack channel).