Illustration: U.S.-China Trade
We then create trend departure categories for simplicity. For U.S.-China trade in 2020, the size of the 95 percent confidence interval is $151 billion. If the trend departure is less than 30 percent of the confidence interval ($45 billion in this example), it’s in the no trend departure category. If the trend departure is between 30 percent and 100 percent of the confidence interval ($45 to $151 billion in this example), it’s in the moderate trend departure category (+ or -). And if the trend departure is greater than 100 percent of the confidence interval (greater than $151 billion in this example), it’s in the large trend departure category (+ or -).2
Finally, we compare the actual trend departure with the crowd’s forecasted trend departure. In this example, the crowd forecasted that the value of U.S.-China trade in 2020 would be $510 billion, or $125 billion below the projected value. After dividing that value by the size of the projection’s confidence interval ($151 billion), we get a forecasted trend departure value of -0.8. In fact, U.S.-China trade in 2020 was $560 billion, or $75 billion below the projected value, which yields an actual trend departure value of -0.5. The difference between the two—0.3 in this example—is the crowd error. Table 1 provides a summary of this example.
|Table 1. Calculating trend departure values for U.S.-China trade in 2020|
|A||Projected value||$635 billion|
|B||Size of 95 percent confidence interval for projected value||$151 billion|
|C||Crowd forecast||$510 billion|
|D||Actual value||$560 billion|
|Actual trend departure: (D-A) / B||-0.5|
|Forecasted trend departure: (C-A) / B||-0.8|
|Projection error: Absolute value of actual trend departure||0.5|
|Crowd error: Difference between forecasted and actual trend departures||0.3|
We show the crowd’s accuracy in Table 2 by comparing the mean crowd forecast to the projection. We do this in two ways:
One approach considers all metrics together and compares the mean projection error with the mean crowd error. By this measure, the crowd significantly outperformed the projection: the crowd’s mean error is 0.28 and the projection’s mean error is 0.54.
A second approach compares the crowd and projection on a question-by-question basis. In Table 2, we score each question as a win for the crowd (green cells), a win for the projection (red cells), or a tie (yellow cells). A question is a tie if the difference between the crowd and projection error is less than 0.1. By this measure, the crowd again significantly outperformed the projection: the crowd outperformed the projection on 56 percent (9/16) of the metrics; the projection outperformed the crowd on 13 percent (2/16) metrics; and the two approaches tied on 31 percent (5/16) metrics.
|Table 2. Comparing actual and forecasted trend departures for sixteen metrics|
|Metric||Actual Trend Departure||Forecasted Trend Departure||Projection Error||Crowd Error|
|Remote software engineer jobs||1.6||1.7||1.6||0.1|
|O-1 visas to Chinese nationals||-1.5||-0.7||1.5||0.8|
|Privacy-Security news articles||1.0||0.7||1.0||0.2|
|Ratio of AI publications to job postings||-0.9||-0.5||0.5||0.4|
|American unfavorable views on China||0.6||0.4||0.6||0.2|
|Facial recognition negative framing||-0.5||0.1||0.5||0.6|
|DoD AI grants||-0.3||0.0||0.3||0.3|
|New funding for facial recognition companies||-0.4||0.0||0.4||0.4|
|New funding for private tech companies||-0.3||0.2||0.3||0.5|
|DoD AI contracts||0.4||0.5||0.4||0.2|
|Jobs requiring ML skills||-0.2||-0.2||0.2||0.0|
|Big Tech Revenue||-0.1||-0.1||0.1||0.0|
|New funding for startups||0.0||0.1||0.0||0.1|
|Percentage computer vision papers||0.0||-0.2||0.0||0.2|
|Large actual trend departure||Crowd outperforms projection|
|Moderate actual trend departure||Tie|
|No actual trend departure||Projection outperforms crowd|
1 The projections are made using the AAA ETS exponential smoothing algorithm. We describe the trend departure methodology in greater detail in Page et al., Future Indices (2020).
2 The boundaries of these categories are arbitrarily selected.
3 The underlying data is here.
4 For questions where the risk of point three seemed particularly high, we also calculated the crowd forecast at an earlier date, before some or all of the data relevant to the forecast question was available. In all cases we checked, the difference was immaterial.
To stay updated on what we’re doing, follow us on Twitter @CSETForetell.