"I have known no wise people who didn't read all the time — none, zero." – Charlie Munger


Print Friendly, PDF & Email

The U.S. intelligence community (IC) is estimated at over 100,000 people and has a budget of over $50bn.  Despite the vast resources, the IC has had several recent high profile failures, including the prediction that Saddam Hussein was hiding weapons of mass destruction in Iraq.  After the Iraq mistake, the leaders in the IC took unconventional action and engaged IARPA (Intelligence Advanced Research Projects Activity) to conduct research on forecasting.  The study ran from 2011-2015 and the Good Judgment Project (formed by Tetlock) was one of the five teams to compete.  The Good Judgement Project significantly outperformed the other teams in forecasting results (after two years the other teams from Michigan and MIT were dropped) and the books describes the methods used by the Superforecasters of the Good Judgement Project and other lessons learned.

The author compares the current state of forecasting to that of medicine before early 1900’s – experts were reluctant to collect data and keep score and relied on instinct.  There are examples of physicians throughout history, including Galen, the second-century physician to Roman emperors, who never applied basic scientific methods and were grossly overconfident.  Even in the early 1900’s, early proponents of using healthcare data were shunned by medical institutions and their peers.  Tetlock thinks there is a similar opportunity with forecasting and improvements can be made in government and business.

The book has practical advice for improving your forecasting that I found very similar to investing – seek out disconfirming evidence, don’t fall to overconfidence and commitment bias, apply different mental models, leverage intelligence of others, and keep track!

The Superforecasters used the following strategies:

  • Start with an outside view (base rate of occurrence) and then use an inside view
  • Break the problem down into knowable and unknowable components
  • Are very precise with their forecasts and keep a record of the accuracy of their forecasts
  • Frequently update their forecasts when new information became available
  • Improve their results when they interact with other superforecasters
  • Seek out disconfirming evidence

Superforecasters had the following common characteristics:

  • Above average intelligence, although intelligence was not the best predictor of success
  • Had an intense curiosity and passion for learning
  • They understand uncertainty and probability and acknowledge it in their forecasts

Detailed Notes

Notable Quotes

  • “I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal.  This may seem basic, but it is amazing how often it is not done and how hard it is to get right” – Bill Gates
  • “Never tell people how to do things. Tell them what to do and they will surprise you with ingenuity” – George Patton
  • “All of which is to say that I’m not sure what 2010 will look like, but I’m sure that it will be very little like what we expect, so we should plan accordingly” – Linton Wells


  • Good Judgement Product was part of a larger effort sponsored by the Intelligence Advanced Research Projects Activity (IARPA)
  • IARPA ran a research project from September 2011 to June 2015 and posed nearly 500 questions on world affairs and gathered over 1 million judgements about the future – Good Judgement Project was one of five teams to compete

Common Illusions of Knowledge

  • Current state of the field of forecasting can be compared to early medicine, in which experts were reluctant to collect data and keep track – most notably Galen, the second-century physician to the Roman emperor
    • Randomized controlled trials were revolutionary to medicine
    • Example of Ernest Codman, a Boston doctor who pushed for more data driven evaluations of hospitals – he lost his Harvard teaching post and was kicked out of MA General Hospital
  • There is a tendency for humans to find explanation for results
    • Experiment in which split brain patients were shown image on one side and then point to an image that was related to it – they were shown a picture of a snowstorm on one side and then pointed to a shovel. Then the other side was shown an image of a chicken claw and asked why they were pointing to a shovel and the person would say something like “because chickens live in a coop and you need a shovel to clean up the chicken coop”
  • People also use attribute substitution– a shortcut assumption that avoids the real question – “should I trust this person/expert to make a decision for me?” vs. “is the decision correct?” and availability bias – substituting easy and available info with the right data and questions
  • Humans use two systems – System 1 is automatic perceptual and cognitive operations and System 2 is conscious thought – System 1 or intuition works well in pattern recognition, but requires hours/years of experience to develop this
  • One of the main shortfalls in IC forecasts were they were not specific enough – example was when the CIA was planning to take over Castro government by landing a small army of Cuban expatriates the Bay of Pigs, JFK’s staff said the plan had “fair chance” of success or 3:1 odds against success, but JFK took that as being very likely
    • It took years for the government to start using more specific numbers in their predictions – CIA told Obama that there was a 70%-90% chance that bin Laden was in the Pakistani compound
  • Forecasts have two characteristics: (i) calibration or how close a series of probabilistic predictions are (numerous measurements) – when someone says there is a 40% chance 10 times, how close are the actual results to 40%? And (ii) resolution – how accurate a forecast is weighted with certainty (90% chance of happening)
  • Forecasts are measured with Brier scores – 0 – 2.0 with 2.0 being the best

Expert Political Judgement Project  

  • Mid 1980s, Tetlock performed a study on forecasting with ~30 people that lasted 21 years – Expert Political Judgement and the participants fell into two major groups:
    • The average expert was no better than random and worse than random in long-term forecasts
    • There was one group that barely outperformed random
  • The underperforming group was very confident and tended to be ideological – piled on support for why they would be right, while the second recognized risk factors/uncertainty. The outperforming group pulled information from a wide range of sources/disciplines (foxes) vs. the experts (hedgehogs)
  • Research shows that confidence and accuracy are positively correlated, but people over estimate confidence
  • British Scientist Francis Galton performed a study on estimating the weight of an ox and found that aggregating the groups estimates was more accurate – published studies in the Wisdom of the Crowds
    • Wisdom of crowds works well because they pull information from multiple sources/perspectives – similar to the mind of a Fox
    • Example of being able to think through other people’s perspectives – poker instructor notes that many poker players will see another play raise and think that they have a strong hand, but when they have a strong hand, they don’t raise as to not scare off the other players, so it doesn’t make sense that they would correlate a raise to a strong hand when they would not raise themselves with a similarly strong hand — she teaches them to think like a dragonfly with 1,000 eyes/perspectives

Good Judgement Project Background and Results  

  • Independent review of the decision to invade Iraq on belief of WMD:
    • The decision making and judgement was sound, but the process was rife with errors
    • Statements of estimates were 100% confident and there were no teams set up to question the prevailing views
    • IARPA was formed in 2006 as a result of the mistake and they held the forecasting competition
  • The Good Judgement project found that the following group of 200 outperformed with the following tactics:
    • Calculate the average of the whole group, but give extra weight to the 40 top forecasters and then extremize the forecast – push it closer to 100% or 0%
      • The wisdom of the crowd gathers all information that is dispersed among all the people – but if everyone had all the information they would become more confident, so that is why they extremize the results
    • Slow regression is more often seen in activities dominated by skill, while faster regression is more associated with chance – and superforecasters in year 1 of the GJP actually did better in year 2 and 3
    • Teams were 23% more accurate than individuals
    • Superforecasting has limits – accuracy of expert predictions declined after 5 years
  • Strategy of superforecasters:
    • Breakdown the question into knowable and unknowable and into as many specific parts as possible – technique from Enrico Fermi – an Italian physicist who asked his question how many piano tuners are there in Chicago? – Number of pianos in Chicago, how often Pianos are tuned each year and how long it takes
    • Start with a Base rate and then go deeper. Example, you are given a description of a family and its members and then asked if the family has a pet.  Most people would get wrapped up in the details, but superforecasters start with a base rate – what % of the broader population has pets and then adjust due to specifics of the case
    • Research is very specific and purposeful – it is not an amble that can lead to distractions
    • Superforecasters often write down their forecast and then take a break – by writing it down they are separating themselves from their idea – George Soros practices this
    • Superforecasters update their forecasters much more frequently and learn how to react to new information
      • Bayes Theorem – your new belief should depend on two things, your prior belief multiplied by the “diagnostic value” of new information
    • Superforecasting requires constant feedback on what works and what doesn’t – police investigators suffer from this because the accuracy of their judgements are proven years later
    • Discussed examples of superforecasters being susceptible to personal bias (emotional connections to certain questions) and also how there is a tendency to be scope insensitive which is due to emotional responses (how much would you pay to clean up 2k birds in oil spill vs. 200K?)

Application to Leadership

  • Moltke was a very successful general in the German military in 1870’s who taught that battle is uncertain and plans should change – leaders were taught to think and recognize uncertainty (Auftragstaktik), but when a decision was made, they were taught to act
    • Orders were very simple, but had a why included
    • George Patton and Petraeus employed these principles
  • Discussion of Random Taleb and Black Swan – Taleb believes what can’t be forecasted is what matters (black swans), but Tecklock believes there are many valuable applications for forecasting and notes that Black Swan events are often defined by their consequences and the scale of those consequences (e.g. actions U.S. took after 9-11) and those consequences are largely forecastable
  • Summarize Superforecasters
    • Philosophical Outlook – nothing is certain, reality is infinitely complex, what happens not meant to be and does not have to happen (alternative histories)
    • Abilities and Thinking Styles – Actively open minded, intellectually curious, reflective, numerate
    • Methods – pragmatic, analytical, dragonfly-eyed, probabilistic, thoughtful updaters
    • Forecasting methods can be imperfect, but are huge improvements – think of credit scores
    • A very important piece of forecasting applications is are they asking the right questions – and the author proposes that pundits may serve this purpose where superforecasters may fall short
  • Other Characteristics of superforecasters
    • Superforecasters were only in the top 20% of the population for intelligence
    • Superforecaster personalities are high on “openness to experience”
    • Superforecasters are comfortable with numbers and embrace probabilistic thinking
    • Ten Commandments for Aspiring Superforecasters:
      1. Triage – focus on questions where your hard work is likely to pay off – concentrate on “Goldilocks zone of difficult”
      2. Break seemingly intractable problems into tractable sub-problems – Enrico Fermi
      3. Strike the right balance between inside and outside views (or base rates) – “How often do things of this sort happen in situations of this sort?”
      4. Strike the right balance between under and overreacting to evidence (advice is not specific enough)
      5. Look for the clashing causal forces at work in each problem – understanding counterarguments
      6. Strive to distinguish as many degrees of doubt as the problem permits but no more – example of specific odds common in sports, but less specificity in national security
      7. Strike the balance between under and over confidence, between prudence and decisiveness
      8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases – conduct unflinching postmortems
      9. Bring out the best in others and let others bring out the best in you
      10. Master the error-balancing bicycle – deep deliberate practice

About the author


Add comment

By BoardofBooks
"I have known no wise people who didn't read all the time — none, zero." – Charlie Munger

Recent Posts

Recent Comments




Buy This Theme
Skip to toolbar