Superforecasting: The Art and Science of Prediction

Notebook Export
Superforecasting: The Art and Science of Prediction
Tetlock, Philip; Gardner, Dan
Citation (Chicago Style): Tetlock, Philip, and Gardner, Dan. Superforecasting: The Art and Science of Prediction. Random House, 2015. Kindle edition.
1. An Optimistic Skeptic
Highlight(orange) – Location 122
I believe it is possible to see into the future, at least in some situations and to some extent, and that any intelligent, open-minded, and hardworking person can cultivate the requisite skills. Call me an “optimistic skeptic.”
Highlight(orange) – Location 252
“I have been struck by how important measurement is to improving the human condition,” Bill Gates wrote. “You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal. … This may seem basic, but it is amazing how often it is not done and how hard it is to get right.”
Highlight(orange) – Location 312
The difference between heavyweights and amateurs, she said, is that the heavyweights know the difference between a 60⁄40 bet and a 40⁄60 bet.
Highlight(orange) – Location 317
When physicians finally accepted that their experience and perceptions were not reliable means of determining whether a treatment works, they turned to scientific testing—and medicine finally started to make rapid advances. The same revolution needs to happen in forecasting.
Highlight(orange) – Location 335
superforecasting demands thinking that is open-minded, careful, curious, and—above all—self-critical. It also demands focus. The kind of thinking that produces superior judgment does not come effortlessly. Only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance.
Highlight(orange) – Location 382
Human thought is beset by psychological pitfalls, a fact that has only become widely recognized in the last decade or two. “So what I want is that human expert paired with a computer to overcome the human cognitive limitations and biases.” 14
2. Illusions of Knowledge
Highlight(orange) – Location 411
We have all been too quick to make up our minds and too slow to change them.
Highlight(blue) – Location 449
frivolous
Highlight(blue) – Location 453
tantalizingly
Highlight(orange) – Location 453
discovery in 1747, when a British ship’s doctor named James Lind took twelve sailors suffering from scurvy, divided them into pairs, and gave each pair a different treatment: vinegar, cider, sulfuric acid, seawater, a bark paste, and citrus fruit.
Highlight(orange) – Location 469
It isn’t perfect. There is no perfection in our messy world. But it beats wise men stroking their chins.
Highlight(orange) – Location 470
Randomized controlled trials
Highlight(orange) – Location 479
cargo cult science has the outward form of science but lacks what makes it truly scientific.
Highlight(orange) – Location 480
“Doubt is not a fearful thing,” Feynman observed, “but a thing of very great value.”
Highlight(orange) – Location 481
It’s what propels science forward. When the scientist tells you he does not know the answer, he is an ignorant man. When he tells you he has a hunch about how it is going to work, he is uncertain about it. When he is pretty sure of how it is going to work, and he tells you, “This is the way it’s going to work, I’ll bet,” he still is in some doubt. And it is of paramount importance, in order to make progress, that we recognize this ignorance and this doubt. Because we have the doubt, we then propose looking in new directions for new ideas. The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test. 11 It was the absence of doubt—and scientific rigor—that made medicine unscientific and caused it to stagnate for so long.
Highlight(orange) – Location 491
The idea of randomized controlled trials was painfully slow to catch on and it was only after World War II that the first serious trials were attempted. They delivered excellent results. But still the physicians and scientists who promoted the modernization of medicine routinely found that the medical establishment wasn’t interested, or was even hostile to their efforts.
Highlight(blue) – Location 500
balked.
Highlight(blue) – Location 505
preliminary
Highlight(blue) – Location 506
vociferous
Highlight(blue) – Location 513
incarceration
Highlight(blue) – Location 513
spartan
Highlight(blue) – Location 520
millennia.
Highlight(blue) – Location 521
palpable
Highlight(blue) – Location 522
eminent
Highlight(blue) – Location 522
riddled
Highlight(blue) – Location 523
meekly
Highlight(blue) – Location 523
acquiesced.
Highlight(blue) – Location 524
pathologist’s
Highlight(blue) – Location 528
swath
Highlight(orange) – Location 529
Why did a man who stressed the importance of not rushing to judgment rush to judgment about whether he had terminal cancer?
Highlight(blue) – Location 540
interrogating
Highlight(orange) – Location 541
decision making is this: first System 1 delivers an answer, and only then can System 2 get involved, starting with an examination of what System 1 decided.
Highlight(blue) – Location 549
sober
Highlight(orange) – Location 553
We tend to go with strong hunches. System 1 follows a primitive psycho-logic: if it feels true, it is.
Highlight(orange) – Location 554
In the Paleolithic world in which our brains evolved, that’s not a bad way of making decisions. Gathering all evidence and mulling it over may be the best way to produce accurate answers, but a hunter-gatherer who consults statistics on lions before deciding whether to worry about the shadow moving in the grass isn’t likely to live long enough to bequeath his accuracy-maximizing genes to the next generation. Snap judgments are sometimes essential. As Daniel Kahneman puts it, “System 1 is designed to jump to conclusions from little evidence.”
Highlight(blue) – Location 563
heuristics—
Highlight(orange) – Location 567
Kahneman gave them an ungainly but oddly memorable label: WYSIATI (What You See Is All There Is).
Highlight(orange) – Location 570
we are creative confabulators hardwired to invent stories that impose coherence on the world.
Highlight(blue) – Location 591
conjures
Highlight(blue) – Location 591
plausible
Highlight(blue) – Location 603
perpetrators
Highlight(orange) – Location 611
no matter how tempting it is to anoint a pet hypothesis as The Truth, alternative explanations must get a hearing. And they must seriously consider the possibility that their initial hunch is wrong. In fact, in science, the best evidence that a hypothesis is true is often an experiment designed to prove the hypothesis is false, but which fails to do so.
Highlight(blue) – Location 630
lump
Highlight(blue) – Location 630
armpit.
Highlight(blue) – Location 630
carcinoma
Highlight(orange) – Location 635
bait and switch : when faced with a hard question, we often surreptitiously replace it with an easy one.
Highlight(blue) – Location 641
oblivious
Highlight(blue) – Location 656
humming,
Highlight(blue) – Location 656
babbling
Highlight(blue) – Location 656
brook
Highlight(blue) – Location 662
dichotomy.
Highlight(orange) – Location 664
While Daniel Kahneman and Amos Tversky were documenting System 1’s failings, another psychologist, Gary Klein, was examining decision making among professionals like the commanders of firefighting teams, and discovering that snap judgments can work astonishingly well.
Highlight(blue) – Location 673
hunkered
Highlight(blue) – Location 674
polemics.
Highlight(orange) – Location 676
pattern recognition.
Highlight(blue) – Location 678
repertoire.
Highlight(blue) – Location 696
prodigy
Highlight(orange) – Location 700
The tip-of-your-nose perspective can work wonders but it can also go terribly awry, so if you have the time to think before making a big decision, do so—and be prepared to accept that what seems obviously true now may turn out to be false later.
Highlight(orange) – Location 710
That’s the power of the tip-of-your-nose perspective. It is so persuasive that for thousands of years physicians did not doubt
Highlight(blue) – Location 711
gargantuan
3. Keeping Score
Highlight(blue) – Location 725
enshrined
Highlight(blue) – Location 731
parse
Highlight(blue) – Location 738
tad
Highlight(blue) – Location 738
reminiscent
Highlight(blue) – Location 741
subsidized
Highlight(blue) – Location 746
truncated
Highlight(blue) – Location 747
bust
Highlight(blue) – Location 751
brash
Highlight(blue) – Location 751
dismissive.
Highlight(blue) – Location 751
scoff
Highlight(blue) – Location 752
nuanced
Highlight(blue) – Location 770
grapple
Highlight(orange) – Location 771
The truth is, the truth is elusive.
Highlight(blue) – Location 773
HOLOCAUST
Highlight(blue) – Location 775
arsenals
Highlight(blue) – Location 784
germane
Highlight(blue) – Location 788
frail
Highlight(blue) – Location 790
stern
Highlight(blue) – Location 796
anointed
Highlight(blue) – Location 803
crumbling
Highlight(blue) – Location 804
ascension
Highlight(blue) – Location 809
adept
Highlight(blue) – Location 811
shiver
Highlight(blue) – Location 823
tacit
Highlight(blue) – Location 824
tedious
Highlight(blue) – Location 834
reckless
Highlight(blue) – Location 837
omnipotent
Highlight(blue) – Location 839
semantic
Highlight(blue) – Location 839
quibble.
Highlight(blue) – Location 852
rigor.
Highlight(blue) – Location 853
seeping
Highlight(blue) – Location 870
topple
Highlight(blue) – Location 870
expatriates
Highlight(blue) – Location 875
unmitigated
Highlight(blue) – Location 892
revulsion.
Highlight(blue) – Location 893
tacky
Highlight(blue) – Location 904
fallacy.
Highlight(blue) – Location 913
sagacious
Highlight(blue) – Location 915
prevalence
Highlight(blue) – Location 916
pilloried
Highlight(blue) – Location 921
rubbery
Highlight(blue) – Location 928
vacuous
Highlight(blue) – Location 931
repudiation
Highlight(blue) – Location 937
asterisk
Highlight(blue) – Location 945
calibration.
Highlight(blue) – Location 952
perturb
Highlight(blue) – Location 960
“resolution.”
Highlight(blue) – Location 970
hubris
Highlight(orange) – Location 978
Brier scores.
Highlight(orange) – Location 978
measure the distance between what you forecast and what actually happened.
Highlight(orange) – Location 993
Nate Silver,
Highlight(orange) – Location 994
Sam Wang,
Highlight(blue) – Location 997
overwrought.
Highlight(blue) – Location 1002
dredging up
Highlight(blue) – Location 1014
anonymity
Highlight(orange) – Location 1017
Age? (The average was forty-three.) Relevant work experience? (The average was 12.2 years.) Education? (Almost all had postgraduate training; half had PhDs.)
Highlight(blue) – Location 1022
lay-people.
Highlight(orange) – Location 1027
Expert Political Judgment: How Good Is It? How Can We Know?
Highlight(orange) – Location 1028
“EPJ.”
Highlight(orange) – Location 1029
the average expert was roughly as accurate as a dart-throwing chimpanzee.
Highlight(orange) – Location 1030
But as students are warned in introductory statistics classes, averages can obscure. Hence the old joke about statisticians sleeping with their feet in an oven and their head in a freezer because the average temperature is comfortable.
Highlight(orange) – Location 1037
The critical factor was how they thought
Highlight(blue) – Location 1039
cornucopian
Highlight(blue) – Location 1052
Archilochus:
Highlight(blue) – Location 1058
prototypic
Highlight(blue) – Location 1070
budge.
Highlight(blue) – Location 1084
emerald
Highlight(blue) – Location 1096
pesky
Highlight(blue) – Location 1096
caveats
Highlight(blue) – Location 1098
crayon.
Highlight(orange) – Location 1103
DRAGONFLY EYE
Highlight(orange) – Location 1106
James Surowiecki’s bestseller The Wisdom of Crowds
Highlight(blue) – Location 1112
demystify.
Highlight(blue) – Location 1121
nullifying
Highlight(orange) – Location 1124
aggregating the judgments of an equal number of people who know lots about lots of different things is most effective because the collective pool of information becomes much bigger.
Highlight(orange) – Location 1125
Aggregations of aggregations can also yield impressive results.
Highlight(orange) – Location 1132
they aggregate.
Note – Location 1132
兼听则明
Highlight(blue) – Location 1163
unflappably
Highlight(blue) – Location 1173
bulging
Highlight(orange) – Location 1179
Unfortunately, aggregation doesn’t come to us naturally. The tip-of-your-nose perspective insists that it sees reality objectively and correctly, so there is no need to consult other perspectives.
Highlight(orange) – Location 1186
Annie Duke,
Highlight(orange) – Location 1198
“They would never raise with any of these really great hands because they don’t want to chase me away.”
Highlight(orange) – Location 1198
Then Duke asks them: Why did you assume that an opponent who raises the bet has a strong hand if you would not raise with the same strong hand? “And it’s not until I walk them through the exercise,” Duke says, that people realize they failed to truly look at the table from the perspective of their opponent.
Highlight(blue) – Location 1202
dilettantes
Highlight(orange) – Location 1208
My fox/hedgehog model is not a dichotomy. It is a spectrum.
Highlight(blue) – Location 1208
“hybrids”—
Highlight(orange) – Location 1213
No model captures the richness of human nature. Models are supposed to simplify things, which is why even the best are flawed. But they’re necessary. Our minds are full of models. We couldn’t function without them. And we often function pretty well because some of our models are decent approximations of reality. “All models are wrong,” the statistician George Box observed, “but some are useful.” The fox/hedgehog model is a starting point, not the end.
4. Superforecasters
Highlight(blue) – Location 1224
grabber
Highlight(blue) – Location 1224
atrocities
Highlight(blue) – Location 1225
oust
Highlight(blue) – Location 1226
menace
Highlight(blue) – Location 1231
gin up
Highlight(blue) – Location 1240
churning
Highlight(blue) – Location 1242
derisively
Highlight(blue) – Location 1246
grim,
Highlight(orange) – Location 1248
Jervis is the author of Why Intelligence Fails
Highlight(orange) – Location 1262
This particular bait and switch—replacing “Was it a good decision?” with “Did it have a good outcome?”—
Highlight(blue) – Location 1263
pernicious.
Highlight(blue) – Location 1263
Savvy
Highlight(blue) – Location 1263
blunder.
Highlight(blue) – Location 1268
oxymoronic
Highlight(blue) – Location 1273
devastating,
Highlight(blue) – Location 1282
dollops
Highlight(blue) – Location 1282
fell prey to
Highlight(blue) – Location 1282
hubris.
Highlight(blue) – Location 1283
Postmortems
Highlight(orange) – Location 1284
“There were no ‘red teams’ to attack the prevailing views, no analyses from devil’s advocates, no papers that provided competing possibilities,”
Highlight(blue) – Location 1286
tartly
Highlight(orange) – Location 1289
postmortem on the 1979 failure to foresee the Iranian revolution—the biggest geopolitical disaster of that era—
Highlight(blue) – Location 1298
synthesize
Highlight(orange) – Location 1303
my 2005 book Expert Political Judgment .
Highlight(orange) – Location 1303
100% Archie Cochrane: don’t believe until you test.
Highlight(orange) – Location 1311
Some analysts think the training is so intuitively compelling that it doesn’t need to be tested. Sound familiar?
Highlight(blue) – Location 1317
capricious,
Highlight(orange) – Location 1321
Did you consider alternative hypotheses? Did you look for contrary evidence?
Highlight(orange) – Location 1323
To have accountability for process but not accuracy is like ensuring that physicians wash their hands, examine the patient, and consider all the symptoms, but never checking to see whether the treatment works.
Highlight(blue) – Location 1326
prodding
Highlight(blue) – Location 1329
cloak-and-dagger
Highlight(blue) – Location 1334
vista.
Highlight(blue) – Location 1336
cushy
Highlight(blue) – Location 1336
exile
Highlight(orange) – Location 1343
human cognitive systems will never be able to forecast turning points in the lives of individuals or nations several years into the future—and heroic searches for superforecasters won’t change that.
Highlight(orange) – Location 1355
Archie Cochrane saw the solution: Quit pretending you know things you don’t and start running experiments. Give the training to one randomly chosen group of forecasters but not another. Keep all else constant. Compare results. If the trainees become more accurate, while the untrained don’t, the training worked.
Highlight(blue) – Location 1361
gauntlet
Highlight(blue) – Location 1363
gutsy
Highlight(blue) – Location 1388
avatar
Highlight(orange) – Location 1404
Doug’s overall Brier score was 0.22,
Highlight(orange) – Location 1405
the Brier score measures the gap between forecasts and reality, where 2.0 is the result if your forecasts are the perfect opposite of reality, 0.5 is what you
Highlight(orange) – Location 1406
would get by random guessing, and 0 is the center of the bull’s-eye.
Highlight(orange) – Location 1409
To get a 0.22, Doug’s average judgment across the eleven-month duration of the question had to be no at roughly 68% confidence—
Highlight(orange) – Location 1411
In year 2, Doug joined a superforecaster team and did even better, with a final Brier score of 0.14,
Highlight(blue) – Location 1445
well-remunerated
Highlight(blue) – Location 1445
loath
Highlight(orange) – Location 1463
Randomness is invisible from the tip-of-your-nose perspective. We can only see it if we step outside ourselves.
Highlight(orange) – Location 1469
“illusion of control,”
Highlight(blue) – Location 1477
incessantly
Highlight(blue) – Location 1479
fallacy
Highlight(blue) – Location 1486
rummaging
Highlight(blue) – Location 1487
fawning
Highlight(blue) – Location 1495
pesky
Highlight(blue) – Location 1495
dichotomies
Highlight(orange) – Location 1498
Michael Mauboussin, a global financial strategist, in his book The Success Equation .
Highlight(orange) – Location 1499
“regression to the mean.”
Highlight(blue) – Location 1518
clustered
Highlight(orange) – Location 1526
Mauboussin notes that slow regression is more often seen in activities dominated by skill, while faster regression is more associated with chance.
Highlight(blue) – Location 1539
irreducible
Highlight(blue) – Location 1550
anointed
5. Supersmart? 105
Highlight(blue) – Location 1568
sclerosis.
Highlight(blue) – Location 1569
debilitating.
Highlight(blue) – Location 1574
dictated
Highlight(blue) – Location 1581
voracious
Highlight(blue) – Location 1590
‘tingle’
Highlight(blue) – Location 1596
rattled
Highlight(blue) – Location 1596
highbrow
Highlight(blue) – Location 1600
Jeopardy!
Highlight(blue) – Location 1603
daunting
Highlight(blue) – Location 1603
vindicated,
Highlight(blue) – Location 1606
plausible,
Highlight(blue) – Location 1608
grueling
Highlight(blue) – Location 1609
crunching
Highlight(blue) – Location 1637
replete
Highlight(orange) – Location 1640
imperiled.
Highlight(blue) – Location 1647
tuners
Highlight(orange) – Location 1648
Enrico Fermi—
Highlight(blue) – Location 1648
concocted
Highlight(blue) – Location 1709
ominous.
Highlight(blue) – Location 1712
exhumed
Highlight(blue) – Location 1722
volleys
Highlight(blue) – Location 1743
nudged
Highlight(orange) – Location 1745
OUTSIDE FIRST
Highlight(blue) – Location 1759
evocative
Highlight(orange) – Location 1797
So a forecaster who starts by diving into the inside view risks being swayed by a number that may have little or no meaning. But if she starts with the outside view, her analysis will begin with an anchor that is meaningful. And a better anchor is a distinct advantage.
Highlight(orange) – Location 1803
If you aimlessly examine one tree, then another, and another, you will quickly become lost in the forest. A good exploration of the inside view does not involve wandering around, soaking up any and all information and hoping that insight somehow emerges. It is targeted and purposeful: it is an investigation, not an amble.
Highlight(orange) – Location 1836
looking for more perspectives.
Highlight(orange) – Location 1848
George Soros exemplifies it. A key part of his success, he has often said, is his mental habit of stepping back from himself so he can judge his own thinking and offer a different perspective—to himself. 14
Highlight(blue) – Location 1860
dissonant
Highlight(blue) – Location 1863
rife
Highlight(blue) – Location 1863
dialectical
Highlight(blue) – Location 1863
banter.
Highlight(blue) – Location 1867
futile.
Highlight(blue) – Location 1871
succumb
Highlight(blue) – Location 1871
migraines.
Highlight(blue) – Location 1871
blurt
Highlight(blue) – Location 1872
thrice—
Highlight(blue) – Location 1875
slogs.
Highlight(orange) – Location 1876
superforecasters score high in need-for-cognition tests.
Highlight(orange) – Location 1877
“Big Five” traits is “openness to experience,”
Highlight(orange) – Location 1887
when people read for pleasure they naturally gravitate to the like-minded.
Highlight(orange) – Location 1900
For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded. It
6. Superquants?
Highlight(blue) – Location 1906
intimidating,
Highlight(blue) – Location 1910
occult
Highlight(orange) – Location 1917
I have yet to find a superforecaster who isn’t comfortable with numbers and most are more than capable of putting them to practical use.
Highlight(blue) – Location 1920
cognoscenti.
Highlight(blue) – Location 1921
esoteric
Highlight(blue) – Location 1921
affinity
Highlight(blue) – Location 1922
sleight
Highlight(blue) – Location 1922
presto!
Highlight(blue) – Location 1926
nuanced
Highlight(blue) – Location 1935
riveted
Highlight(blue) – Location 1935
cluster
Highlight(blue) – Location 1937
strands
Highlight(orange) – Location 1940
Zero Dark Thirty .
Highlight(blue) – Location 1946
concur,”
Highlight(blue) – Location 1949
clusterfuck,
Highlight(orange) – Location 1966
“I encourage the people around me not to tell me what they thought I wanted to hear but what they believed, and to be honest,” Panetta said.
Highlight(orange) – Location 1968
he considered obtaining and presenting diverse views to be a critical part of his job.
Highlight(blue) – Location 1973
mewl
Highlight(blue) – Location 1973
battering
Highlight(blue) – Location 1973
ram.
Highlight(blue) – Location 1974
rubble
Highlight(blue) – Location 1974
scrawled
Highlight(blue) – Location 1975
stomp
Highlight(blue) – Location 1978
“cowed.”
Highlight(blue) – Location 1978
weaklings.
Highlight(blue) – Location 1985
paranoid
Highlight(blue) – Location 1985
schizophrenia.
Highlight(blue) – Location 2001
sanguine.
Highlight(blue) – Location 2007
comically
Highlight(blue) – Location 2007
contrive
Highlight(blue) – Location 2016
sketchy
Highlight(orange) – Location 2099
“Nothing is one hundred percent.”
Highlight(blue) – Location 2104
eradication.
Highlight(orange) – Location 2106
“Uncertainty is real,” Byers writes. “It is the dream of total certainty that is an illusion.”
Highlight(orange) – Location 2108
Scientific facts that look as solid as rock to one generation of scientists can be crushed to dust beneath the advances of the next. 16 All scientific knowledge is tentative. Nothing is chiseled in granite.
Highlight(orange) – Location 2123
Robert Rubin
Highlight(orange) – Location 2124
there is no provable certainty and “it just clicked with everything I’d sort of thought,”
Highlight(orange) – Location 2126
In an Uncertain World
Highlight(blue) – Location 2131
lavishly.
Highlight(blue) – Location 2134
cubicle
Highlight(orange) – Location 2138
this discussion of probabilities,’”
Highlight(blue) – Location 2146
“epistemic”
Highlight(blue) – Location 2147
“aleatory”
Highlight(orange) – Location 2147
Epistemic uncertainty is something you don’t know but is, at least in theory, knowable.
Highlight(blue) – Location 2148
pry
Highlight(orange) – Location 2149
Aleatory uncertainty is something you not only don’t know; it is unknowable. No
Highlight(orange) – Location 2152
Aleatory uncertainty ensures life will always have surprises, regardless of how carefully we plan.
Highlight(orange) – Location 2153
When they sense that a question is loaded with irreducible uncertainty—say, a currency-market question—they have learned to be cautious, keeping their initial estimates inside the shades-of-maybe zone between 35% and 65% and moving out tentatively. They know the “cloudier” the outlook, the harder it is to beat that dart-throwing chimpanzee.
Highlight(orange) – Location 2164
Superforecasters were much more granular.
Highlight(blue) – Location 2174
“stochastic”
Highlight(blue) – Location 2175
granularity
Highlight(blue) – Location 2175
bafflegab.
Highlight(blue) – Location 2187
repertoire,
Highlight(orange) – Location 2207
Only the naive ask “Why?” Those who see reality more clearly don’t bother.
Highlight(blue) – Location 2208
trenchant
Highlight(blue) – Location 2210
consoling
Highlight(orange) – Location 2211
Oprah Winfrey,
Highlight(blue) – Location 2211
stupendous
Highlight(orange) – Location 2212
“there is no such thing as failure. Failure is just life trying to move us in another direction. … Learn from every mistake because every experience, encounter, and particularly your mistakes are there to teach you and force you into being who you are.”
Highlight(orange) – Location 2220
Meaning is a basic human need.
Highlight(orange) – Location 2220
As much research shows, the ability to find it is a marker of a healthy, resilient mind. Among survivors of the 9/11 attacks, for example, those who saw meaning in the atrocity were less likely to suffer post-traumatic stress responses.
Highlight(blue) – Location 2225
avalanche,
Highlight(blue) – Location 2227
cosmos.
Highlight(blue) – Location 2235
imbued
Highlight(orange) – Location 2253
A probabilistic thinker will be less distracted by “why” questions and focus on “how.”
Highlight(blue) – Location 2254
semantic
Highlight(blue) – Location 2254
quibble.
Highlight(orange) – Location 2280
superforecasters got the lowest score of all, firmly on the rejection-of-fate side.
Highlight(orange) – Location 2283
the more a forecaster embraced probabilistic thinking, the more accurate she was.
Highlight(blue) – Location 2287
existential
7. Supernewsjunkies?
Highlight(orange) – Location 2290
Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others—and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.
Highlight(orange) – Location 2316
“When the facts change, I change my mind,” the legendary British economist John Maynard Keynes declared. “What do you do, sir?” The superforecasters do likewise, and that is another big reason why they are super.
Highlight(blue) – Location 2377
prosaic.
Highlight(blue) – Location 2383
placate
Highlight(blue) – Location 2388
astute.
Highlight(blue) – Location 2395
saboteurs.
Highlight(blue) – Location 2405
intransigent—
Highlight(blue) – Location 2406
internment
Highlight(blue) – Location 2409
obstinacy
Highlight(blue) – Location 2409
budge,
Highlight(blue) – Location 2410
grudgingly,
Highlight(blue) – Location 2415
apocryphal.
Highlight(orange) – Location 2432
it’s a lot harder to pull that block out without upsetting other blocks—which makes Jean-Pierre reluctant to tamper with it.
Highlight(orange) – Location 2439
When a block is at the very base of the tower, there’s no way to remove it without bringing everything crashing down. This extreme commitment leads to extreme reluctance to admit error,
Highlight(blue) – Location 2442
sledgehammer
Highlight(orange) – Location 2443
This suggests that superforecasters may have a surprising advantage: they’re not experts or professionals, so they have little ego invested in each forecast.
Highlight(blue) – Location 2451
stereotype
Highlight(blue) – Location 2453
sadomasochistic
Highlight(blue) – Location 2467
tidbit
Highlight(blue) – Location 2474
ephemeral
Highlight(blue) – Location 2476
gin rummy,”
Highlight(blue) – Location 2488
prose
Highlight(orange) – Location 2491
Good updating is all about finding the middle passage.
Highlight(blue) – Location 2497
trawls
Highlight(orange) – Location 2531
But the forecaster who carefully balances old and new captures the value in both—and puts it into her new forecast.
Highlight(orange) – Location 2532
The best way to do that is by updating often but bit by bit.
Highlight(blue) – Location 2533
billiards
Highlight(orange) – Location 2541
Thomas Bayes.
Highlight(orange) – Location 2545
P(H|D)/P(-H|D) = P(D|H) • P(D|-H) • P(H)/P(-H) Posterior Odds = Likelihood Ratio • Prior Odds
Highlight(orange) – Location 2547
your new belief should depend on two things—your prior belief (and all the knowledge that informed it) multiplied by the “diagnostic value” of the new information.
Highlight(blue) – Location 2559
botched
Highlight(blue) – Location 2578
skeleton
Highlight(blue) – Location 2589
barbarous.”
Highlight(blue) – Location 2591
caveats.
Highlight(blue) – Location 2591
nuanced
8. Perpetual Beta
Highlight(blue) – Location 2600
cratered.
Highlight(orange) – Location 2605
Simpson has a “growth mindset,” which Dweck defines as believing that your abilities are largely the product of effort—that you can “grow” to the extent that you are willing to work hard and learn.
Highlight(orange) – Location 2635
John F. Wasik, the author of a book on Keynes’s investments.
Highlight(blue) – Location 2637
insatiably
Highlight(blue) – Location 2638
ungrudgingly.
Highlight(orange) – Location 2639
“There is no harm in being sometimes wrong, especially if one is promptly found out,”
Highlight(orange) – Location 2641
“Keynes is always ready to contradict not only his colleagues but also himself whenever circumstances make this seem appropriate,” reported a 1945 profile of the “consistently inconsistent” economist.
Highlight(blue) – Location 2643
rebukes
Highlight(blue) – Location 2646
unblemished.
Highlight(orange) – Location 2646
In 1920 he was nearly wiped out when his foreign currency forecasts turned out to be horribly wrong. He found his footing again and made a fortune for himself and others in the 1920s.
Highlight(orange) – Location 2647
But just like Mary Simpson in 2008, Keynes didn’t see the disaster of 1929 coming and he again lost big. But he bounced back and did even better than before.
Highlight(orange) – Location 2648
For Keynes, failure was an opportunity to learn—to identify mistakes, spot new alternatives, and try again. After his bad currency calls, Keynes didn’t retreat to the safe and easy. He embraced new ideas in the early 1920s,
Highlight(blue) – Location 2650
stodgy
Highlight(orange) – Location 2652
Stock prices do not always reflect the true value of companies, so an investor should study a company thoroughly and really understand its business, capital, and management when deciding whether it had sufficient underlying value to make an investment for the long term worthwhile. In the United States, about the same time, this approach was developed by Benjamin Graham, who called it “value investing.” It became the cornerstone of Warren Buffett’s fortune.
Highlight(orange) – Location 2656
The one consistent belief of the “consistently inconsistent” John Maynard Keynes was that he could do better. Failure did not mean he had reached the limits of his ability. It meant he had to think hard and give it another go. Try, fail, analyze, adjust, try again: Keynes cycled through those steps ceaselessly.
Highlight(blue) – Location 2661
flop
Highlight(blue) – Location 2676
curvature
Highlight(blue) – Location 2681
“tacit
Highlight(blue) – Location 2682
bruising
Highlight(blue) – Location 2684
ordeal
Highlight(orange) – Location 2685
It should be equally obvious that learning to forecast requires trying to forecast. Reading books on forecasting is no substitute for the experience of the real thing.
Highlight(blue) – Location 2693
infuse
Highlight(blue) – Location 2693
pallid
Highlight(blue) – Location 2704
calibration—
Highlight(blue) – Location 2707
torrential
Highlight(blue) – Location 2710
putt
Highlight(blue) – Location 2716
fuzzy
Highlight(blue) – Location 2721
astrology,
Highlight(blue) – Location 2728
incumbent
Highlight(blue) – Location 2732
dredging
Highlight(blue) – Location 2747
slanted
Highlight(blue) – Location 2756
swish
Highlight(orange) – Location 2768
So if you were thinking of becoming a better political or business forecaster by playing bridge, forget it. To get better at a certain type of forecasting, that is the type of forecasting you must do—over and over again, with good feedback telling you how your training is going, and a cheerful willingness to say, “Wow, I got that one wrong. I’d better think about why.”
Highlight(blue) – Location 2774
drubbing
Highlight(blue) – Location 2781
sparse
Highlight(blue) – Location 2792
apartheid
Highlight(blue) – Location 2800
GRIT
Highlight(blue) – Location 2803
tentative.
Highlight(blue) – Location 2808
synapses.”
Highlight(blue) – Location 2808
oncologist
Highlight(blue) – Location 2815
hippie
Highlight(blue) – Location 2827
revel
Highlight(blue) – Location 2827
adulation,
Highlight(blue) – Location 2829
masquerading
Highlight(blue) – Location 2833
tenacity
Highlight(blue) – Location 2833
daunting.
Highlight(blue) – Location 2838
revealing
Highlight(blue) – Location 2840
inert
Highlight(orange) – Location 2853
In philosophic outlook, they tend to be: C AUTIOUS : Nothing is certain H UMBLE : Reality is infinitely complex N ONDETERMINISTIC : What happens is not meant to be and does not have to happen In their abilities and thinking styles, they tend to be: A CTIVELY OPEN-MINDED : Beliefs are hypotheses to be tested, not treasures to be protected I NTELLIGENT AND KNOWLEDGEABLE, WITH A “NEED FOR COGNITION” : Intellectually curious, enjoy puzzles and mental challenges R EFLECTIVE : Introspective and self-critical N UMERATE : Comfortable with numbers In their methods of forecasting they tend to be: P RAGMATIC : Not wedded to any idea or agenda A NALYTICAL : Capable of stepping back from the tip-of-your-nose perspective and considering other views D RAGONFLY-EYED : Value diverse views and synthesize Judge using many grades of maybe T HOUGHTFUL UPDATERS : When facts change, they change their minds G OOD INTUITIVE PSYCHOLOGISTS : Aware of the value of checking thinking for cognitive and emotional biases In their work ethic, they tend to have: A GROWTH MINDSET : Believe it’s possible to get better G RIT : Determined to keep at it however long it takes
9. Superteams
Highlight(blue) – Location 2894
guerrilla
Highlight(blue) – Location 2905
harebrained.
Highlight(blue) – Location 2907
blatant
Highlight(blue) – Location 2907
contingency
Highlight(blue) – Location 2914
novice
Highlight(blue) – Location 2921
naval
Highlight(blue) – Location 2921
blockade.
Highlight(blue) – Location 2922
averted,
Highlight(blue) – Location 2922
exhaled.
Highlight(blue) – Location 2923
nadir,
Highlight(blue) – Location 2923
zenith,
Highlight(blue) – Location 2925
bungled
Highlight(orange) – Location 2927
Victims of Groupthink
Highlight(orange) – Location 2927
Irving Janis—
Highlight(blue) – Location 2933
tacitly
Highlight(blue) – Location 2936
implausible
Highlight(blue) – Location 2937
botched
Highlight(orange) – Location 2938
It identified cozy unanimity as the key problem and
Highlight(orange) – Location 2951
Teams can cause terrible mistakes. They can also sharpen judgment and accomplish together what cannot be done alone. Managers tend to focus on the negative or the positive but they need to see both. As mentioned earlier, the term “wisdom of crowds” comes from James Surowiecki’s 2004 bestseller of the same name, but Surowiecki’s title was itself a play on the title of a classic 1841 book, Extraordinary Popular Delusions and the Madness of Crowds , which chronicled a litany of collective folly. Groups can be wise, or mad, or both. What makes the difference isn’t just who is in the group, Kennedy’s circle of advisers demonstrated. The group is its own animal.
Highlight(orange) – Location 2959
teams might foster cognitive loafing.
Highlight(orange) – Location 2960
Worse, forecasters can become too friendly, letting groupthink set in.
Highlight(orange) – Location 2960
These two tendencies can reinforce each other.
Highlight(orange) – Location 2962
groups also let people share information and perspectives.
Highlight(orange) – Location 2963
It helps make dragonfly eye work, and aggregation is critical to accuracy.
Highlight(orange) – Location 2963
aggregation can only do its magic when people form judgments independently,
Highlight(orange) – Location 2971
If forecasters can keep questioning themselves and their teammates, and welcome vigorous debate, the group can become more than the sum of its parts.
Highlight(blue) – Location 2982
gleaned
Highlight(orange) – Location 2982
On the one hand, we warned, groupthink is a danger. Be cooperative but not deferential. Consensus is not always good; disagreement not always bad. If you do happen to agree, don’t take that agreement—in itself—as proof that you are right. Never stop doubting.
Highlight(orange) – Location 2984
On the other hand, the opposite of groupthink—rancor and dysfunction—is also a danger. Team members must disagree without being disagreeable, we advised.
Highlight(orange) – Location 2986
Practice “constructive confrontation,” to use the phrase of Andy Grove, the former CEO of Intel. Precision questioning is one way to do that.
Highlight(blue) – Location 3006
unequivocal:
Highlight(orange) – Location 3006
on average, teams were 23% more accurate than individuals.
Highlight(blue) – Location 3008
anointed
Highlight(blue) – Location 3013
hubris
Highlight(blue) – Location 3018
harangues.
Highlight(blue) – Location 3055
nucleus
Highlight(blue) – Location 3062
tidbits,
Highlight(orange) – Location 3074
If I think a stock is a good value at a certain price, I may offer to buy yours. If you agree with my judgment, you won’t sell. If you think I’m wrong, you will.
Highlight(blue) – Location 3083
ardent
Highlight(blue) – Location 3088
slackens.
Highlight(orange) – Location 3117
My Wharton colleague Adam Grant categorizes people as “givers,” “matchers,” and “takers.”
Highlight(blue) – Location 3119
chump
Highlight(orange) – Location 3121
the pro-social example of the giver can improve the behavior of others, which helps everyone, including the giver—which explains why Grant has found that givers tend to come out on top.
Highlight(blue) – Location 3129
marinate
Highlight(blue) – Location 3146
sketchy
10. The Leader’s Dilemma
Highlight(orange) – Location 3168
Confidence
Highlight(orange) – Location 3169
Decisiveness
Highlight(blue) – Location 3170
ruminate
Highlight(orange) – Location 3170
vision—
Highlight(blue) – Location 3178
hierarchy,
Highlight(orange) – Location 3186
Helmuth von Moltke.
Highlight(orange) – Location 3193
The most urgent is to never entirely trust your plan. “No plan of operations extends with certainty beyond the first encounter with the enemy’s main strength,”
Highlight(orange) – Location 3197
Improvisation is essential.
Highlight(blue) – Location 3208
cavalry
Highlight(orange) – Location 3215
If necessary, discuss your orders. Even criticize them. And if you absolutely must—and you better have a good reason—disobey them.
Highlight(blue) – Location 3217
fractious
Highlight(orange) – Location 3226
“If one wishes to attack, then one must do so with resoluteness. Half measures are out of place,”
Highlight(orange) – Location 3237
What ties all of this together—from “nothing is certain” to “unwavering determination”—is the command principle of Auftragstaktik . Usually translated today as “mission command,”
Highlight(blue) – Location 3249
rattled.
Highlight(blue) – Location 3254
improvise.
Highlight(orange) – Location 3282
“The command of an army and its subordinate units requires leaders capable of judgment, with clear vision and foresight, and the ability to make independent and decisive decisions and carry them out unwaveringly and positively.”
Highlight(blue) – Location 3295
vindicated
Highlight(orange) – Location 3308
George Patton was one. “Never tell people how to do things,” he wrote, succinctly capturing the spirit of Auftragstaktik: “Tell them what to do, and they will surprise you with their ingenuity.”
Highlight(blue) – Location 3312
serene,
Highlight(blue) – Location 3334
ammunition.
Highlight(blue) – Location 3340
insurgency
Highlight(blue) – Location 3347
spout
Highlight(blue) – Location 3347
platitudes,
Highlight(blue) – Location 3348
bumper
Highlight(blue) – Location 3364
mush,
Highlight(blue) – Location 3366
prattle
Highlight(blue) – Location 3367
tenure.”
Highlight(blue) – Location 3376
iconoclasts.”
Highlight(blue) – Location 3377
AUFTRAGSTAKTIK
Highlight(orange) – Location 3379
“We let our people know what we want them to accomplish. But—and it is a very big ‘but’—we do not tell them how to achieve those goals.”
Highlight(orange) – Location 3383
“Have backbone; disagree and commit” is one of Jeff Bezos’s fourteen leadership principles drilled into every new employee at Amazon. It continues: “Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.”
Highlight(blue) – Location 3398
HUMILITY
Highlight(blue) – Location 3398
vexing
Highlight(blue) – Location 3399
cadet
Highlight(blue) – Location 3401
chutzpah
Highlight(orange) – Location 3405
“There is nothing like danger to focus the mind,” Soros
Highlight(blue) – Location 3409
isolationist
Highlight(blue) – Location 3410
Gibraltar
Highlight(blue) – Location 3427
riddled
Highlight(blue) – Location 3431
inaugural
Highlight(blue) – Location 3434
dangling
Highlight(blue) – Location 3437
dissonance
Highlight(blue) – Location 3442
dissonance
Highlight(orange) – Location 3442
“The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function,” F. Scott Fitzgerald observed in “The Crack-Up.”
Highlight(blue) – Location 3447
conflate
Highlight(blue) – Location 3454
squirm?
11. Are They Really So Super?
Highlight(orange) – Location 3457
Daniel Kahneman
Highlight(blue) – Location 3459
banter
Highlight(orange) – Location 3461
Kahneman cut to the chase: “Do you see them as different kinds of people, or as people who do different kinds of things?”
Highlight(blue) – Location 3499
inoculate
Highlight(blue) – Location 3501
adversarial collaboration,
Highlight(orange) – Location 3503
scope insensitivity.
Highlight(orange) – Location 3514
Scope recedes into the background—and out of sight, out of mind.
Highlight(orange) – Location 3572
many events dubbed black swans are actually gray.
Highlight(blue) – Location 3573
dazzling
Highlight(blue) – Location 3574
plot
Highlight(blue) – Location 3580
fretting
Highlight(blue) – Location 3589
adept.
Highlight(orange) – Location 3591
history is not just about black swans. Look at the inch-worm advance in life expectancy. Or consider that an average of 1% annual global economic growth in the nineteenth century and 2% in the twentieth turned the squalor of the eighteenth century and all the centuries that preceded it into the unprecedented wealth of the twenty-first. 7 History does sometimes jump. But it also crawls, and slow, incremental change can be profoundly important.
Highlight(blue) – Location 3597
specify
Highlight(orange) – Location 3605
On July 14, 1789, a mob took control of a prison in Paris known as the Bastille,
Highlight(orange) – Location 3608
July 14 is the national holiday of France.
Highlight(blue) – Location 3617
wrath
Highlight(blue) – Location 3617
cusp
Highlight(blue) – Location 3626
collaboration,
Highlight(blue) – Location 3660
ritualistic
Highlight(blue) – Location 3662
smack
Highlight(orange) – Location 3663
“Plans are useless,” Eisenhower said about preparing for battle, “but planning is indispensable.”
Highlight(blue) – Location 3683
cardinal
Highlight(orange) – Location 3707
Europe is about to plunge into a world war that will kill ten million—followed by another world war that will kill sixty million.
Highlight(blue) – Location 3709
crackpot.
Highlight(blue) – Location 3712
analogous
Highlight(blue) – Location 3714
looming
Highlight(blue) – Location 3724
annihilation,
Highlight(orange) – Location 3726
Daniel Kahneman
Highlight(blue) – Location 3729
zygote
Highlight(blue) – Location 3729
fetus
Highlight(orange) – Location 3730
That means there was only a 12.5% chance that all three leaders would be born male, and an 87.5% chance that at least one would be born female.
Highlight(orange) – Location 3734
Immersion in what-if history can give us a visceral feeling for Taleb’s vision of radical indeterminacy. Savoring how history could have generated an infinite array of alternative outcomes and could now generate a similar array of alternative futures, is like contemplating the one hundred billion known stars in our galaxy and the one hundred billion known galaxies. It instills profound humility.
Highlight(blue) – Location 3740
puny,
Highlight(blue) – Location 3740
sniff
12. What’s Next? 250
Highlight(orange) – Location 3748
The one highly probable outcome was that whatever happened pundits would immediately explain why.
Highlight(blue) – Location 3762
blithely
Highlight(blue) – Location 3766
vagary
Highlight(orange) – Location 3780
Forecast, measure, revise: it is the surest path to seeing better.
Highlight(blue) – Location 3781
gulled
Highlight(blue) – Location 3784
heed
Highlight(blue) – Location 3790
tumultuous
Highlight(blue) – Location 3796
acumen—
Highlight(blue) – Location 3799
demoted
Highlight(blue) – Location 3800
prophet
Highlight(blue) – Location 3800
stumblebum.
Highlight(blue) – Location 3802
unseemly
Highlight(blue) – Location 3804
livid
Highlight(blue) – Location 3806
slightest.
Highlight(blue) – Location 3809
adornments
Highlight(orange) – Location 3810
It follows that the goal of forecasting is not to see what’s coming. It is to advance the interests of the forecaster and the forecaster’s tribe. Accurate forecasts may help do that sometimes, and when they do accuracy is welcome, but it is pushed aside if that’s what the pursuit of power requires.
Highlight(blue) – Location 3813
holocaust
Highlight(blue) – Location 3814
rouse
Highlight(blue) – Location 3824
dogmatic.
Highlight(blue) – Location 3826
cusp
Highlight(blue) – Location 3833
flabbergasting
Highlight(blue) – Location 3837
intemperate
Highlight(blue) – Location 3838
unfurled
Highlight(blue) – Location 3839
unassailable.
Highlight(blue) – Location 3842
fledgling
Highlight(blue) – Location 3847
ramparts.
Highlight(blue) – Location 3854
contingent
Highlight(blue) – Location 3854
stringent
Highlight(orange) – Location 3854
Those that deliver are expanded, those that fail are shuttered. In line with Bill Gates’s insistence on having clear goals and measures, the Gates Foundation, one of the world’s largest, is renowned for the rigor of its evaluations.
Highlight(blue) – Location 3875
puff
Highlight(blue) – Location 3876
whipsawed
Highlight(orange) – Location 3892
cautiously optimistic that my work may contribute to an evidence-based forecasting movement.
Highlight(blue) – Location 3900
smitten
Highlight(orange) – Location 3900
“Not everything that counts can be counted,” goes a famous saying, “and not everything that can be counted counts.” 13
Highlight(blue) – Location 3905
rampant,
Highlight(blue) – Location 3907
sacred
Highlight(blue) – Location 3907
totems
Highlight(blue) – Location 3908
wretched
Highlight(orange) – Location 3909
Numbers must be constantly scrutinized and improved, which can be an unnerving process because it is unending. Progressive improvement is attainable. Perfection is not. 15
Highlight(blue) – Location 3917
whimsically
Highlight(blue) – Location 3918
“flighty
Highlight(blue) – Location 3924
unnerving
Highlight(blue) – Location 3926
trivial.
Highlight(blue) – Location 3926
colonels
Highlight(blue) – Location 3928
glided
Highlight(blue) – Location 3928
bravura
Highlight(blue) – Location 3939
armistice
Highlight(blue) – Location 3943
pertinent
Highlight(blue) – Location 3944
rebuff
Highlight(blue) – Location 3944
artillery
Highlight(blue) – Location 3948
clustering
Highlight(blue) – Location 3949
pointillism.
Highlight(blue) – Location 3949
dabbing
Highlight(blue) – Location 3951
sweeping
Highlight(orange) – Location 3955
Foresight is one element of good judgment, but there are others, including some that cannot be counted and run through a scientist’s algorithms—moral judgment, for one. Another critical dimension of good judgment is asking good questions.
Highlight(orange) – Location 3958
one way to identify a good question is what I call the smack-the-forehead test: when you read the question after time has passed, you smack your forehead and say, “If only I had thought of that before!”
Highlight(blue) – Location 3961
gnaws
Highlight(blue) – Location 3966
sectarianism
Highlight(blue) – Location 3971
incisiveness
Highlight(blue) – Location 3974
ramifications
Highlight(blue) – Location 3976
largesse,”
Highlight(blue) – Location 4003
coalesced:
Highlight(blue) – Location 4012
agnostic
Highlight(blue) – Location 4013
vehemently
Highlight(blue) – Location 4020
fracas
Highlight(blue) – Location 4020
fraternities.
Highlight(blue) – Location 4034
gloating,
Highlight(blue) – Location 4035
arenas,
Highlight(blue) – Location 4035
strident
Highlight(blue) – Location 4037
bullhorns.
Highlight(blue) – Location 4037
voluble
Epilogue
Highlight(blue) – Location 4042
Epilogue
Highlight(blue) – Location 4044
Cornhusker,”
Highlight(orange) – Location 4046
precision—
Highlight(orange) – Location 4051
intellectual humility.
Highlight(orange) – Location 4051
Bill knows what he doesn’t know, and respects those who do.
Highlight(orange) – Location 4054
“There are good pundits and bad pundits for my purposes, of course.
Highlight(orange) – Location 4055
The bad ones issue their predictions with no supporting arguments, expecting their readers to treat their pronouncements like the word from Mt. Sinai; or they back their forecasts with anecdotes rather than useful facts. The good ones argue the case for their forecasts; in fact, I see them as functioning something like lawyers in an adversarial judicial system: they put forth the best argument they can for why X is going to happen, and I consider everybody’s arguments, dig further into background as necessary, and come up with my own forecast as a weighted sum of theirs.”
Highlight(blue) – Location 4061
amenable
Highlight(blue) – Location 4065
irreducible
Highlight(orange) – Location 4065
These are the moments when the laws of statistics tell us to shift bets and put our money on massive regression to the mean.
Highlight(blue) – Location 4070
salutary
Highlight(orange) – Location 4072
I can’t imagine a better description of the “try, fail, analyze, adjust, try again” cycle—and of the grit to keep at it and keep improving. Bill Flack is perpetual beta.
Appendix: Ten Commandments for Aspiring Superforecasters
Highlight(blue) – Location 4090
contenders,
Highlight(orange) – Location 4091
Certain classes of outcomes have well-deserved reputations for being radically unpredictable (e.g., oil prices, currency markets).
Highlight(blue) – Location 4097
ballparking
Highlight(blue) – Location 4097
extraterrestrial
Highlight(blue) – Location 4099
verbiage.
Highlight(blue) – Location 4103
vicinity
Highlight(blue) – Location 4107
daunting
Highlight(blue) – Location 4123
tardiness
Highlight(blue) – Location 4126
hygiene.
Highlight(blue) – Location 4127
teasing
Highlight(blue) – Location 4128
ferret
Highlight(blue) – Location 4130
peninsula.
Highlight(blue) – Location 4133
verbiage,
Highlight(blue) – Location 4139
devout
Highlight(blue) – Location 4139
dove
Highlight(blue) – Location 4140
devout hawk
Highlight(blue) – Location 4141
nudge
Highlight(blue) – Location 4157
granular
Highlight(blue) – Location 4162
rigorous
Highlight(blue) – Location 4166
dawdling
Highlight(blue) – Location 4172
unflinching
Highlight(blue) – Location 4172
postmortems:
Highlight(blue) – Location 4174
ramifications).
Highlight(orange) – Location 4182
“Managing is like holding a dove in your hand. If you hold it too tightly you kill it, but if you hold it too loosely, you lose it.” 4
Highlight(blue) – Location 4189
deliberative
Highlight(orange) – Location 4190
“It is impossible to lay down binding rules,” Helmuth von Moltke warned, “because two cases will never be exactly the same.” 5 As in war, so in all things. Guidelines are the best we can do in a world where nothing is certain or exactly repeatable.