Black swans, randomness, and the "perfect" forecast

I am in the middle of the second of two books by N. N. Taleb that have been tough for me to put down: Fooled by Randomness, and The Black Swan.  "Iconoclastic" is the best one-word description for both books.  I enjoy a good whack on the side of the head every now and then, and these two books are delivering a good combination. 

Themes: Especially in social outcomes, luck (i.e., randomness) is a much bigger factor than we think.  Highly improbable, unpredictable, surprise events (i.e., black swans) have overwhelming consequences that few of us can think to prepare for.  Those of us who can figure out a way to prepare, by the way, can reap large (relative) benefits—because the huge cost (or benefit) of a black swan is always the major component of the unpredictable surprise. 

Here's an example of his point about randomness: How many times have you heard about mutual fund X's "superlative performance over the last five years"?  Our typical reaction to that message is that mutual fund X must have better managers than other funds.  Reason: Our minds are built to assign cause-and-effect whenever possible, in spite of the strong possibility that random chance played a big role in the outcome. 

Below is an illustration of a "perfect forecaster" of the stock market.  (It's a hypothetical example I set up to illustrate Taleb's point about our natural tendency to assign cause-effect to events, and to ignore the role randomness can play.)

Winthrop B. Carlisle looks like a great forecaster, doesn't he?  Twelve consecutive correct forecasts can't be random chance, can it?  Carlisle is "obviously" a talented guy with one heck of an accurate algorithm. 

Let's see what that algorithm is...

Now click on the following thumbnail, a picture of one possible Winthrop B. Carlisle, implementer of the "perfect algorithm":

In summary, randomness and hindsight bias tell us:
• there will always be someone who gets it right twelve consecutive times;
• we won't know who it will be ahead of time; but...
• we will certainly hear about it after the fact from the one who did. 

Taleb's point: Randomness plays a much larger role in social outcomes than we are willing to admit—to ourselves, or in our textbooks.  Our minds, uncomfortable with randomness, are programmed to employ hindsight bias to provide retroactive explanations for just about everything.  Nonetheless, randomness is frequently the only "reason" for many events. 

By the way, I am enjoying Taleb's sense of humor.  Early on, it becomes obvious that he has little to no use for the typical attitudes and methods in a few specific professions; he pillories economists, statisticians, and Wall Street traders with entertaining regularity in both books. 

I was pleasantly surprised to find out that F. A. Hayek is one of his favorite thinkers from the past (...he doesn't consider Hayek an "economist").   Here's an excerpt:

In 1974 [Hayek] received the Bank of Sweden Prize in Economic Sciences in Memory of Alfred Nobel, but if you read his acceptance speech you will be in for a surprise.  It was eloquently called "The Pretense of Knowledge," and he mostly railed about other economists and about the idea of the planner.  He argued against the use of the tools of hard science in the social ones, and depressingly, right before the big boom for these methods in economics.  Subsequently, the prevalent use of complicated equations made the environment for true empirical thinkers worse than it was before Hayek wrote his speech.  Every year a paper or book appears, bemoaning the fate of economics and complaining about its attempts to ape physics.

I am as thoroughly enjoying The Black Swan as much as I did Fooled by Randomness.  I'll be looking at just about everything with an even more skeptical eye from now on. 

You can get an 80-minute feel for his themes by listening to Russell Roberts's interview of Taleb at this EconTalk web page: Taleb on Black Swans.

Comments

[Dave]

Although I havent read his books I have read about Taleb. An interesting note about him: he developed mouth cancer, but never smoked a day in his life! The odds of that happening are incredibley low. He, himself, was a black swan!

For those unfamiliar with the meaning of the title, it comes from the following (forgive me if I paraphrase):

No number of observances of white swans can prove that all swans are white; yet the single observance of a black swan proves the opposite.

[Dave]

Steve, I would also recommend "When Genius Failed" regarding the blow up of LTCM.

[Mike H.]

Steve,

Just wanted to let you know you were just quoted on Kudlow's blog about your Lou Dobbs article. Congrads!

[Bob]

Steve,

With all respect this is nothing new as it pertains to the stock market, analysts and money managers. Pick up "A Random Walk Down Wall Street" published some time ago for timeless insight.

Also, if I'm not mistaken, there are more mutual funds listed on the NYSE than individual stocks. And always remember...past performance is no guarantee of future results. Well, maybe you can be one of the four or five who do remember that sound advice.

Gotta go now....I have to do some stochastic analysis to go with the linear regression on the volume of
puts and calls on SPDRS over the last 2 months.

[Steve]

Bob:
The Black Swan is not the same as modern portfolio theory. Picture a revolver with one billion bullet chambers in the cylinder; the black swan is the single bullet in a single one of those billion chambers. Random walkers and modern portfolio players spin that cylinder every day, put the barrel in their mouth, and pull the trigger; they count on it never, ever going off -- and they are right so frequently that they forget all about that single bullet. And each time the hammer clicks into an empty cylinder, they make a few bucks on average, and marvel at the smoothness and reliability of their trend line -- similar to the smooth trend line of a turkey's daily weight gain, right up to the day before Thanksgiving.

Whenever there's a black swan event (eg, 10/19/87, DJI down 22% in one day), the random walkers and modern portfolio players "blow up" (Taleb's words). That single bullet wipes out everything that's happened before it, and more. Although Taleb doesn't crow about it, he made a boatlaod of money on 10/19/87, while the portfolio theory guys and random walkers were jumping out windows.

[Bob]

Hhhmmm, Steve. If he made a ton of money on 10/19/87 he had to do it in one of of three ways:

1. Been sufficiently hedged against an unpredictable drop which means he likely traded gains before the drop to protect himself.

2. Bought during the drop which means he made his money after the drop. Or..

3. Shorted stocks at the opening bell which means he is awful fast and very lucky.

My guess is that he took precautions against that one unpredictable event (#1) content to rack up slow and steady gains as opposed to going all in. Pigs do get slaughtered, you know.

[woodchuck64]

This post has some relevance to the credit problems roiling the markets right now. Financial "apologists" and out-going CEOs are invoking black swans, aren't they -- extremely low probability events that somehow picked this year of all years to manifest themselves.

But are these mounting losses really due to black swan events or rather just really really bad financial "innovation"? I'm going with the latter.

[Bob]

EA,

I'm with you on the latter. This has happened before and will happen again. Financial withcraft will always be with us.

[Ron Hardin]

I always enjoyed portfolio insurance, as implemented in 1987. It's theoretically impossible to lose money.

So how does anybody lose money? By violating an assumption, namely that option markets continue to function.

Assumptions always turn up, would be a good rule.

[John S.]

I read the first of his books. I kept wishing he would come to the point. By the end of the book it seemed he still hadn't. Perhaps he finally gets there in the second book?

[Steven]

I’m being anal I’ll admit it.

Randomness is not a concept that is given much credence in science or mathematics. Currently randomness is so illusory in the science arena that they have given up on the concept; quantum mechanics is probabilistic in nature. Probabilities are a very different concept from randomness, in the fact that probabilities pre-define possible outcomes.

In mathematics we are able to define precisely what randomness is, but as of current have not been able to present a truly random sequence. Chaos mathematical theory has just about slammed the door shut on randomness.

This should not degrade from the book; it just means that the author is describing highly improbable events and not random events.

I spent over 15 years in this area of study and I do understand that my comments are more than likely viewed as esoteric semantics, however, thank you for this indulgence.