Selection bias
Putting success anecdotes in perspective
When people tell career or business stories, they are typically offering interesting anecdotes, but not really evidence. Such a story typically takes the form of “this business took these actions and it succeeded,” with an implicit – sometimes explicit – claim that if some other business took similar actions, it would succeed as well. Likewise, in the career space, there are many examples of people who explain how they made career choices, or how they think about their career. Some famous, rich, and/or powerful person talks about the choices that they made. In their story there is, again, some implicit sense that if you were to do roughly what they had done, roughly the way that they did it, you could also be roughly as famous / rich / powerful as they have become.
Usually, there is no dishonesty intended in these kinds of presentations. For the career stories in particular, the situation can be as simple as asking someone how they got where they are, and they answer the question. All the inference about repeatability is happening in the minds of the audience. But that kind of inference is incorrect. A better reaction by an audience member would be, “I wonder how many other people pursued a similar strategy and did not have as good a result?”
Whatever the strategy is, whether it's “following your dreams,” or “work very hard,” or “stay in school,” the fact that it worked for that person or perhaps even some small set of people doesn't give much useful information about whether it is a successful strategy in general. This is the problem of selection bias. At the same time, there's nothing that means that one of these strategies is a bad idea either. What's hard is to treat such an anecdote as simply one instance, and not to turn it into some larger lesson.
It would be genuinely novel and helpful in these business / career stories if the tellers would fill in the missing elements:
What happened to people or companies who didn’t pursue this strategy?
What happened to other people or companies who did pursue this strategy?
If this is a genuinely beneficial strategy, does it remain that way even if everyone pursues it?
The last point is important, but subtle. Many situations are zero-sum, meaning not only that there are winners and losers but also that the bigger the winner’s winnings, the bigger the losers’ losses. In such a situation, it’s hopeless for everyone to pursue the same strategy – or at least, most of them will be disappointed.
(Of course, people often demonstrate a capacity for self-deception and unjustified optimism. One story tells about an officer briefing his troops that the upcoming battle will be so fierce that only a quarter of them will survive; each individual soldier then thinks, “those poor bastards” about his soon-to-die comrades – but no-one thinks about his own 75%-likelihood death.)
However, not everything is zero-sum. Sometimes, there is a situation that is not about picking winners and losers. That may be a situation where there is a genuine technical shift, and something objectively better replaces something worse. Closely aligned, although not necessarily identical, is a genuine economic shift in which there is an overall net gain of wealth, not just its reallocation from some particular parties to some other parties.
My experience in the technical space of networking includes a time around 2005, selling one of those genuine increments in technical functionality. We had quite a few situations in which people confused our technology with an older approach that hinged on Quality of Service (QoS). QoS is a classic example of picking winners and losers: the technology makes no difference at all until the network is overloaded, and at that point what happens is effectively a kind of triage. When capacity is exhausted, QoS means that some traffic is favored, other traffic is disfavored, and still other traffic is discarded entirely. Within that paradigm, it’s great to be a winner, but inherently the winners win because losers lose. No amount of exhortation or positive thinking is sufficient for everyone to win if a QoS system is managing an overload.
Our technology was much more appealing than QoS. With our approach, network traffic was both smaller and faster than with QoS. Better yet, those effects were visible even on lightly-loaded networks, so the benefits were apparent without overload. For people who understood the difference, the advantage was clear. Instead of winners taking something away from losers, everyone was winning.
(Curiously, there were some people who were so invested in the QoS paradigm that they dismissed our approach until we had QoS capabilities ourselves. We eventually decided that it was easier to build those features and recommend against using them. When our argument against QoS looked like competitive defensiveness, customers couldn’t take it seriously. When we instead had QoS features and supported them – but recommended our proprietary features as superior – it became clear that we had a genuine advantage. I think that’s probably an example of “the customer is always right, even when they’re wrong.”)
All too often, business and career stories seem to be like anecdotes that might be told by “winner” traffic, unaware that it won by forcing others to lose and unaware that its paradigm requires mostly losing. It would be more interesting if business stories and career stories were at least aware of potential selection bias. Of course, it would be better yet if those stories told of situations more like the dramatic-improvement-for-all.