In designing great digital experiences for our clients we bring into play multiple research inputs. Two of which, site analytics and comparative review, can have a potentially damaging Achilles Heel – survivorship bias.
What is survivorship bias?
A type of selection bias, the basic premise of survivorship bias is that we tend to distort data sets by focusing on successful examples and ignoring failures, as they did not survive to be measured.
An often cited example was the work done during World War II on improving bomber losses due to enemy fire. When bombers were returning from missions with heavy damage, say in their tail section, engineers were looking at this and suggesting that the tail needed to be reinforced. However, this analysis did not include the planes that had been shot down, which means that it could have been a potential weakness in say, the wings, that was causing the losses, and the tails were already strong enough. The engineers could only see the surviving aircraft and this biased their thinking.
We see survivorship bias often raising its ugly head in studies of human success. We all regularly see click-bait headlines and new inspirational books along the lines of “The 50 Habits of the Most Successful CEOs” or similar propositions. As we read, the study reveals that, apparently these 50 CEOs all eat oatmeal for breakfast. But, by looking at just the successful CEOs, we don’t see the full data set, including unsuccessful CEOs and everyone else on the planet that may happen to eat oatmeal for breakfast. This is a classic example of survivorship bias.
So, how does this play out in digital experience design?
The first challenge is data. We use data to look at the success of current experiences, such as the value of content on a certain page, or whether one call to action works better than another, etc. Yet, what we are seeing today on a site or experience is the surviving content, design and interactions. Content could have been deleted during development, pages evolved over time, interactions tweaked. So, while we can see how that specific experience is doing at that time, we can’t see what might have been, because essentially we have just one survivor to review.
Survivorship bias also kicks in when looking at competitor and comparator digital experiences to benchmark against. Let’s say we are working with an airline, and we look at its direct competitors, we are not, by default, looking at competitors that may have failed in the past, gone bankrupt, merged, etc. While it may be argued that we don’t want to copy failure, we can still learn a lot by understanding the widest range of customer experiences as possible.
A good example of this is from a past client in the direct-to-consumer software space. They were very analytics and data-driven, optimizing their main site continuously. When they saw that a comparison table was increasing conversion on a product category page, they started applying the table concept on many more pages. Unfortunately, these changes started to negatively impact overall site conversion. Just as with the CEOs and oatmeal example, they had focused on one success instead of looking at the full picture.
What can we do to avoid survivorship bias?
Survivorship bias is a natural human tendency and in digital experience design we are often dealing with incomplete data sets and research inputs. So, the first step is to understand how we are prone to this type of bias and specifically challenge it using techniques such as:
- Multiple data inputs: Find as many different inputs as possible for the design process. For example, contrast analytics data with primary user research.
- Imaginary scenarios: Use alternate mental models to ask ‘what if’? As with our CEOs and oatmeal example, ask what if the CEOs had eaten eggs for breakfast? What would this have done to our conclusions, and hence, is the conclusion valid?
- Understand context: For specific design elements try and pull in data and research inputs that help you understand the context. In our example of the comparison table on the product category page, context would say why this is a great idea – on that page, users are making a decision as to which one of multiple products to pick, so a comparison table works well. But, what is the context the user has in mind on a different page, and hence is the table useful there?
- Increase data with testing: Where possible, eliminate the bias by running multi-variant testing on the experience. Don’t just test A/B scenarios, but test multiple versions completely to ensure the failures survive in the data set.
Now, I’m off to eat my oatmeal.