Quality-related data collection is useful, but statistics can also deliver misleading and even dysfunctional results when incomplete. This is often the case when information is collected only from surviving people or products, extremely satisfied or dissatisfied customers, or propagators of bad news about relatively rare incidents. This is simply an extension of the basic principle that a sample should reflect the entire population rather than just a portion—especially a portion that might self-select for the indicated reasons.

Survivor bias

Survivor bias occurs when samples include only people or items that survive harmful conditions. When World War I started in 1914, the soldiers of most armies wore cloth or leather caps that provided almost no protection against shrapnel and small arms fire. The French, for example, replaced their world-famous kepi with the equally recognizable Adrian helmet only in 1915. The British followed suit and introduced the Brodie, later called “Tin Kelly,” helmet. Even the famous German spiked helmet was made of leather and not steel, to be replaced by the Stahlhelm a year into the war.

 …