Solving the Wrong Problem

One of my favorite places to visit in New Orleans and to take friends and family is the World War II Museum (also known as the D-Day Museum). During a recent visit, I learned about efforts taken to protect British and American aircraft from enemy bombardment during World War II. Deadly accurate flak tore through the planes of the Allied bombers. And when things went wrong on an airplane that received damage from antiaircraft fire, the options at 30,000 feet were limited—so limited, in fact, that injured crew members often were thrown out of the plane in the hope that the enemy would find them, patch them up, and send them to a POW camp.

When Air Force engineers realized they were losing hundreds of aircraft, they decided to use better armor to protect their planes to increase their survival. They studied the returning aircraft, and experts decided to place reinforced armor on the planes. The question was, where to put the armor? As the planes returned from their missions, the engineers counted up all the bullet holes on various areas of the planes. The planes showed similar concentrations of damage in three areas: the fuselage, the outer wings, and the tail. The bomber command came up with a solution: bombers should be more heavily armed, to reduce the damage brought by flak and enemy fighter planes. But, of course, you can’t use heavy armor on the whole airplane, the way you can with a tank, because the weight will make it so heavy that it won’t take off. So, the challenge was where to put the additional armor. The obvious answer was to look at planes that returned from missions, count up all the bullet holes in various places on the returning planes, and then put extra armor in the areas that sustained the most fire (Figure 1).

Figure 1. Diagram showing the battle damage to the aircraft demonstrating the areas that sustained the greatest fire during a bombing raid. (From Survivorship Bias. Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Survivorship-bias.png . Accessed November 29, 2021.)

After analyzing where its planes had suffered the most damage, the engineers determined they needed to reinforce the planes’ wingtips, central body, and elevators, because that is where the bullet holes were most frequently seen on returning airplanes. The Air Force engineers thought they had discovered where their planes were suffering the most damage. But—what they actually had discovered was where the planes could take damage without being destroyed and return safely to the airfields in England. The engineers weren’t looking at the whole problem.

As Hungarian-born mathematician Abraham Wald explained, if a plane makes it back safely to the airfields in England, even with multiple bullet holes in its wings, that means that the bullet holes in the wings aren’t very dangerous to the safety of the plane. The real solution was to place the armor on the areas that, on average, don’t have any bullet holes, such as the cockpit, where the pilot sits, and the plane’s engines—because planes with bullet holes in those vulnerable places never made it back to the airfield. That’s why you don’t see any bullet holes there on the ones that return.

Wald said that the military would make a terrible mistake by upgrading the armor along with the wings and the tail of the planes. Why? Because the military was only looking at the damage on returned planes. They hadn’t factored in damage on planes that didn’t return. Planes that didn’t return were the ones that sustained damage in ways not seen on returned planes—that is, they took enemy anti-aircraft fire on their engines or on the cockpit where the pilot was sitting. Unlike the body, tail, and wings, the engine and the cockpit were extremely vulnerable. Planes hit in those vulnerable locations crashed and, therefore, did not return to the Air Force base to have their damage evaluated. Wald explained that if a plane made it back safely with, say, bullet holes in the fuselage, it meant those bullet holes weren’t very dangerous to the survival of the plane and the crew. Armor was needed on the sections that, on average, had few bullet holes—because planes with bullet holes in those parts never made it back. That’s why you didn’t see bullet holes on sensitive parts of the planes that did return.

A clear understanding of survivor bias leads to improved decision-making, building a more cohesive team, and making data-based decisions that identify the real source of the problem.

As a result of this reasoning, armor was added to protect the engines and the cockpit of the plane. This application of additional armor saved many lives and planes, and probably turned the tide of the war.

What does this have to do with healthcare?

This phenomenon of excluding the aircraft that had crashed and never made it back is called survivorship bias. Survivorship bias explains why people often believe that cars that were made 50 years ago last longer than those made today—even though these ideas are not based on facts and are invariably wrong. Survivorship bias clouds your judgment and distracts you from getting to the root cause of a problem within your practice. It makes it easy to conflate correlation with causation.

Understanding survivorship bias and how it can cloud your judgment is the key to solving problems and avoiding trial and error or the Band-aid approach, which manages the symptom but does not identify the cause and, as a result, lends itself to repeating the defective process. A clear understanding of survivor bias leads to improved decision-making, building a more cohesive team, and making data-based decisions that identify the real source of the problem. It leads to a better, or perhaps a permanent, solution.

To apply this knowledge to our medical lives—patients usually go to the doctor when an illness has become so symptomatic that it is affecting the patient’s quality of life. Sometimes, by the time the patient has requested medical care, the illness is at its worst, and the patient is just about to recover and has nowhere to go but up. Sure enough, after patients see the doctor they improve. But it’s not always because of something the doctor did. Sometimes it’s a matter of when patients choose to see the doctor and the natural history of the medical condition.

The same error in perception also can occur in the medical practice management world. An office manager or managing partner might hear about a marketing ploy that worked well for a competitor and decide to implement something similar in their practice. But for some reason, their patients aren’t receptive. Why is that? It’s because those studying the issue only heard about their competitor’s success stories. They didn’t hear about the backlash from users who hate receiving unsolicited notifications or using a phone tree to access the practice. To avoid making this kind of mistake, look at the marketing and the practice promotion strategies from all sides and make decisions based on the root cause of the problem.(1) Root cause analysis has become the main way medicine investigates mistakes and tries to prevent future mistakes.

Another example of survivorship bias is seen in patients who are diagnosed with cancer. Survival rates are based on the number of months or years post-diagnosis that patients survive. However, the patients with the best prognosis are the ones with the lowest risk in terms of age, gender, and whether the tumor is less or more aggressive. If more of the patients have aggressive disease, the survival rates are disproportionately represented, favoring those with less aggressive disease.

What isn’t taken into account is the number of patients who die shortly after they’ve been diagnosed. Because they die, they aren’t included in survival rate calculations, thereby inflating the predicted survival outcome of those with less aggressive disease (Figure 2).

Figure 2. Survival rates for mesothelioma. American Cancer Society, https://www.cancer.org/cancer/malignant-mesothelioma/detection-diagnosis-staging/survival-statistics.html

Another example is seen when patients complain about having to wait to be taken to an exam room to be seen by the doctor, PA, NP, or medical assistant. This is a problem that most practices experience when there is poor time management. Excessive waiting impacts efficiency, productivity, and patient satisfaction scores. One solution is to keep the patients involved and active in participating in their care so that they do not feel that they are just sitting and waiting to see the doctor. For example, a successful ophthalmology practice found a solution to this problem by having a series of way stations, each with two soft chairs. If a long wait is anticipated, the patient sits in one of the soft chairs for five minutes, then sits in another chair at a distance from the first chair. At each “station,” a task is completed with or by the patient, such as signing a privacy policy form, completing a health questionnaire, undergoing an eye exam with and without glasses, or having their eyes dilated. This “keep them moving” concept can take 20 minutes, which usually is an adequate buffer. The practice creates the illusion that the patient is not really waiting. The managing partner said he has not received any negative comments that the practice is stalling.

Another option that helps patients to arrive on time is give out appointments ending in the numeral five instead of zero. An observation is that patients tend to be late when given appointments at 9:00, 9:30, or 10:00 and are more likely to be on time if the appointment time is 10:05, 10:15, 10:25, 10:35, 10:45, or 10:55.

Bottom Line: Why is it important to identify the right problem? I suggest you think of the big picture when trying to select which changes you are going to make to solve problems in the practice. Rather than focusing only on ideas that appear to be the low-hanging fruit and an easy solution to the problem, take a closer look to make sure you have identified the real problem. This will give additional information into how well your practice is performing over time and provide ideas on how to improve your service.

Reference

1. Wu AW, Lipshutz AK, Pronovost PJ. Effectiveness and efficiency of root cause analysis in medicine. JAMA. 2008;299:685-687.