Nursing the Numbers

by Howard Fienberg and Iain Murray
August 3, 2000

When we send loved ones to the hospital we are putting our trust in the doctors to heal them and return them to us safe and sound. We accept that this may not always happen and the doctors may not be able to save them. But when our loved ones die as a direct result of doctor error, that is one of the most shocking outcomes of all; our trust has been betrayed and our confidence in hospitals shattered. Unsurprisingly, therefore, there was an outcry when an Institute of Medicine (IOM) report in 1999 estimated that as many as 98,000 Americans die each year as a result of medical errors. But now a new study in the Journal of the American Medical Association (JAMA) says that there may be as few as 5,000 such deaths annually. That sounds like good news, but is this new study accurate? Do we really know enough about deaths from medical errors to know what to do about them?

The IOM's estimates that between 44,000 and 98,000 hospital patients die each year because of medical mistakes (more than breast cancer, AIDS or highway accidents) were little more than "guesstimates," derived from hospital discharge data from a study in New York and from one in Colorado and Utah. There were several problems with the figures: it is not clear the three states concerned were representative of the country as a whole, and the classification of a doctor error was very broad (if a patient didn't tell the doctor about a medication which then caused complications, that was classified as a doctor error). Also, the studies were observational in nature and not designed to investigate causal relationships between errors and deaths. But perhaps the best indicator of the IOM figures' validity is that, at the upper level, they would mean that 3 out of every 4 hospital doctors see one patient a year die as a result of medical error.

There are therefore enough question marks over the IOM figures to make the new JAMA figures, from a study by Dr. Rodney A. Hayward, attractive. But they have their own problems.

Dr. Hayward reviewed cases of deaths involving medical errors in order to determine if "optimal" medical care could have prevented what might have been an inevitable death. The study estimated that, under conditions of optimal care, only 6 percent of the patients subjected to medical errors would have left the hospital alive and only 0.5 percent would have lived three months or more "in good cognitive health." Hence, the estimate of only 5,000-15,000 deaths.

But the study utilized medical records from Veterans Affairs hospitals, when community hospitals would have been more representative of the medical profession as a whole. Additionally, Dr. Lucian L. Leape, one of the authors of the IOM study, criticized Dr. Hayward for using much smaller samples. He also accused the study as using "statistical torturing" to obtain its results. But, while it is true that the study involved a lot of statistical manipulation, all the methods employed are statistically sound and commonly used in assessing risk.

In addition, the IOM and JAMA studies have some flaws in common, since the measurement of medical error is as uncertain as the counting. Modern medicine is a complex affair. Although evidence of error is sometimes obvious, associations can be indirect, difficult to make, and open to debate. It is not an exact science. The assessment of blame appears too subjective to allow an accurate calculation of total cases. Indeed, Dr. Hayward discovered in his JAMA study that doctors rarely agree on whether or not an error led to a patient's death.

All in all, you've got to wonder about the validity of studies whose margins of error are more than 100 percent. Indeed, if we simply followed the numbers over time in the different studies, it would appear that medical error incidence has massively decreased. But the truth is we just don't know. Certainly none of the numbers on medical errors is reliable enough on which to make public policy decisions.

For instance, the IOM report originally called for a 50 percent reduction in medical errors. But with no reliable figures on error incidence, how can anyone measure their success? In a commentary in the New England Journal of Medicine (Apr. 13, 2000), Dr. Troyen A. Brennan, author of the 1984 study used by the IOM, lamented that "the effort to prevent errors may deteriorate into a marketing ploy, especially after the relatively easy solutions, such as ensuring that prescriptions are legible, have been undertaken."

Nevertheless, Dr. Hayward believes the attention paid to the IOM study has been beneficial and has improved the quality of medical care. At the same time, he worries that such attention can become sensationalist and eventually scare off those who most need medical treatment. There is definitely a problem with medical errors, but we do not yet know enough about its scale to decide what we need to do about it. An informed public debate about the issue is a good thing, but simply throwing out scary numbers and calling for action is not. There is no reason yet for us not to trust our loved ones to the care of hospitals. Until we have better numbers, policy makers run the risk of committing a malpractice themselves.

HOWARD FIENBERG is Research Analyst and IAIN MURRAY is Senior Analyst at the Statistical Assessment Service, a nonprofit nonpartisan think tank in Washington, D.C.

return to Howard Fienberg's page