One goal of the Affordable Care Act of 2010 was to improve the quality of medical care. Key to this effort is the Hospital Readmissions Reduction Program (HRRP), which aims to reduce 30-day readmissions, for any reason, for patients who were previously hospitalized for heart attack, heart failure or pneumonia. The HRRP is a “pay-for-performance” measure that reduces Medicare payments to hospitals that have higher-than-expected rates of readmission. After the HRRP took effect, hospital readmissions fell substantially. Most observers credited the HRRP, concluding that the threat of financial penalties had served as a powerful incentive to improve patient care.

But was it really such a success? J. Michael McWilliams, MD, PhD, is the Warren Alpert Foundation Professor of Health Care Policy at the Blavatnik Institute at Harvard Medical School (HMS), a Professor of Medicine at HMS, and a general internist at Brigham and Women’s Hospital in Boston. He answers our questions about his research on the HRRP and discusses drawbacks of pay-for-performance programs.

Interview edited and condensed for clarity

You recently published a paper in Health Affairs entitled “Did Hospital Readmissions Fall Because Per Capita Admissions Rates Fell?” Tell us about the question you were trying to answer in this study.

Because the decrease in hospital readmissions coincided with implementation of the HRRP, many policy makers and researchers concluded that the HRRP caused the reduction. But something else was happening that went largely unnoticed: per capita hospital admissions also were falling at the same time.

We don’t know exactly why, but there are many potential reasons. The number of hospital beds has been declining for years. We’ve gotten better at treating conditions on an outpatient basis. And in 2010 the national Recovery Audit Program was implemented to crack down on unnecessary short stays, to prevent hospitals from admitting patients for just a few hours to collect full payment. Whatever the reason, we know that the decline in hospital admissions was not related to the HRRP because it occurred across all payers, not just Medicare.

In our study, we wanted to understand how the fall in hospital admissions may have affected readmission rates.

What were your findings, and what are the implications for using 30-day readmission rates as a quality metric associated with financial penalties?

We found that the decline in hospital admissions that occurred between 2009 and 2014 was responsible for much of the decline in 30-day readmissions during that period. That makes sense, because as a matter of simple statistics we should expect the readmission rate to fall if there are fewer admissions per patient. If there are fewer darts on the dart board, the probability of throwing the next dart close to another is lower.

The implication of our study and other research is that the effect of the HRRP is probably much smaller than previously thought. Unfortunately, we don’t know the precise impact of the HRRP because it wasn’t rolled out in a way that allows for strong study design. The HRRP has also had unintended consequences, such as shifting resources from hospitals serving sicker and poorer patients to hospitals serving the healthier and wealthier.

Putting those two strands of evidence together, it appears that the HRRP offers a cautionary tale about coupling performance on quality measures with financial incentives. Most studies of other pay-for-performance programs have also found pretty lackluster results.

I think it’s important to think about the problem of using a utilization measure, like readmissions, to measure quality. Research has shown that many readmissions are not due to problems in care in the preceding admission. A patient who is hospitalized for pneumonia, for example, might be readmitted within 30 days for a gastrointestinal problem. In fact, I’ve often thought the word “readmission” is somewhat misleading because it implies there is always a connection, but these hospitalizations are often independent events.

Many people think that pay-for-performance will work if we just come up with better measures and attach stronger incentives. But I tend to think the problems are more intractable. Rather than double down on pay-for-performance, I think we are better off taking a step back and thinking more broadly about all the potential ways we can improve care delivery without relying so heavily on measures and payment incentives. Already, we get a lot of value-based care without value-based payment. That’s because patients demand high-quality care and providers are intrinsically motivated to provide it. I think we can tap into those resources more effectively to address the many deficits we still face.

What is your opinion about the role of randomized controlled trials or natural experiments to measure the impact of new policy measures before rolling them out widely?

I think this is really a must. Because if we continue to roll out these programs without thinking about how they can be evaluated, then we may learn very little about what works and what doesn’t work.

There is now precedent for rolling out a program in a different way. The Comprehensive Care for Joint Replacement (CJR) model is a bundled-payment approach that is being randomized across geographic areas for five years. This allows for stronger evaluations to determine if the program is truly improving care.

—Ann MacDonald

Continue the conversation with us @HMS_ExecEd or with Dr. McWilliams @JMichaelMcW.