Jagpreet Chhatwal, PhD, an assistant professor at Harvard Medical School and decision scientist at MGH Institute for Technology Assessment, discusses the challenges of building an interactive COVID-19 modeling tool and its real-world impact.
Health care models allow policy makers — and, to varying degrees, all of us — to posit a range of “what-ifs” on issues that change the course of economies and lives. In late April, as confirmed coronavirus cases, illnesses and deaths spread unevenly across the United States, a team of researchers from Massachusetts General Hospital (MGH)/Harvard Medical School, Boston Medical Center and Georgia Institute of Technology launched a COVID-19 simulator designed to illuminate different social distancing scenarios. Options include a complete lockdown similar to China or Italy; current US state interventions, which vary from stay-at-home orders to partially open; and minimal restrictions, described as an open economy with people returning to work, and no interventional tactics other than newly learned health behaviors, such as frequent handwashing and avoiding close contact when sick. This interactive tool allows users to spotlight or more broadly compare national and state projections on important health trends, such as new COVID-19 cases, hospital or ICU beds needed and cumulative deaths.
Team leader Jagpreet Chhatwal, PhD, is an assistant professor at Harvard Medical School and decision scientist at MGH Institute for Technology Assessment. His previous work in health care modeling includes a hepatitis C calculator exploring treatment cost-effectiveness globally and a hepatitis C state policy simulator intended to help policy makers in each state make decisions and investments around hepatitis C elimination. He answers our questions about the challenges of building and validating the COVID-19 simulator and its real-world impact.
Interview edited and condensed for clarity
We’re learning about this disease on a weekly basis, so our projections of it evolve. They need to be changed over time. Actions we take today will determine the course of the pandemic. We know, for example, that people are protesting stay-at-home restrictions, and we hope that this tool can inform the general public about the consequences of their actions.
We built this simulator using standard methodology that has been used for infectious disease modeling for decades. These are mathematical models based on different compartments: susceptible, exposed, infectious and recovered (SEIR). Our objective is to refresh the data at least once a week. We are also incorporating mobility data tracked by people’s cell phones to understand the effects of lifting restrictions in different states on the spread of COVID-19.
There are several challenges. For one, we don’t know how many people with undiagnosed disease are in the population, so we have to rely on the reported COVID-19 cases, which depend on the testing rate in a given state. If the testing rate increases in a given week, the reported case count increases, but this may not necessarily mean that disease transmission has increased this week. These are the challenges we’re facing — and I believe every modeling team is facing — but we have to make decisions based on where we are and what we’re learning. We can’t say, “Let’s wait for ideal, robust information to come in” and then decide what actions to take. We don’t have that luxury.
With any model, some criticisms can be addressed by using the best methods you believe are relevant for the situation you’re modeling, and by using reliable data. One common criticism is that people recommend that we should be using mathematical modeling to inform the trajectory of the pandemic. But mathematical models have their own limitations. They can be very sensitive to whatever assumptions people make in the model. In this pandemic we don’t know a lot about this disease, and whatever values you put in can change the trajectory substantially. As we learn more about COVID-19 — such as the underlying prevalence and mortality rate — we must revise our models for better predictions.
People have to understand that we don’t have a crystal ball that can foretell where the pandemic is heading. Our projections are probabilistic numbers, and many of them could be wrong when we compare them to the actual number of cases. We have to keep that in mind. It’s not like a weather prediction: next week, we’ll have a rainy day. It’s like predicting the chance of rain or snow a month or two in advance. The validation we can do is looking at historical data only. We can compare next week’s projections to actual data when this becomes available: our projections versus the actual numbers. A good use of such models is to evaluate different what-if scenarios and inform decisions that are driven by data — even limited data — instead of people’s feelings.
Right now, the most important question we’re trying to answer is the case level of partial lifting of restrictions. States may define that partial lifting differently. We’ll have to look into how each state defines it, and how much traffic that will generate — how much mobility will start happening because of partial lifting. That will tell us something about how the disease transmission rate will increase or decrease over time.
New York, for example, came up with a plan for which businesses can open that is needs-based, more what people need than what they want, starting with categories like manufacturing and construction. Of course, people going to malls is very different than people resuming work at construction sites. Our model cannot really tell the potential impact of opening a specific type of business. We don’t have enough data to inform that granular-level decision. But by looking at mobility, or how mobility has changed, we can project the trajectory of the pandemic in a given state.
Our model is interactive and available online for anyone’s use. Anyone, including policymakers as well as the general public, can use this tool to evaluate and test different scenarios to get a sense of what will happen. From past experience with the World Health Organization and the Centers for Disease Control and Prevention, we’ve learned that when policymakers or stakeholders have some control over the analysis, they’re more likely to trust it.
– Francesca Coltrera
Continue the conversation on Twitter by connecting with us @HMS_ExecEd or with Dr. Chhatwal @JagChhatwal.