2021 FDA Science Forum
A Quantitative Evaluation of COVID-19 Epidemiological Models
- Authors:
- Center:
-
Contributing OfficeCenter for Biologics Evaluation and Research
Abstract
Background
COVID-19 pandemic has taken a significant human toll.
Purpose
Quantifying how accurate epidemiological models of COVID-19 forecast the number of future cases and deaths can help frame how to incorporate mathematical models to inform public health decisions.
Methodology
Here we analyze and score the predictive ability of publicly available COVID-19 epidemiological models on the COVID-19 Forecast Hub. Our score uses the posted forecast cumulative distributions to compute the log-likelihood for held-out COVID-19 positive cases and deaths.
Results
Scores are updated continuously as new data become available, and model performance is tracked over time. We use model scores to construct ensemble models based on past performance. We found that different modeling strategies score comparably despite varying assumptions. When we look at the distribution of scores, we see that scores decrease for long-term forecasts substantially and models are in general much better (with higher score) at forecasting cumulative death counts than in forecasting weekly incidental case counts. We also noticed that the models failed to capture abrupt changes in the observed case counts, specifically first week of July 2020 and mid November 2020. We calculated the scores of the unweighted and score-weighted ensemble models at each target end date. Overall, we found that the score-weighted ensemble performed better than the unweighted ensemble model.
Conclusion
Our publicly available quantitative framework may aid in improving modeling frameworks and assist policy makers in selecting modeling paradigms to balance the delicate trade-offs between the economy and public health.
