Which COVID-19 Data Best Tracked the Pandemic? CDC's 2023 Evaluation

This page describes an evaluation CDC carried out as the COVID-19 public health emergency was ending in May 2023. Surveillance systems have changed since.

When the U.S. COVID-19 public health emergency declaration expired on May 11, 2023, some national COVID-19 data would end or move to other sources. CDC's authority to collect certain data — including test positivity from COVID-19 electronic laboratory reporting (CELR) — would lapse, national aggregate case and death counts would stop, and hospitalization data would become the only source available at county level. To prepare, CDC evaluated how well and how quickly its surveillance indicators tracked COVID-19, using data from October 1, 2020, to March 22, 2023.

The planned changes

• Deaths: from aggregate counts to provisional data from the National Vital Statistics System (NVSS), based on death certificates.
• Test positivity: from CELR to the National Respiratory and Enteric Virus Surveillance System (NREVSS), a long-running network of more than 450 laboratories, reported by region.
• Community risk: COVID-19 Community Levels — a composite of hospital admissions, hospital bed use and cases — would be replaced by county hospital admission levels (low, medium or high), per 100,000 people.

How the comparison worked

Researchers compared 7-day average trends in each indicator using cross-correlation (to measure how many days one indicator leads or lags another), pairwise correlation, autocorrelation (a measure of signal versus noise) and a measure of geographic consistency across states. They also estimated how each indicator scaled against hospital admissions.

What they found

Timing relative to hospital admissions (April 2022–March 2023):

• NVSS death data, tracked by date of death, showed trends 13 days earlier than aggregate death counts, tracked by date of report.
• Hospital admission rates correlated well with other indicators, had a high signal-to-noise ratio and were consistent across states — making them easy to interpret and reliable.
• The replacement sources tracked the old ones closely over the whole period: NREVSS and CELR test positivity correlated at 0.79, and NVSS and aggregate death data correlated strongly too. Since April 2022, correlations were lower for deaths (0.41) and slightly lower for test positivity (0.70).
• Test positivity rose relative to hospital admissions after April 2022 as testing fell, CELR more so, possibly because of how negative results were reported. ICU use and deaths fell relative to admissions — likely reflecting less severe infections due to widespread immunity from vaccination and infection, better treatment and changing variants.
• Hospital admission levels matched Community Levels in more than 99% of county-weeks across 3,220 counties from February 2022 to March 2023. Most mismatches came during high-incidence weeks in February and March 2022, and when they differed, Community Levels were the higher.

Normalized trends in leading (A) and lagging (B) COVID-19 surveillance indicators, October 1, 2020–March 22, 2023. CDC figure.

Conclusions

• Hospital admission rates from the National Healthcare Safety Network are a suitable, timely main indicator of COVID-19 trends, available down to the health service area.
• NVSS death data improve the timeliness of tracking severity by up to 13 days, and are more complete — for race and ethnicity, for example.
• Emergency department visits (from the National Syndromic Surveillance Program, which had expanded to most jurisdictions) and test positivity give early warning, about 4 days ahead of admissions, though with more limited geographic coverage.

One weakness: the admission-level thresholds were not fine-grained enough to detect changes when incidence was low.

Limitations: the analysis couldn't separate the natural delay from infection to outcome from reporting delays, so real-time lags will be longer, especially for recent weeks; data availability would keep changing after the emergency ended; and results may vary by jurisdiction. States could use the report as a model for their own evaluations.

Sources

• Scobie HM, Panaggio M, Binder AM, et al. "Correlations and Timeliness of COVID-19 Surveillance Data Sources and Indicators ― United States, October 1, 2020–March 22, 2023." Morbidity and Mortality Weekly Report 72(19), Centers for Disease Control and Prevention.
• The report gives two slightly different values for the correlation between NVSS and aggregate death data since October 2020; this page gives neither.