Evaluating an EHDI Information System: CDC's Six-Step Guide

From chapter 6 of CDC's EHDI Guidance Manual, on monitoring and evaluation.

EHDI programs use an information system, the EHDI-IS, to identify deaf and hard of hearing (D/HH) infants and children and connect them with services. This chapter shows program staff how to plan, carry out and build evaluation of that system into their routine work, whatever stage of development the system is at. It is based on CDC's Framework for Program Evaluation and the updated guidelines for evaluating public health surveillance systems published in the MMWR on July 27, 2001.

Why evaluate

Evaluation improves public health programs and makes them accountable to policy makers and partners. Built into routine work, it helps make sure that:

• the data describe each child's real screening, diagnostic and early intervention status — accurate, complete, consistent, timely, unique and valid;
• the system has the acceptability, flexibility, simplicity and stability it needs, and both users and reporters can operate it;
• each jurisdiction has a useful system for tracking children through the EHDI process and connecting D/HH children with services;
• the resources behind it are used well.

The chapter defines program evaluation as the systematic collection of information about a program's activities, characteristics and outcomes, to judge the program, improve it or inform decisions about its future.

A common misconception is that evaluation happens at the end of a project. In fact it aims to improve the system, to prove that it reaches its intended outcomes, or both — and settling the purpose at the start avoids misunderstandings about how the findings will be used.

Monitoring and evaluation

Monitoring tracks whether things are being done; evaluation asks why the program is or is not working. Monitoring feeds evaluation.

Staff can use the system as an early warning system, spotting data problems at screening, diagnosis and enrolment in early intervention as they work towards the EHDI 1-3-6 benchmarks. Each jurisdiction decides how often to check what: completeness of hospital submissions might be checked daily or weekly, duplicate records, errors and missing data monthly. Ad hoc reports from the system help build these checks.

The six steps

CDC Program Evaluation Framework.

1. Assess the context

Consider an evaluability assessment first: are the system's goals attainable through its activities, are there resources (funding, staff, data), and is there interest in an evaluation?

• Interest holders, or partners, are anyone with a stake in the evaluation or its results — hospitals, audiologists, medical homes, early intervention programs, state health departments. Involve them early and often.
• Place means the social and historical context: how the system operates, who manages it, who decides, and the power dynamics among the people around it.
• Evaluation capacity covers the funds, staff, volunteers, time, technology and data available, and which partners are willing and able to help — and what they each assume evaluation is.

2. Describe the system

Everyone should share a clear picture of the system: why it is needed, its stage of development, its intended outcomes, the activities meant to reach them, its resources, and its social and political context. A logic model summarises this in four parts:

Every outcome should state the direction of change — increase, decrease or maintain. A logic model is a living document that changes with the system.

An example of an overarching EHDI-IS logic model. CDC.

3. Focus the questions and design

The team settles the purpose, who will use the findings and how, the type of evaluation, the questions and the design. Resources never cover everything, so questions must be prioritised, with utility and feasibility in mind.

• Process evaluation asks whether the system is being implemented as planned: are reporters willing to report on time? Are hospital screening staff following protocols? What training is needed? What causes loss to follow-up?
• Outcome evaluation asks whether it achieves the desired changes: are more users active as more facilities are trained? Did shared data lead to actions that helped connect D/HH babies with services?

Questions can target the system's attributes:

For data quality, programs can choose among these dimensions, adapting them to their needs:

Data quality dimensions. CDC.

Designs can be experimental, quasi-experimental or non-experimental. Many surveillance evaluations need only a simple non-experimental design — for example, a survey of partners' satisfaction with the data, or interviews with audiologists about the reporting form. Quasi-experimental designs collect the same data at several points or use a comparison group: tracking whether more audiologists report after training, or testing knowledge before and after it.

4. Gather credible evidence

With partners, choose data collection methods and sources that users will trust, considering the quantity, quality and context of the information. Indicators — specific, observable, measurable signs of progress — tie back to the objectives, the logic model and the questions. Common EHDI indicators, such as loss to follow-up or loss to documentation for diagnosis, can show progress on outcomes. Data can come from questionnaires, observation, databases, focus groups, interviews and document review.

5. Generate and support conclusions

Link the findings to the questions and tell the program's story, with the audience in mind. A report should compare actual with intended outcomes and with previous years, and state its limitations: possible biases, and the validity, reliability and generalisability of the results. A formal report is not the only way to share findings.

6. Act on the findings

Always ask "so what?". Use the results to improve the system, strengthen what works and change what does not, and tailor how findings are shared — a report for hospitals, perhaps, a presentation for early intervention programs.

Sources

• Centers for Disease Control and Prevention, "EHDI Guidance Manual – Chapter 6: Monitoring and Evaluation." https://www.cdc.gov/hearing-loss-children/guidance-manual/chapter-6.html
• The chapter points to CDC's Planning the Evaluation of your EHDI-IS, The Six Dimensions of EHDI Data Quality Assessment and Evaluation Reporting: A Guide to Help Ensure Use of Evaluation Findings.
• Rewritten in hubnx's own words.