CDC's Program Evaluation Framework, 2024

Program evaluation is the systematic collection and analysis of data about programs, policies and organizations to judge how effective and efficient they are. Unlike research, which aims at generalizable knowledge, evaluation aims to improve programs continuously and inform decisions: what is working, why a program is or is not being carried out as planned, whether its resources and assumptions hold up, and what effects, intended or not, it has. The Foundations for Evidence-Based Policymaking Act of 2018 (the Evidence Act) made evaluation a critical function of federal agencies, and evaluation works best when it is adequately resourced and built into a program's whole life, from design to conclusion.

Since 1999, CDC's evaluation work has been guided by its Framework for Program Evaluation in Public Health, a practical, nonprescriptive tool with six steps and four standards. It has been cited in about 300 peer-reviewed articles, used in about 50 countries on six continents, and applied beyond public health in clinical research, education and the military. In 2024 CDC published an update, aiming to keep the framework practical and simple, refresh rather than replace it, and bring it in line with 25 years of advances in evaluation and public health, practical experience, and current federal policy.

How it was updated

An 80-person CDC work group, guided by a 21-member steering committee, collected input from March 2022 to February 2023 from about 850 people and groups who had used the framework: a survey of federal staff, listening sessions with CDC-funded recipients and with evaluators working with American Indian and Alaska Native communities, interviews with evaluation leaders from seven federal agencies, responses to a public request for information, and sessions with work group members and at a CDC Evaluation Day. A review of evaluation and public health literature from 2013 to 2023 screened 3,436 publications and analyzed 290 in full.

Key terms

• Program: any set of related activities meant to achieve an intended outcome, deliberately broad enough to cover interventions, surveillance systems, policies, outbreak investigations, emergency responses, laboratory work, communication campaigns, training, community efforts, research initiatives and whole systems.
• Interest holder: anyone with a stake in the evaluation: people served or affected by the program, those who plan or run it, those who might use the findings, and skeptics. The term replaces "stakeholder," which can imply a power imbalance and carries a violent connotation for some American Indian and Alaska Native tribes and members.
• Health equity: a state in which everyone has a fair and just opportunity to attain their highest level of health, requiring ongoing efforts to address historical and current injustices, overcome economic, social and other barriers, and eliminate preventable disparities.

Common types of evaluation include formative (is an approach feasible, appropriate and acceptable before full rollout?), process or implementation (is it being carried out as its theory of change intends?), outcome (has it achieved its intended results? this cannot establish cause), impact (comparing outcomes with and without the program, usually to establish cause) and economic evaluations (effects relative to costs).

What changed

The framework still has six steps and a set of quality standards. Some step names and content have changed, the first step is new, and the standards are now the five federal evaluation standards. The biggest change is the addition of three cross-cutting actions that apply to every step.

Five standards

These standards, part of the Evidence Act's implementation, are deliberately broad and sometimes have to be balanced against one another, but flexibility does not mean ignoring them.

• Relevance and utility: address questions that matter to interest holders, with actionable findings delivered on time in understandable, culturally responsive form.
• Rigor: produce findings people can rely on, with limitations explained, using the most appropriate design and methods for the questions, run by qualified evaluators, within the constraints of goals, scale, time, feasibility and resources.
• Independence and objectivity: insulate evaluations from political and other undue influence and avoid conflicts of interest and bias; evaluators should regularly examine their own potential biases.
• Transparency: document the purpose, access, design, methods and release plans before starting, and report findings fully and promptly so others can review and reproduce the work.
• Ethics: protect participants' dignity, rights, safety and privacy, and be equitable, fair and just, accounting for cultural and contextual factors.

Three cross-cutting actions

1. Engage collaboratively. Share ownership of the evaluation with interest holders from planning through interpretation, drawing on both lived and professional experience, and build an environment of trust where all views are respected.
2. Advance equity. Examine the drivers of health inequities, conduct evaluation in a culturally responsive way that respects people's languages, values, customs and ways of knowing, consult communities about how their stories are shared and give something back for their contributions, and have evaluators reflect on how their own backgrounds shape what they see and do.
3. Learn from and use insights. Look for learning throughout the evaluation, not only at the end. Evaluation can shift how staff think, clarify shared goals and build an organization's culture of questioning what works, for whom and under what conditions. To make findings useful: engage interest holders, plan for use from the start, clarify who will use the findings and how, stay flexible, and share results in time to inform decisions.

Six steps

1. Assess context. Judge readiness for evaluation; identify interest holders (people served or affected, those running the program, likely users of the findings and skeptics); understand place, including the program's history, purpose, setting and environment; and gauge organizational and individual capacity to do and use evaluation. This step is new.
2. Describe the program. Write a narrative covering the need, inputs, activities, expected short-, intermediate- and long-term outcomes, outside factors and stage of development, and draw a logic model or program roadmap linking activities to intended effects. Treat both as living documents.
3. Focus the evaluation questions and design. State the purpose, choose the type of evaluation, identify the intended users and uses, write broad, open-ended evaluation questions, and pick a design: experimental (random assignment), quasi-experimental (comparisons without random assignment) or observational (such as case studies, useful for questions that are not about cause).
4. Gather credible evidence. Agree with interest holders on what evidence will answer the questions and what would count as success; choose data collection methods; define indicators for inputs, activities and outcomes; use new or existing data sources; collect enough good-quality data without excess; and plan for timing, infrastructure, storage and cultural context.
5. Generate and support conclusions. Analyze the data with a plan set in advance, interpret results against the expectations set in Step 4 with interest holders, make recommendations that address improvements and build on strengths, and state strengths and limitations openly.
6. Act on findings. Plan for use from the earliest steps, prepare findings for the audiences who will act on them, and help turn insights into action, sharing information whenever opportunities arise.

Three misconceptions

• "Evaluation is too costly and slow." Cost depends on the questions and the precision needed; a simple, low-cost evaluation can be valuable. Starting early and timing evaluation to feed decisions makes it part of the program rather than an afterthought.
• "Evaluation is too technical." Most public health evaluations do not need controlled settings or elaborate analysis; practical, context-sensitive methods that fairly summarize quantitative and qualitative information usually suffice.
• "Evaluation is punitive." The framework aims to be helpful and welcoming. Any penalty should come not from negative findings but from failing to learn from them.

The framework is intentionally general and can be combined with other evaluation approaches, tools and methods. The authors see it as one of several tools for improving programs, and as a way to build a culture in which organizations routinely ask why things are happening and how to keep improving.

Sources

• Kidder DP, Fierro LA, Luna E, et al. CDC Program Evaluation Framework, 2024. MMWR Recomm Rep 73(6). https://www.cdc.gov/mmwr/volumes/73/rr/rr7306a1.htm
• Methods for the update: https://stacks.cdc.gov/view/cdc/160381
• The report was written by a CDC work group with outside contributors; its figures are not reproduced here.