How Should the Government Be Evaluating Agencies and Programs?

Performance Measurement, Program Evaluation, Or Both; plus the importance of data in evaluation.

The government is pushing evaluation, especially President Obama and the OMB. However, not many people have a clear understanding of the difference between performance measurement and program evaluation. Below, I outline the differences and discuss the importance of data.

Performance Measurement: Performance measurement (PM) is the term used to describe activities for recording, collecting, and analyzing data concerning a particular program’s resource inputs, as well as its outputs and results. PM is a management tool providing feedback on key metrics on what actions are taken, what resources are used, and what results are realized. Typically, PM harnesses readily available program specific data to track over time performance. Good performance measures should focus on appropriate information about either program procedures or preferred program outcomes. PM does not entail in-depth examination of the efficacy of a program, nor does it attempt sophisticated analysis of cause and effect relationships within a program. Think scorecard: how much was done in the program to support the organization’s mission priorities?

Program Evaluation: Program evaluation (PE) seeks to generate reasonably reliable and valid information about the impacts and merits of a program. PE attempts to provide more in-depth knowledge about a program’s actions and outcomes that result. That is, it explores evidence of whether or not a program performs as intended. In PE, a systematic approach for collecting, analyzing, and using data is taken to ascertain if and how a program is performing. Questions addressed in PE go beyond the managerial-administrative metric information compiled in PM. Evaluations explore overriding policy concerns of whether a program works and should resources continue to be dedicated to it. Think big picture: are a program’s outcomes originally intended to strengthen the organization’s mission priorities?

Complimentary Processes: Performance measurements and program evaluation studies all generate information that can be used to improve a program’s delivery of services and ensure that a program is achieving the organization’s overall mission. PM is an important contributor to PE. Performance data on program inputs, processes, activities, and outputs are helpful in answering the “does a program achieve its intended goals” policy question addressed by PE.

Importance of Data, Measurement, and Evidence: Efforts to integrate data and evidence into decision-making may be seen at all levels of government and public service organizations in the U.S. There is recognition by policy makers and administrators alike that some data (and analysis) are needed to identify effective or ineffective performance and potential misalignments with mission priorities. Data are a valuable resource and can be gathered from many sources, such as agency administrative records, follow-up surveys/contacts, and other specially designed survey instruments. To be useful, data must be both valid and reliable. Validity concerns the extent to which data yield a fair, unbiased indicator of performance. Reliability concerns how objective, precise, and dependable the data are in measuring performance. For analysis to be meaningful, it must be founded on “good” data.

What do you think?

Should the government use performance measurement, program evaluation, or both to evaluate agencies and programs?

What evaluation methods does your organization or programs utilize?