We’ve defined and written about How to Write Good Evaluation Questions before. We’ve even shared some examples of evaluation questions based on the sector or content area. But there is another way to think about evaluation questions. Let’s look at how using different evaluation strategies or frameworks can help you to craft those perfect evaluation questions.
To describe how evaluation questions differ based on type of evaluation, let’s make up a hypothetical program to evaluate.
You’ve been hired to evaluate a program that has been operating for one year. The program aims to engage youth in an after-school program with the goal of keeping at-risk youth safe. They offer physical activity programs (e.g., basketball), tutoring, and life-skills programs (e.g., cooking classes), as well as lounge areas.
The purpose of formative evaluation is to assess how the program is being implemented . Key evaluation questions should focus on enablers and barriers of implementation. For example:
Awareness
Utilization
Implementation
I’ve found that looking to the field of implementation science can help to outline themes and even specific questions that might be relevant to formative evaluation.
After a year of operations, the program leaders believe it is time to assess the impact of the program. Your summative key evaluation questions should focus on outcomes:
Utilization
Outcomes
First a quick note: most of these scenarios are not mutually exclusive. RE-AIM is well suited to both formative and summative evaluation. For an overview of RE-AIM read this.
RE-AIM essentially comes with its own questions, that only need to be adapted to your specific program. The key evaluation questions are based on the 5 focus areas of RE-AIM:
The program is interested in really understanding how they can be adaptable to the current context and how they can continue to develop and grow to have a sustainable impact on the complexity presented in at-risk youth. In the last year they haven’t yet landed on a stable program – constantly adapting to new information and changing environments. They want to use Developmental Evaluation to support rapid growth and emergent innovations. Key evaluation questions should focus on emergent learning and opportunities for development:
For more great examples of DE evaluation questions, check out Developmental Evaluation Exemplars: Principles in Practice edited by Michael Quinn Patton, Kate McKegg and Nan Wehipeihana.
Again, Utilization-focused evaluation is not at all mutually exclusive from previously described scenarios. In this scenario, you determine that the primary purpose of evaluation is to give the staff something they can action (or use). In this case, the approach to development of the evaluation questions is key: engage your stakeholders often, early, and deliberately. What do they need to know? What decisions do they need to make? Key evaluation questions may include:
Most Significant Change evaluation moves away from standard indicators and measuring. It uses storytelling to evaluate the success of a program through the lens of various stakeholders. For this evaluative approach questions may be tailored to each audience:
For program staff:
For youth:
Outcome Harvesting is a participatory evaluation methodology. Instead of key evaluation questions, Outcome Harvesting focuses on Outcome Descriptions. You may ask questions of the stakeholders like:
Principles-focussed evaluation aims to assess how an applied set of principles translated into action or behaviour. Your methodologies may be very similar to other evaluation approaches, but the questions may centre on a pre-defined list of principles. Let’s say the program had a list of core values that included:
Your evaluation questions may then look like:
Again, Michael Quinn Patton has an entire guide to Principles-Focussed Evaluation that can help you develop the right evaluation questions.
