Finding the "Critical Moments": Using Reflection in Evaluation

The great Paul Howard of the Institute for Healthcare Improvement (IHI) and I wrote this a few months ago to talk about our use of the Critical Moments Methodology for the American Evaluation Association's Community Psychology week. In the hustle and bustle, I neglected to cross post here. So enjoy!

---

Community psychology puts a strong emphasis on participation and inclusion in all stages of a project; from design to implementation to evaluation. Naturally, those who identify as community psychologist gravitate to participatory and Empowerment Evaluation methods.

Spreading Community Accelerators for Learning and Evaluation (SCALE), a Robert Wood Johnson Foundation-funded, IHI-convened initiative, aimed to help healthy community coalitions develop readiness to transform how they approached health improvement projects. There was A LOT of data, and it was sometimes hard to know how to make sense of it. Because SCALE was a formative evaluation, we needed a process to delve deeply into emerging results on an ongoing basis.

To do so, we had periodic “Critical Moment” Sessions. The Critical Moment method uses a structured narrative approach in which the evaluator poses questions to elicit participant emotions, reactions, and beliefs on topics of interest. Reflective activities that involve stakeholders can be used contextualize the ongoing narrative of a project and ascribe meaning to experiences (e.g., What has been the most memorable moment or experience on this project? What about it made the moment so memorable? How did the moment affect you?)
​
We conducted these in group settings to allow all relevant stakeholders (implementation team, funder, and community representatives) the opportunity to process together. First, we sent out an “inquiry” question to participants ahead of time, which we then synthesized and grouped all responses. We shared this grouping during the session (see visual below; yup, it is crowded, there were many comments!)

After allowing everyone to voice what they thought were the key moments, we held a voting process to focus discussion around 2-3 moments to explore in more depth.

This general process was very successful, so much so that we conducted over ten critical sessions over the course of the 4-year project. Further, many community participants adopted these methods in their settings. The process tapped into the spirit of community psychology, putting the meaning-making into the hands of the people tasks with making program changes.
​
Check out this online paper from MIT understand and dive into this powerful tool. http://web.mit.edu/cil/web_scripts/www/Critical%20Moments%20Methodology%20Brief%20CoLab.pdf