Minding the gap between intention and reality

By Wendy Vaulton

Did it work? What are the results? These are the Siren song of educational evaluation. But, beware. We can easily find ourselves in dangerous territory when we dedicate all our evaluation resources toward monitoring outcomes and pay little or no attention to implementation.

Whether results are awesome, disappointing, or mixed, we can neither take credit nor resolve challenges unless we can describe what was actually put into place that may have caused those results. When schools see disappointing student outcomes, the collective instinct may be to jettison whatever new program or practice is being tested. Without having carefully defined and monitored implementation, we can’t tell if the idea was bad or we just did it poorly.

Implementation is where our big ideas meet reality, and it is sometimes rocky, continually evolving, and not linear or tidy. Strong implementation of any school improvement effort requires careful planning, leadership, articulated goals, and well-defined operational expectations, among other things. We have to address questions like: What are our specific goals? What exactly will we do to achieve our goals? How many hours of instruction, what kind of training, for whom, when? To monitor implementation, we have the additional challenges of selecting what to measure, figuring out how to measure it, and getting that information collected and thoughtfully analyzed. Assuming you don’t have a home cloning machine enabling you to be 10 places at once, how can this all be accomplished?

The answer is in small steps, with a team, and with the understanding that it will be imperfect and will evolve. Even if your first steps are wobbly, the act of trying to monitor implementation will likely strengthen your implementation. Figuring out how to measure something forces your team to “operationalize” important practices-to take them from vague to specific. It will help you foresee challenges before they arise by making you think through schedules, timelines, and realistic expectations. It also communicates what is valued and prioritized.

Strengthen implementation and monitoring by becoming specific and detailed about the practices your school is focusing on right now. Your goals may be to increase teacher knowledge of the reading process. If so, set specific targets for the activities that will promote this and select at least one way to figure out whether or not your activity is having the desired effect on teachers. How many professional development hours were actually provided? How did teachers self-assess their knowledge and skills before and after training?

As you think about developing measurement tools for implementation, you may consider issues of quality, quantity, and responsiveness. In other words, when we think of any specific practice or approach we have tried to put into place, we may ask are we doing it well, are we doing it enough, and are people engaged in it? These are all important components of implementation.

It is likely that you will need some new data collection in order to monitor implementation. Think about low-cost, low-burden tools that, with minor tweaking, could give you good information. These might include:

  • Coaching logs
  • Sign in sheets
  • Satisfaction surveys
  • Self-assessments of teacher knowledge or reaction to training
  • Weekly self-report checklists of important practices
  • Classroom walk through

For more information about critical components of Literacy Collaborative implementation, please take a look at our Fidelity of Implementation Tool (FIT). Items on the FIT can be turned into self-reported survey questions via Survey Monkey or other online survey tools.

comments powered by Disqus