I have long been a fan of the Kirkpatrick training evaluation scheme since I first encountered it during my education fellowship. It is a useful framework for instructional design and assessment, and for educational research – especially for research about continuing medical education (CME). The original framework relies on four basic levels, which were developed primarily for industrial and business training evaluations:
- Reaction – How did the learners enjoy the session? Were the facilities adequate? Was the instructor entertaining? Were the donuts good?
- Knowledge Gain – Did the participants learn anything? Could be short-term or long-term learning.
- Process Change – Are the learners changing their behaviors consistent with the training? Are they doing things better?
- Results – Did the training result in meaningful improvements for the organization? More widgets produced, more revenue obtained, etc.?
This framework has been modified by Barr to add some gradations in levels 2, 3 and 4. These gradations add frankly more subjective outcomes that can charitably be perceived as complementing the more objective outcomes at each level. Did the learners “feel differently” about the topic of instruction? Did the learners “self-report” a change in their behavior. (citation) These outcomes are, of course, easier to measure, and I certainly have used these modifications in curricular assessment design myself. But I think they stray a bit from the intended purpose of the framework – to push the direction of assessment and research to more objective outcomes.
A recent article by Baines, et al. in BMC Med Ed nicely discusses the use of patient (or multi-source) feedback to augment this framework. The authors described the limitations of this work as including some distrust of the objectivity and validity of this feedback. This got me thinking about the possibility of using patient reported outcome measures. These validated measures can provide a sort of “objective feedback” to clinician learners at levels 3 and 4 in a way that feels more like outcome to me, since they can describe process changes and outcomes as filtered through the patient’s lens. The federal research funding agencies and CMS are all pushing PRO measures, and I believe they would be a useful adjunct to CME evaluation, learner assessment, education research and quality improvement initiatives in addition to clinical care and research.
P.S., I’m not sure what prompted me to quote Sir Paul McCartney for the title of this post…it just sprang to mind…We should listen to what any of our patients say, not simply those that identify as men….