National Comprehensive Cancer Network

Interactive CME

Educational Events & Programs

NCCN Outcomes Assessment Methodologies

NCCN utilizes a number of methodologies to assess the outcomes of our educational activities. The type of methodology we use will depend on the educational format of a continuing medical education (CME) activity and the level of outcome that we are aiming to assess.

NCCN adopted the model described by Donald Moore, Jr., PhD, Joseph Green, PhD, and Harry Gallis, MD[i],[ii] in developing our outcomes-based evaluations. We work closely with our partners to determine which techniques will yield the best results.

The Moore, Green, and Gallis model describes 7 outcome levels as follows:

  • Level 1           Participation
  • Level 2           Satisfaction
  • Level 3A         Learning: Declarative Knowledge (Knows)
  • Level 3B         Learning: Procedural Knowledge (Knows How)
  • Level 4            Learning: Competence (Shows How)
  • Level 5            Performance (Does)
  • Level 6            Patient Health
  • Level 7            Community Health

All of our activities are assessed for Levels 1, 2 and 3 by our registration data (Level 1) and our standard activity evaluation which asks participants to rate their level of satisfaction with the activity (Level 2) and the degree to which they believe the learning objectives were met (Level 3A).

Measurement of objectives achieved

Activity participants are tested based on the behavioral learning objectives established for a CME activity. For example, one of the objectives might be – “At the conclusion of this activity, participants will be able to Identify opportunities for improved clinician-patient-family communication when treating patients with cancer.” As part of the evaluation form, participants would be asked to actually list if the activity helped to reach this objective. In the absence of a pre-test, there is no guarantee that the learning occurred as a result of the activity. Yet, this process demonstrates whether or not the objectives were achieved – important information for the faculty and CME staff. This method can be employed with live meetings, enduring, and Internet-based CME activities.

NCCN currently uses the following types of outcomes assessments to measure Level 3B, Level 4, and Level 5 outcomes, i.e. procedural learning, competence, and performance based changes.

Pre- and post-tests

Activity participants complete multiple-choice questions concerning activity content before and immediately after a CME activity. This method measures learning that occurred as a result of the activity. The benefit of this type of measurement is that the participants, the faculty and the CME staff have immediate feedback regarding what learning has occurred (Level 3B measurement). This method may not necessarily predict retention of the learning or change in performance. Pre- and post-tests can be used in conjunction with live meetings, printed enduring materials, and Internet-based CME activities.

Commitment to Change

Participants of live and enduring material activities are asked to write one to three changes that they plan to make a change as a result of our activities (Level 4 measurement). Jocelyn Lockyer and her associates have found that a commitment to change (CTC) predicts actual change in practice [iii]. According to Lockyer, et. al., “Three quarters of CTCs were fully or partially implemented” in her study (p. 76). A summary of these reveals the immediate impact of the CME activity, providing useful needs assessment data for planning future activities.

Post activity surveys

Post activity surveys go further in measuring change by venturing into performance based change – the Level 5 outcome. Participants are asked, at the conclusion of a CME activity, to list three changes that they intend to make as a result of the activity.

Within one to three months of the CME activity, the NCCN staff will fax or e-mail our CME activity participants and ask them if they have fully implemented, partially implemented or were unable to implement the changes they intended to make.

The limitation of this data is that it is self-reported. However, in the absence of actual observation of a physician’s performance in practice, this information serves as a surrogate marker that, according to Lockyer’s research, is indicative of actual change.

Case based assessment

In a comparison of chart audits, standardized patients (where actors take on the role of patients and physicians are evaluated on their interactions with the “patients”), and case vignettes, case vignettes were found to be as effective as the other two methods in determining outcomes[iv]. Aimed at measuring Level 5 outcomes, we have asked physicians in live meetings to answer key multiple choice questions in response to a case presentation. The cases and questions are presented before and after the CME activity to measure learning. Case vignettes can also be administered to a control group, i.e. a group of physicians who share a professional profile with the activity participants but who did not participate in the activity.

Performance Improvement Initiatives

Following the AMA’s guidelines on awarding AMA PRA Category 1 Credit™ for performance improvement (PI), NCCN CME staff will work with physicians to undertake customized practice-based performance improvement initiatives, facilitating the measurement of change through chart reviews (Level 6 outcomes). In addition, NCCN will work on system level performance improvement projects at our member institutions (Level 7 outcomes). 

Performance Improvement Initiatives – Virtual

Through a unique design, NCCN develops Virtual Communities of Practice working with physicians and allied health practitioners from varied locations to implement practice improvement initiatives. Activities are supported through a custom-designed website, e-mail, and telephone interviews.

Summary

NCCN will continue to seek new ways to obtain outcomes assessment data for our CME activities. In keeping with the charge set forth by Moore, Green, and Gallis, we will strive, in particular, to incorporate formative assessment processes into our activities that provide participants with opportunities for practice and feedback.


[i] Barnes, B, Davis, D, Fox, R. Continuing Professional Development of Physicians: From Research to Practice. Chicago. AMA Press,  2003: 249 – 274.

[ii] Moore, D, Green, J., Gallis, H. Achieving Desired Results and Improved Outcomes: Integrating Planning and Assessment Throughout Learning Activities. Journal of Continuing Education in the Health Professions. Winter 2009; 29: 1 – 15.

[iii] Lockyer, J, Fidler, H, Hogan, D, Pereles, L, Wright, B, Lebus, C, Gerritsen, C. Assessing Outcomes Through Congruence of Course Objectives in Reflective Work. JCHEP 2005; 25: 76-86.

[iv] Peabody, J, Luck, J, Glassman, P, Dresselhaus, T, Lee, M. Comparison of Vignettes, Standardized Patients, and Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring Quality. JAMA 2000; 283: 1715 – 1722.