A CE must be able to

1. Apply principles of assessment in developing a system of assessment for a program

2. Select assessment instruments that are relevant and appropriate for the learning context

3. Describe the challenges of self-assessment

• Official Royal College Requirements Document for the Assessment Unit

• Primary Review Form (To be filled by Unit Supervisor)

• McMaster Task List for Assessment Unit

• McMaster Reading List for Assessment Unit

By the end of this Unit, the CE will be able to: 

1.  Demonstrate comprehension of the principles of assessment, by

• recognizing the multiple purposes of assessment in a system of education

• analyzing how a system of assessment fits within the cycle of curricular design

• describing the differences between summative and formative assessment 

•  applying the concept of “utility” to individual assessment instruments and methods 

• identifying and describing the professional responsibilities of educators to ensure that the assessment system meets the needs of trainees and the public

• aligning types of assessment with the corresponding levels of Miller’s pyramid

• comparing and contrasting the terms “assessment” and “evaluation”

• justifying the importance of both self-directed assessment and reflection in the development and maintenance of competence of a clinician

• describing the enablers of and barriers to self-assessment

2.  Conduct a survey of important assessment instruments, by

• comparing and contrasting the following methods or instruments of assessment, with an emphasis on their structure, the types of objectives they can assess, the context for which they are best suited (e.g., clinical setting, classroom environment, etc.), and their advantages and disadvantages:

  • written exams (constructed response and selective response)

  • oral examinations

  • OSCEs and their derivatives (e.g., OSATs)

  • ITERs and other rating-scale–based expert assessment forms

  • instruments for direct observation, such as Daily Encounter Cards, CEX

  • Multi-Source Feedback

  • chart audits and practice assessments

  • portfolios

  • other

• selecting and implementing appropriate assessment instruments within a curriculum

3.  Demonstrate comprehension of the principles of psychometrics, by

• describing modern validity theory and the concept of reliability and applying these to the evaluation of assessment strategies

• applying the concept of utility and analyzing the implications of the assessment system in the context of the candidate’s jurisdiction

• comparing and contrasting classical test theory and generalizability theory

• defining principles of standard setting and selecting appropriate methods for a given assessment instrument and context

• applying the principles of qualitative research in the context of judging the quality of a portfolio as a composite assessment method (using principles of credibility, consistency, triangulation, transferability, etc.) 

4. Demonstrating competence in remediation, by 

• applying a remediation framework to an encounter the candidate has had with a learner in difficulty, while emphasizing a step-wise approach

• Please refer to the McMaster Task List for Assessment Unit for a complete listing of the summative and formative assessments required for this unit. 

• Primary Review Form (To be filled by Unit Supervisor)

Formative assessment

• Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81–112.

• van de Ridder JM, Stokking KM, McGaghie WC, ten Cate OT. What is feedback in clinical education? Med Educ. 2008;42(2):189–97.

Introductory psychometrics

• Downing SM, Yudkowsky R. Assessment in the health professions education. New York: Routledge; 2009.

• Govaerts MJB, van de Wiel MWJ, Schuwirth LWT, van der Vleuten CPM, Muijtjens AMM. Workplace-based assessment: raters’ performance theories and constructs. Adv Health Sci Educ Theory Pract. 2012 May 17.

• Kane MT. An argument-based approach to validity. Psychol Bull. 1992;112(3):527–35.

• Messick S. Validity. In: Linn Rl, editor. Educational Measurement. 3rd ed. American Council on Education Series on Higher Education. Phoenix (AZ): Oryx Press; 1989.

• Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011;33(10):783–97.

Principles of assessment

• Harden RM. Ten questions to ask when planning a course or curriculum. Med Educ. 1986;20(4):356–65.

• Holmboe ES, Hawkins RE. Practical guide to the evaluation of clinical competence. Philadelphia (PA): Mosby; 2008.

• Miller GE. The assessment of clinical skills/competence/performance. Acad Med.  1990;65(9):S62–67.

• Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–14.

• Schuwirth LW, van der Vleuten CP. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–85.

Remediation

• Hauer KE, Ciccone A, Henzel TR, Katsufrakis P, Miller SH, Norcross WA, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature. Acad Med. 2009;84(12):1822–32.

• Zamost S, Griffen D, Ansari A. Exclusive: Doctors cheated on exams. CNN Health. Available at: www.cnn.com/2012/01/13/health/prescription-for-cheating/index.html 

Self‐assessment

• Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094–1102.

• Eva KW, Regehr G. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2005;80(10 Suppl):S46–54.

• Sargeant J, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212–20.

Standard setting

• Turnbull J. What is … normative versus criterion-referenced assessment? Med Teach. 1989;11(2):145–50. 

Survey of important assessment instruments

• Bandiera G, Sherbino J, Frank JF. The CanMEDS assessment tools handbook: an introductory guide to assessment methods for the CanMEDS competencies. Ottawa (ON): The Royal College of Physicians and Surgeons of Canada; 2006.             

• Gagliardi AR, Brouwers MC, Finelli A, Campbell CM, Marlow BA, Silver IL. Physician self-audit: a scoping review. J Cont Ed Health Prof. 2011;31(4):258–64.

• Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 2011;45(1):1048–60.

• Kogan JR, Holmboe ES, Hauer KR. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–26.

• Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No. 12. Med Teach. 2009; 31(4):299–318.

• Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach. 2006;28(2):117–28.