Development of the competency-based objective resident
education using virtual patients system

Taylor Sawyer,¹ Alan Stein,² Holly Olson,³ C. Becket Mahnke¹

¹Department of Pediatrics, Tripler Army Medical Center, Honolulu, HI; ²Zargis Medical Corporation, Princeton, NJ;
³Directorate of Health Education and Training, Tripler Army Medical Center, Honolulu, HI, USA

Correspondence: Taylor Sawyer, Department of Pediatrics, MCHK-PE, Tripler Army Medical Center, 1 Jarrett White Rd, Honolulu, HI 96859. Tel. 808-433-6345. E-mail: taylorleesawyer@hotmail.com

Key words: competency, resident education, virtual patients, Accreditation Council for Graduate Medical Education, graduate medical education, performance.

Acknowledgements: this project was funded by the Telemedicine & Advanced Technology Research Center (TATRC), U.S. Army Medical Department Advanced Medical Technology Initiative (AAMTI).

Disclaimer: the views expressed in this manuscript are those of the author(s) and do not reflect the official policy or position of the Department of the Army, Department of Defense, or the US Government.

Conflict of interest: the authors report no conflicts of interest.

Received for publication: 18 December 2010.
Revision received: 3 March 2011.
Accepted for publication: 7 march 2011.

©Copyright T. Sawyer et al., 2011
Licensee PAGEPress, Italy
Medical Education Development 2011; 1:e5
doi:10.4081/med.2011.e5





Introducion

The Accreditation Council for Graduate Medical Education (ACGME) was developed to oversee the quality of physician graduate medical education (GME) in the U.S. The ACGME Outcomes Project requires each GME program to teach and evaluate their trainees in six core competencies: patient care, medical knowledge, practice-based learning and improvement, communication skills, professionalism, and systems-based practice.1 Developing innovative methods to meet the ACGME requirements is an ongoing area of research in medical education. Computerized virtual patients have been used extensively in undergraduate medical education and are being increasingly used in GME.2 Virtual patients offer the convenience of allowing trainees to practice patient care at any time/place and provide educators with the ability to standardize pathology in order to objectively evaluate medical trainee knowledge and clinical skills, and to provide a core curriculum of patient cases to trainees. As such, virtual patients have significant potential for use in the teaching and objective evaluation of the ACGME core competencies. In addition, virtual patients can provide a core curriculum of patient cases to trainees, ensuring that each trainee is exposed to necessary clinical scenarios in their specialty during their training which is not always possible relying solely on clinical encounters alone.
In 2009 Kash et al. reported the use of an online Objective Simulated Computerized Clinical Encounters (OSCCE), developed by the American Headache Society Neurology Resident’s Program, to teach and evaluate neurology resident’s ACGME competencies concerning headache management.3 They were able to show that a web-based OSCCE resulted in significant improvements in 5 of 6 ACMGE competencies. Their quantitative analysis focused on pretest and post-test scores which consisted of 50 randomized questions that tested baseline performance and then improvement on specific core competencies and understanding of headache. We could find no other reports of web-based OSCCE applications currently in use for GME training. Here we describe our work to develop an open-format, web-based OSCCE application which we have entitled the Competency-based Objective Resident Education using Virtual Patients (CORE-VP) system.

Innovation

Application design

CORE-VP was based on a linear-interactive (e.g. progression in one direction with choices) and student-authored design4 and complies with the MedBiquitous Virtual Patient data specifications, established by the Med­Biquitous Consortium Virtual Patient Working Group.5 The MedBiquitous Virtual Patient data specification architecture is comprised of four interlinked components, which include virtual patient data, media resources, a data availability model and an activity model. In this vernacular the virtual patient data comprises all the textual content of the virtual patient case including clinical data, narrative elements and questions and answers. The media resources includes binary media such as digital images, audio and video files. The data availability model aggregates the virtual patient data and media resources element for access through the activity model. The activity model accesses the data availability model, which references data points within the virtual patient data and media resources.
1. Virtual Patient Data: a MySQL database was used to store case information. For each component of the virtual patient encounter (history, physical exam, laboratory, radiography, management) a default choice list was created and populated with a default list of questions, physical exam findings, laboratory, radiologic, other studies and management options. Each laboratory and radiologic study was assigned a cost in U.S. dollars.
2. Media resources: the MySQL database was also used to store media files including WAV, MP3, JPEG, TIFF, pdf etc.. Basic HTML markup was employed for text fields, permitting links to externally hosted media. CORE-VP media could be viewed with the user’s native web-browser applications.
3. Data availability model: CORE-VP utilized a custom content management system which supported browser-based application access. All aspects of the clinical encounter were reduced to a fundamental Data Availability Model consisting of query, response, and simple storage/retrieval of supplemental materials (eg, audio, video, and images). Because the text elements employ basic markup, simple formatting and URL links to access externally hosted materials was possible. CORE-VP contains tools for case management, authorship and execution. The patient avatar was developed using MakeHuman™, an open source software program for the modeling of 3-Dimensional humanoid characters.6 A Flash (Adobe Systems Inc.) component was overlayed on the avatar to allow graphical navigation of the anatomy without changing the underlying Data Availability Model.
4. Activity model: the user interface was built on WaveMaker (WaveMaker Software™, Inc.) technology including AJAX and javascript. A predominantly menu-based format for information exchange was chosen over a natural language user interface, in which the user free type in questions, based on prior studies showing users favoring a menu-based design.2 This menu approach permits efficient navigation of the numerous clinical details that are explored during an encounter while retaining user autonomy. Because the number of menu options extended into hundreds of items (e.g., available laboratory and radiographic studies), the menu options were set to appear in hierarchical groups, and a real-time search function permitted an alternative navigation mechanism. The activity model allowed a natural progression from history and physical exam, through studies and management, but any aspect of the encounter may be revisited at any time. Free text fields for differential diagnosis and the clinical note are on-screen at all times. For efficiency, cut-and-paste could be used to populate these fields, though the content is solely at the discretion of the user.

Case authoring

The open-format structure of the CORE-VP system allows authors to create their own unique cases. Cases are designed to mimic a real-life patient encounter and includes; History, Physical Exam, Laboratory and Radiologic Studies, Diagnosis and Management sections. A set list of default history questions, physical exam findings, laboratory and radiologic studies results and management options are available. Answers to questions and test results can be edited by the case author using either text or uploaded media files via a browser-based interface. During case authoring each answer in the history, physical exam finding, laboratory and radiologic studies results and management decision modified by the case creator is assigned a weighted importance score of 1 to 5 which factors into the performance summary (PS) scoring.

Case execution

Cases are assigned by the case author or instructor to a resident using an administrative page in the CORE-VP application. The execution of the virtual clinical encounter begins with a case overview (Figure 1). In the History sections case data is accessed by clicking on a question. Once a question is clicked the answer appears in a new window. In the Physical Exam section an interactive patient avatar is available and physical exam findings (imported as images and video files) can be elicited by clicking the appropriate body part (Figure 2). Laboratory and Radiologic Studies and Management options are ordered by clicking items on the menu. During the case, users are able to move back and forth between the various sections of the virtual encounter by clicking the tabs at the top of the screen. There is an area to write in differential diagnoses and an area to write a patient encounter note.

Figure 1. Screenshot of CORE-VP’s case execution interface showing an initial case overview. Note the tabs at the top of the page allowing the trainee to move back and forth between the different sections of the encounter. The free text area on the right of the screen allows the trainee to add in differential diagnoses and complete the encounter note.

Figure 2. Screenshot of CORE-VP’s case execution interface showing the physical examination graphic user interface and patient avatar. To examine the patient the trainee simply moves the curser over the avatar to access the part of the body to examine. Images and videos of physical exam findings can be accessed by the trainee by clicking the body part. The physical exam findings can also be accessed via the menu, shown near the bottom of the screen in this figure.

Performance evaluation

To provide immediate enhanced feedback, an automated PS was developed. The user receives the automated PS after finished a case. The PS utilized multi-factorial evaluation metrics for each of the six ACGME core competencies (Table 1). The PS provides a percentage score for each AGME core competency (Figure 3). The PS is exportable as a pdf file and can easily be included in the student’s training portfolio. In addition to the PS an Efficiency Score and Cost-effectiveness Score are calculated. The Efficiency Score is derived from the total number of questions asked, exam items performed and tests ordered compared to those deemed important by case creator. The Cost-effectiveness Score is derived from the total cost of laboratory tests ordered and radiologic studies obtained compared to those deemed important by case creator. A multiple choice pre and post-test was incorporated to evaluate of content knowledge concerning the case topic. After each case the user also receives a Case Debriefing which includes an explanation of the case and the learning objectives (Figure 4).

Figure 3. Screenshot of CORE-VP’s Performance Summary with evaluations for all six ACGME core competencies provided in percentages. Further down on the screen (not shown) are the efficiency score, cost effectiveness score, performance scores in each sections of the encounter, the trainee’s list of differential diagnoses and the patient encounter note.

Figure 4. Screenshot of CORE-VP’s Case Debrief. Note the hypertext included within the debrief and the attached pdf article.

Table 1. Measurements of ACGME core competencies with CORE-VP.

Evaluation

Complete evaluation of the system remains to be completed. We plan to evaluate the CORE-VP system using three specific metrics; resident utilization, measures of educational efficacy, and resident and faculty impressions of system.

Resident utilization

Resident utilization of the CORE-VP module will be evaluated by the number of residents who utilize the system within a given time frame. System usage will be monitored by automated system user log-in tracking. We anticipate system usage by residents to approach 100% within our center by 12 months after implementation.
Measures of educational efficacy
At the conclusion of the each CORE-VP case the user receives immediate formative and summative feedback by way of the PS. These PS highlights potential areas for improvement and further study. The PS will be reviewed by the residency Program Director as part of the ongoing resident performance review process and will provide some objective evidence of the resident’s medical competence as required by the ACGME. In addition to the CORE-VP system PS, resident’s scores on specialty in-service examinations will be tracked after institution of the CORE-VP system. These test scores can then be compared retrospectively to test scores prior to institution of the CORE-VP system to determine if the CBL module results in improved scores on standardized medical exams.

User impressions of the system

The overall success of the CORE-VP system will be assessed from both educator and trainee standpoints. From the educator standpoint, we will obtain feedback from both residency Program Directors and the clinical teaching faculty. Specifically, residency Program Directors will be asked to describe results on the CORE-VP system PS, suggest improvements to the system, and report any problems encountered. Clinical teaching faculty will provide insight regarding their experience developing cases for the CORE-VP system, and any difficulties encountered in case development. From the trainee standpoint, we will assess the CORE-VP system’s effectiveness. Specifically we will assess system ease of us, barriers to access, technical problems and suggested system improvements.

Conclusions

We have developed a web-based OSCCE called CORE-VP, a linear-interactive virtual patient simulator that allows users to program their own virtual patient cases and thus can be used by medical educators from any medical specialty and for any training level. Each case is designed to mimic a real-life patient encounter and includes history, physical exam, laboratory/radiology, diagnosis and management. The system contains tools for case authorship, management and execution and permits linking of files or web-based hypermedia content. Upon completion of a case the user receives immediate feedback in the form of an automated, ACGME competency-based, PS. Work remains to be done on the CORE-VP system. We are planning to evaluate the efficacy of the system by tracking resident utilization, measuring educational efficacy, and collecting resident and faculty impressions of system. Future technical hurdles include workflow optimization and expanding compatibility to permit access from a wider selection of network connected devices (smartphones, etc.). Validation of the metrics used to determine competency also remain to be completed. The CORE-VP application is currently undergoing beta testing in the Pediatric Department at Tripler Army Medical Center.7 Once tested and validated we feel CORE-VP may provide a useful tool to medical educators and could function as a benchmark for comparing resident performance within and across training programs. After testing and validation is completed we plan to make the web-based application freely available to medical educators.

References

1. Accreditation Council for Graduate Medical Education. ACGME Outcomes Project. http://www.acgme.org/outcome (last accessed 11-11-2010).
2. Cook D, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: A systematic review and meta-analysis. Acad Med 2010;85:1589-602. [Pubmed]
3. Kash KM, Leas BF, Clough J, et al. ACGME competencies in neurology; Web-based objective simulated computerized clinical encounters. Neurology 2009;72:893-8. [Pubmed]
4. Ellaway R, Poulton T, Fors U, et al. Building a virtual patient commons. Med Teach 2008;30: 170-4. [Pubmed]
5. MedBiquitous Consortium Virtual Patient Working Group. www.medbiq.org (last accessed 12/8/2010).
6. MakeHuman.org. http://www.makehuman. org (last accessed 12/8/2010).
7. Pacific Asynchronous TeleHealth. Tripler Army Medical Center in Honolulu Hawaii. https:// path.tamc.amedd.army.mil (last accessed 12/9/2010).

[TOP]