Use Protocol 
for the 
Competency Project 
(Protocol #08-011)

Timothy Koschmann, Ph.D.
Principal Investigator
Dept. of Medical Education
Southern Illinois University
School of Medicine
(e-mail:  tkoschmann@siumed.edu)

Table of Contents

	1.	1.	Introduction	3
	2.	2.	Background	4
	3.	3.	Research Objectives	5
	4.	4.	Subject Selection	5
	5.	5.	Data Acquisition Protocol	5
	6.	6.	Research Questions	7
	7.	7.	Dissemination Plan	7
	8.	8.	Confidentiality/Data Management Plan	14
	9.	9.	Members of the Project Team	15
	10.	10.	Literature Cited	15

	•	Introduction
The Competency Project is a cross-disciplinary program designed to develop new insights into clinical skills development.  It is based on a shared set of data known as the Competency Data Corpus.  This corpus consists of an amassed collection of data records in which subjects (medical students and residents) worked up cases portrayed by standardized patients (SPs).  Each record has been graded using case-specific checklists and global ratings assigned by panels of clinical experts.  These assessment methods were modeled after those employed in high-stakes, licensure exams.   Researchers from a variety of disciplines including cognitive psychology, developmental linguistics, medical sociology, communication studies and computational linguistics will conduct a variety of studies using this data corpus.
In building such a collection, there is a problem with getting informed consent from the subjects captured on the video.  The problem is that the data once collected might be used in multiple ways and it may not be possible, at the time that the recording is made, to specify precisely what these uses might be.  Further, it is often not known at that time even who will be studying the data, since data corpora are often shared within a research community.  It is generally not feasible to re-consent the subjects when a researcher, sometimes years later, undertakes to study the data in the collection.  This problem is not unique to video-based corpora.  A very similar problem arises, for example, in research involving tissue-banks in which specimens are collected well before research plans have been fully formulated.  One way of protecting the rights of human subjects while making it possible to build and use shared repositories of tissue (or video) is to employ a two-stage model for carrying out the research.  The model employs two types of protocols— collection and use protocols.
Collection protocols must be reviewed and approved by the local IRB of the institution that owns and maintains the data corpus. Such protocols are used to build the collections or corpora to be later analyzed. Each collection protocol has one or more associated informed consent forms (ICFs).  The collection protocol specifies how data will be collected, who will do the collecting, and the constraints on ultimate use (as specified in the ICF[s]).  The materials in the Competency Data Corpus have been collected under  SCRIHS Protocol #06-126. 
To actually do research with the data gathered under a collection protocol, a use protocol is needed.  More than one use protocol might be linked to the same collection protocol.  Use protocols have no associated ICFs, but are bound by the terms of the ICF(s) of their parent (collection) protocol. Like collection protocols, use protocols must be reviewed and approved by the local institutional review board.  The use protocol must specify what data will be accessed, who will have access, how access will be implemented and controlled, specific plans for publication, and plans for destroying all copies of the data when the study is completed.   This is an application for a use protocol for a series of studies to be conducted under the auspices of the Competency Project with pilot funding from NIMH (NIH Grant 5R03MH74979-2).
	•	Background
The National Advisory Mental Health Council report “Bridging Science and Service” (NIMH, 1999) defines practice research as inquiry into “how to improve treatment or service delivery” by examining “how and which treatments or services are provided to individuals within service systems.”  Improvements in medical technology, skills, and therapies will not in and of themselves lead to improvements in patient care absent an adequately prepared and competent workforce of health care providers. Further, the potential losses associated with medical error are not only financial, but include the highest human costs—pain, suffering, and life itself (Institute of Medicine, 2000).  A series of more recent reports (Institute of Medicine, 2001, 2003) call for the development of new competencies for health care.  Competency is treated conventionally as a stable trait of an individual.  It is generally assumed to be measurable using appropriately designed instruments and that it is in some way linked to the possession of a relevant inventory of skills and disciplinary knowledge.  When there are calls for change in professional practice (such as those recently issued by the Institute of Medicine), the relevant inventory of skills and disciplinary knowledge change and, in this way, what counts as competent performance will also be subject to change.
Medical schools and licensing agencies are confronted daily with the task of evaluating and certifying professional competence.  The practices and technologies for doing so have evolved over time.  They include conventional paper and pencil tests (though now more frequently administered on a computer) as well as newer “performance-based assessments” (Newble et al., 1994; van der Vleuten, 1996; van der Vleuten & Swanson, 1990; Wass & van der Vleuten, 2004).  The National Board of Medical Examiners, for example, has begun testing using standardized patients as part of Step 2 of the United States Medical Licensing Examination (USMLE) (NBME, 2005).  This same exam, USMLE Step 2 CS, is now used for licensure of foreign medical graduates (ECFMG, n.d.). This means that all physicians now entering practice in the U.S. must take this exam regardless of how and where they received their training. Given the growing importance of these assessments, it is vital that we understand how they work and what they measure.
	•	Research Objectives
Multiple lines of inquiry will be pursued as a part of this pilot study.  Using the graded examples stored in the data corpus, researchers with diverse disciplinary backgrounds will pursue a variety of questions related to clinical competency and its assessment.  The overall goal is one of improving our understanding of what exactly is meant by professional competence.
	•	Subject Selection
The subject recruitment criteria are described in SCRIHS Protocol # 06-126.  All records in the Competency Data Corpus will be examined.
	•	Data acquisition protocol
The materials collected under SCRIHS #06-126 are structured as a set of data records.  Each record represents the results of a subject working up a particular patient problem.  There are more than one testing case represented in the data corpus and each is based on an actual case from the clinic.  The protocol for working up the patient’s problem has three steps:
Step 1: Patient encounter. Subjects are asked to take a history and perform a physical exam on an SP in the Professional Development Laboratory, a clinical simulation suite equipped with a one-way mirror and two video cameras. The facility was designed to have the look and feel of a conventional clinical examination room.  The encounter with the SP is timed (usually 20 minutes with a 5-min warning).  The encounter is videotaped.  
Step 2: Construction of a SOAP note.  After working up the patient, subjects retreat to an adjacent computer lab and are given 30 min. to compose a SOAP note and to order laboratory tests. The SOAP note is a standardized format for communication among healthcare professionals and consists of the subject’s interpretation of Subjective (patient reported findings) and Objective (observed findings) information gathered from the patient, as well as their Assessment of the significance of these findings and their Plan for further investigations and/or management. This represents information that would ordinarily go into the patient’s chart so the SOAP note might be thought of as a generic chart note for the patient visit.  In this study, SOAP notes are constructed using a program developed by DxR Consulting Group (http://www.dxrgroup.com). The program is web-based and stores the subjects’ input data to a secure SQL server.
Step 3: Debriefing interview.  After completing the SOAP note, subjects progress to a debriefing interview where they are questioned in detail about their preferred diagnosis and supporting reasoning.  The interviews have four distinct phases:
Phase 1:  Case presentation. The interview begins with the subject presenting the case according to standard case presentation form widely used to facilitate communication among healthcare professionals.  Within the form’s parameters, the presentation constitutes an un-cued summary of the subject’s understanding of the significant patient findings, their assessment of these findings and their plan for further investigations and management.  Following the un-cued presentation, the subject’s SOAP note is retrieved from the server and used as a vehicle for focusing the interview which will include questions as to working (preferred) diagnosis, significance of findings, findings that are inadequately accounted for by the data or un-elicited findings that would be expected given the working diagnosis, and actions the subjects on reflection wished they had taken.  Subjects are also asked to identify deficits in their knowledge that have become apparent.   Finally, subjects are asked to rate their confidence in the patient work-up to that point.
Phase 2: Reviewing laboratory findings. Following the first phase of the interview, the subject is given a copy of the requested laboratory findings and/or specialist consultant reports.  These reports are presented via the DxR program.  The interview continues with the same basic questions brought forward to include the newly acquired information.  Questions pertain to the diagnosis, relevant clinical findings, unaccounted for findings, expected findings not elicited, knowledge deficits, how they plan to proceed with the case and confidence in the work-up thus far.  
Phase 3: Reviewing expert report. After presentation of the ordered test results and the consultants’ reports, subjects are given a copy of a SOAP note prepared by an expert physician.  The interview resumes with the same set of questions brought forward to include the subject’s check of his work-up against that of the expert.  In addition to identifying knowledge deficits, the subject is asked to explain the underlying pathophysiology of the case. Further debriefing questions will get at the subject’s awareness of, recognition of, and response to error.  Is the subject surprised by information that came from investigation results, consult reports, or expert opinion?  Are subject’s confidence ratings of the adequacy of the work-up influenced by “expert” reasoning?  Subjects are asked for additional thoughts about the case, knowledge deficits, related cases in memory, and problems in its diagnosis.
Phase 4: General questions. The final phase of the interview is to assess knowledge difference as a potential explanation for performance differences by specifically probing knowledge of the basic and clinical science underlying the case. 
Grading of Data Records.  The records in the data corpus are graded using methods similar to those used in the practical tests conducted as a part of high-stakes medical licensing exams (USMLE Step 2CS).   Each record is graded in two ways.  First, a behavioral checklist is constructed for each testing case.  It consists of a list of questions and physical exam items relevant to the case.  Additional items (e.g., “The doctor/student/resident listened carefully to me.”) address general issues related to communication and professionalism.   The checklist is filled in after the encounter by the SP.  An agreed upon cut-off will be used to make pass/fail decisions.  A second grade will be assigned by a panel of three faculty evaluators who will view the video recording of the SP encounter and the subject’s SOAP note.  The evaluators are asked to individually rate (on a nine-point scale) the subject’s overall performance and its isolated components (e.g., history, physical exam, subjective portion of SOAP, etc.).  After doing so, they repeat the grading as a group to produce a set of consensus ratings.

	•	Research questions
Researchers from a variety of institutions will participate under the auspices of the NIMH-funded Competency Project.  They come from different disciplinary backgrounds and will pursue a variety of research questions.
What are the clinical problem solving strategies underlying competent clinical problem-solving? (Investigator: Seifert).  The extensive literature on diagnostic reasoning in medicine has centered on formal decision-making approaches (c.f. Arkes & Hammond, 1986).  While these approaches continue to address important issues in medical reasoning, they do not model the flow of cognitive processes occurring moment by moment during a patient encounter.  Other work has addressed cognitive processes in medicine by studying reasoning about complex diagnosis in presented cases (c.f. Patel, Arocha & Kaufman, 1994). A Cognitive Protocol Analysis, as described here, however, differs from these earlier approaches by examining problem solving within a processual framework. This analysis provides information about whether relevant domain knowledge is applied in specific circumstances deemed appropriate by experts. Pólya  (1957) postulated problem solving as four step process: Formulating the problem, devising a plan, executing the plan, and verifying the results. The behavioral evidence from a clinician’s handling of a case can be analyzed using this framework.  Work by Ericsson and Charness (1994) suggests expertise requires extensive “deliberate practice,” where one reflects on the aspects of performance that need improvement and practices those aspects in a thoughtful manner.  However, the nature of the cognitive processes involved in the type of practice that leads to expertise is unknown (Ericsson, Krample, Tesch-Romer, 1993).  The proposed work will contribute to the development of the theory of expertise by identifying the cognitive processes involved in metacognition about the problem solving process. Differences in reasoning will be analyzed in the transcripts of the post-encounter debriefing.  Though retrospective reports on cognition must be considered with caution (Ericsson and Simon, 1984), the four-component interview protocol will reveal how the subject dynamically responds to consistent and inconsistent information. It will also allow us to directly ask subjects about their reasoning, though the questions themselves will alter the reasoning observed.  This interview data will complement the straight observational data obtained through the videotaped patient encounter. This methodology allows the observation of the actions taken by the test subject, along with the questions asked and comments made during their patient encounter, supplemented by post interviews.
What knowledge representations underlie competent clinical performance? (investigator: Frederiksen). Related to the investigation of the reasoning used in building a case, is inquiry into the representations used in constructing a patient case.  Cognitive discourse analysis (CDA) is one approach to studying the representations (Frederiksen, 2001). Research on discourse comprehension in the 70’s and 80’s established that understanding involves a process of building semantic representations within concrete situations of cognition and action (Frederiksen, 1975, 2001).  These cognitive representations are related to one’s long-term store of conceptual and relational knowledge.  Semantic representations and knowledge can be modeled in the form of semantic networks (i.e., conceptual graph structures). Natural language discourse represents such information in the form of linguistically encoded meanings (i.e., propositions) that a speaker or writer asserts in describing a situation or knowledge for a listener or reader (Frederiksen & Breuleux, 1990). By this theory, a listener constructs propositions from the linguistic expressions and uses them to infer the relevant cognitive representations of the situation or knowledge being described. CDA begins by constructing models of experts' procedural and declarative knowledge in particular domains of problem solving.  The propositional content of subjects’ discourse contributions can then be mapped with reference to the expert task model so constructed as a means of both capturing the emerging case structure and of diagnosing possible deficiencies in the subjects’ preparation or understanding.  This has been done in tutorial dialog (Frederiksen & Breuleux, 1990; Frederiksen, 1999), but has yet to be attempted in a clinical problem-solving situation.  There are two phases to this form of analysis consisting of developing an “expert model” for each testing case and modeling the performance of individual students in particular case encounters. 
In clinical problem solving, how does ‘the case’ emerge from interactional processes of perspective taking? (Investigator: MacWhinney). Perspective theory has been previously applied within developmental linguistics (MacWhinney, 2005).  The theory assumes that understanding a sentence corresponds to cognizing an information structure with entities, properties, and relations.  It adds two additional assumptions that constitute radical extensions of traditional schema theory and mental model theory.  First, it assumes that representations are ‘embodied’ in the sense that they are mapped onto the functioning and physical constraints of the human body.  Second, it assumes that the successful operation of a mental model is traced by assuming the perspective of the operator. Perspective theory analyzes communication on five different levels (i.e., Direct Experience, Space and Time, Events, Social Roles and Mental Acts).  It examines the ways in which speakers provide cues to each other with regard to perspectives assumed.
Perspective-taking is evident in the Competency Data Corpus in the ways in which ‘the case’ emerges within the sequentially-unfolding interaction of the participants (i.e., subjects, SPs, interviewers).  At every turn, the participants instruct each other in the perspectives to be adopted and in the process collaboratively construct the patient problem.  This line of research applies Perspective theory to analyze how this is accomplished by tracing the succession of perspectives developed from the beginning of the interaction between the subject and SP all the way to the completion of the debriefing interview.
What kinds of performance errors do physicians make when diagnosing and managing patient problems?  (Investigators: Williams, Roberts). A key to moving learners from novice to expert status is recognizing, diagnosing and remediating their performance errors.  In aviation, Jones and Endsley (1996) analyzed reports from the Aviation Safety Reporting System database to categorize the types of errors that occur in the aviation environment.  A parallel effort will be conducted analyzing data within the Competency Data Corpus.  It will seek to develop a modified situation awareness taxonomy for analyzing physician performance deficiencies in the video records of physician-patient encounters and in post-encounter debriefing interviews designed to explore physician comprehension of the patient problem. The research specifically will address the following questions: 1) Which errors can be evaluated based on the physician-patient encounter alone. 2) Which require observing the physician-patient encounter and the interview, and 3) Which cannot be evaluated based on either of those sources of information.  It will also investigate which errors can only be assessed by experts and which can be assessed by trained non-experts. Finally it will compare error analysis ability of this system with that of other approaches developed in conjunction with the Competency Project and in the professional literature.
How might we study (or assess) interactional competence as a practical achievement? (Investigators:  Frankel, Koschmann)  Methods for assessing clinical skills for interviewing and performing physical examinations rely upon a combination of behavioral checklists and categorical grades assigned by trained observers.   Both approaches overlook the interactional and practical aspects of clinical competence, however.  Checklists, which are often case-specific, record whether a behavior was present or absent.  A checklist item, for example, might address whether or not the subject asked a question pertinent to the expected diagnosis.  How the matter is raised as a topic within the conversation, whether it is raised in a way that promotes adequate exploration and how it is actually treated by the interlocutors are all questions which are glossed over by the checklist item, however.  Similarly, categorical ratings such as “appearance and behavior were professional”, the subject “listened carefully,” or the subject “used words that were easy to understand” lose contact with the actual interactional practices whereby these impressions were formed.
In psychology the degree to which a test actually measures that which it was designed to measure is referred to as construct validity.  To assess the “clinical competence construct” (Colliver, 1995, p. 1062), we need to better understand what counts as competence in the clinic.  A substantial literature devoted to studying interactional practices in this setting already exists (e.g., Heritage & Maynard, 2007) and a recent systematic review has demonstrated positive associations between specific communication behaviors and care outcomes (Rao, Anderson, Inui, & Frankel, 2007).  The focus of this line of inquiry, therefore, is methodological.  It has to do with constructing new methods for more directly studying the actual practices through which clinical competency is achieved.  
What are the technical issues involved in machine-grading of SOAP notes? (investigator: Louwerse). SOAP note grading. Performance-based assessments frequently include a written component.  In the USMLE Step 2 CS, test takers are asked to create a “patient note” (NBME, 2005).  Subjects tested within the Competency Project will be asked to produce a SOAP note.  For reasons described below, these SOAP notes will be graded by a computer.  There are descriptions in the literature of automated grading of student-generated essays (Landauer, Foltz, & Laham, 1998). Automated grading has also been used in measuring student performance in tutoring, comparing a student answer to a tutor’s question with an expert’s answer (Graesser, Wiemer-Hastings, Wiemer-Hastings, Kreuz, & Tutoring Research Group, 1999). Latent semantic analysis (LSA) is a statistical, corpus based, technique for representing world knowledge. It computes similarity comparisons for terms and documents by benefiting from the fact that particular words occur in particular documents. LSA takes quantitative information about co-occurrences of words in documents (paragraphs and sentences) and translates this into an N-dimensional space. What is so special about LSA is that the semantic relatedness is not (only) determined by the relation between words, but also by the words that accompany a word (see Landauer & Dumais, 1997; Kintsch, 1998). In other words, terms like consciousness and mind will have a high cosine value (are semantically highly related) not because they occur in the same documents together, but because words that co-occur with one equally often co-occur with the other (see Baeza-Yates & Ribeiro-Neto, 1999, Kintsch, 1998; Landauer & Dumais, 1997; Landauer, Foltz & Laham, 1998). Recently, LSA has been successfully used to identify idiolect and sociolect (Louwerse, 2004), to classify genres and registers (Louwerse, McCarthy, McNamara & Graesser, 2004) and embodied representations like space and time (Louwerse, 2007).  
This study will focus primarily on the SOAP notes using the panel assessments first for training purposes and later to explore agreement with machine-generated grades.  Some exploratory work might also be done with the transcripts from the debriefing interviews.
	•	Dissemination Plan
The Competency project team is multidisciplinary.  When doing project-related writing, however, we view of the work of project as being most relevant to certain specific literatures, namely, research in the cognitive sciences (rep. journals: Cognitive Science, Cognition, Memory &Cognition), medical education and assessment (rep. journals:  Medical Education, Teaching and Learning in Medicine), and communication research (rep. journals: Discourse Studies, Discourse Processes, Research on Language and Social Interaction).
Any plans to publish or present images or recordings (either audio or video) from the corpus will require permission in advance from the corpus curator. Authors/presenters will notify the corpus curator of all presentations and scholarly publications based on materials from the Competency corpus.  All such work will include a notice acknowledging the source of the data.
	•	Confidentiality/Data Management Plan
An agreement included as an addendum to this protocol delineates the researchers’ rights and responsibilities with regard to the data contained in the Competency.  All materials will be de-identified, that is, all proper references to physicians, patients, hospitals, etc. will be removed, before distribution to the project team. Distribution will be in the form of mailed DVD sets or as downloads from a password-protected website.  Each member of the team will treat the distributed materials as confidential research materials and will be responsible for keeping them secure. Accurate records will be maintained of all distributions of recorded materials over the course of the project.  In carrying out their work, project team members are authorized to display and discuss the materials with students and other collaborators (see discussion of “data sessions” in “Research Plan”).  They may make copies of these materials for their own use, but may not disseminate the materials outside the project team.
	•	Project Team Members
Timothy Koschmann (Southern Illinois University) has spent 15 years examining interaction in settings of collaboration and learning (e.g., Koschmann & LeBaron, 2002; Koschmann et al., 2007).  He is the Principal Investigator for the Competency Project.  
Richard Frankel (Indiana University) is sociologist internationally recognized for his contributions to research on doctor/patient interaction (Frankel, 1984; Beckman & Frankel, 1984).  He is an external collaborator and investigator on the Competency Project. 
Carl Frederiksen (McGill University) is a cognitive psychologist with many years experience studying discourse from a cognitive perspective (Frederiksen, 1975; 2001).  He is an external collaborator and investigator on the Competency Project. 
Max Louwerse (Univ. of Memphis) is a computational linguist specializing in semantic analysis (Louwerse, 2004; 2007). He is an external collaborator and investigator on the Competency Project.
Brian MacWhinney (Carnegie-Mellon Univ.) is a psychologist working in the area of developmental linguistics (MacWhinney, 2005a, 2005b). He also has extensive experience building and maintaining large-scale data corpora (e.g., MacWhinney, 2000). He is an external collaborator and investigator on the Competency Project. 
Nicole Roberts (Southern Illinois University) is an educational researcher with special interests in adult learning and leadership. She is an investigator on the Competency Project. 
Colleen Seifert (Univ. of Michigan) is a cognitive psychologist with interests in the role of the memory in decision making and problem solving (e.g., Seifert, Patalano, Hammond, & Converse, 1997; Seifert & Patalano, 2001). She is an external collaborator and investigator on the Competency Project.
Reed Williams (Southern Illinois University) is an educational psychologist with extensive experience in medical education and assessment (e.g., Williams, Tafesse, Sinacore, & Valaski, 1998; Williams, 2004). He is an investigator on the Competency Project.
9. Literature Cited
1. Arocha, J. F., & Patel, V. (1995). Novice diagnostic reasoning in medicine: Accounting for evidence. Journal of the Learning Sciences, 4, 355-384.
Aspegren, K. (1999). BEME Guide No. 2:  Teaching and learning communication skills in medicine—a review with quality grading of articles. Medical Teacher, 21, 563-570.
2. Baeza-Yates, R., & Ribeiro-Neto, B. (Eds.) (1999). Modern information retrieval. New York: ACM Press.
3. Beckman, H., & Frankel, R. (1984). The effect of physician behavior on the collection of data. Annals of Internal Medicine, 101, 692-696.
Boon, H., & Stewart, M. (1998). Patient-physician communication assessment instruments: 1986 to 1996 in review. Patient Education and Counseling, 35, 161-176.
4. Boshuizen, H. P., & Schmidt, H. G. (1992). On the role of biomedical knowledge in clinical reasoning by experts. Cognitive Science, 16, 153-184.
5. Brooks, L. R., LeBlanc, V. R., &  Norman, G. R.  (2000). On the difficulty of noticing obvious features in patient appearance. Psychological Science, 11(2), 112-117.
6. ECFMG (n.d.).  Step 2 CS—General information.  Retrieved February 2, 2006 from http://www.ecfmg.org/usmle/step2cs/index.html.
7. Elstein, A., Shulman, L. S., & Sprafka, S. A. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press.
Flaherty, J. A. (1985). Education and evaluation of interpersonal skills. In A. G. Rezler & J. A. Flaherty (Eds.), The interpersonal dimension in medical education (pp. 101-146). New York: Springer-Verlag.
Frankel, R. (1984). From sentence to sequence: Understanding the medical encounter through micro-interactional analysis. Discourse Processes, 7, 135-170.
Frederiksen, C. (1975). Representing logical and semantic structure of knowledge acquired from discourse. Cognitive Psychology, 7, 371-458.
Frederiksen, C. (2001). Propositional representation in cognitive psychology. In N. J. Smalser & P. B. Baltes (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Amsterdam: Elsevier Science.
8. Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & the Tutoring Research Group. (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51. 
9. Heritage, J., & Maynard, D. (Eds.) (2006). Practicing Medicine: Structure and Process in Primary Care Consultations. Cambridge, U.K.: Cambridge University Press.
10. Hester, S., & Francis, D. (Eds.). (2000). Local educational order: Ethnomethodological studies of knowledge in action. Philadelphia, PA: John Benjamins.
Jones, D.G. & Endsley, M.R. (1996). Sources of situation awareness errors in aviation. Aviation, Space & Environmental Medicine, 67, 507-512. 

11. Institute of Medicine (2000).  To err is human: Building a safer health system.  Washington, D.C.: National Academies Press.
12. Institute of Medicine (2001).  Crossing the Quality Chasm: A new health system for the 21st Century.  Washington, D.C.: National Academies Press.
13. Institute of Medicine (2003).  Health professions education: A bridge to quality.  Washington, D.C.: National Academies Press.
14. Kintsch, W. (1998).  Comprehension: A paradigm for cognition.  NY: Cambridge University Press.
Koschmann, T., LeBaron, C., Goodwin, C., Zemel, A., & Dunnington, G. (2007). Formulating the triangle of doom. Gesture, 7, 97-118.
15. Landauer, T.K., & Dumais, S.T. (1997).  A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge.  Psychological Review, 104, 211-240.
16. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259-284.
17. Louwerse M.M. (2004). Semantic variation in idiolect and sociolect: Corpus linguistic evidence from literary texts. Computers and the Humanities, 38, 207-221.
18. Louwerse M.M. (2007). Symbolic or embodied representations: A case for symbol interdependency.  In T. Landauer, D. McNamara, S. Dennis, & W. Kintsch (Eds.), The handbook of latent semantic analysis (pp. 107-120). Mahwah, NJ: Lawrence Erlbaum.
19. Louwerse, M.M., McCarthy, P.M., McNamara, D.S. & Graesser, A.C. (2004). Variation in language and cohesion across written and spoken registers. In K. Forbus, D. Gentner & T. Regier (Eds.), Proceedings of the twenty-sixth annual conference of the Cognitive Science Society (pp. 843-848). Mahwah, NJ: Erlbaum.
MacWhinney, B. (2000). The CHILDES project:  Tools for analyzing talk. Mahwah, NJ: Lawrence Erlbaum Assoc.
20. MacWhinney, B. (2005a) The emergence of grammar from perspective taking. In D. Pecher & R. Zwaan (Eds.), The grounding of cognition: The role of perception and action in memory (pp. 198-223).  NY: Cambridge University Press.
MacWhinney, B. (2005b). A unified model of language acquisition. In J. F. Kroll & A. M. B. de Groot (Eds.), Handbook of bilingualism:  Psycholingistic approaches (pp. 49-67). New York: Oxford University Press.
Maynard, D., & Heritage, J. (2005). Conversation analysis, doctor-patient interaction and medical communication. Medical Education, 39, 428-435.
21. NBME (2005).  2006 Step 2 Clinical Skills (CS) Content Description and General Information.  Philadelphia, PA: National Board of Medical Examiners.  Retrieved February 2, 2006 as: http://www.usmle.org/step2/Step2CS/Step2CS2006GI/2006Step2CS.pdf
22. Newble, D. et al. (1994).  Guidelines for assessing clinical competence.  Teaching and Learning in Medicine, 6, 213-220.
23. NIH (2003).  NIH data sharing policy and implementation guidance.  Bethesda, MD: Office of Extramural Research, National Institutes of Health.  Retrieved May 30, 2003 as: http://grants.nih.gov/grants/policy/data_sharing/data_sharing_guidance.htm
24. NIMH (1999).  Bridging science and service.  Bethesda, MD: National Advisory Mental Health Council Report.  Retrieved June 25, 2005 as: http://www.nimh.nih.gov/publicat/nimhbridge.pdf
25. Patel, V., & Arocha, J. F. (2001). The nature of constraints on collaborative decision making in health care settings. In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum.
26.Patel, V. L., Arocha, J. F., & Kaufman, D. R. (1994).  Diagnostic reasoning and medical expertise.The Psychology of Learning and Motivation, 31: 187-252. 
27. Patel, V., Evans, D. A., & Groen, G. J. (1989). Biomedical knowledge and clinical reasoning. In D. A. Evans & V. Patel (Eds.), Cognitive science in medicine: Biomedical modeling. (pp. 49-108). Cambridge, MA: MIT Press.
28. Patel, V., Kaufman, D. R., & Magder, S. A. (1996). The acqisition of medical expertise in complex dynamic environments. In A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports and games. (pp. 127-165). Mahwah, NJ: Lawrence Erlbaum.
Rao, J. K., Anderson, L. A., Inui, T. S., & Frankel, R. (2007). Communication interventions make a difference in conversations between physicians and patients:  A systematic review of the evidence. Medical Care, 45(4), 340-349.
Seifert, C., & Patalano, A. L. (2001). Opportunism in memory:  Preparing for chance encounters. Current Directions in Psychological Science, 10, 198-201.
Seifert, C., Patalano, A. L., Hammond, K. J., & Converse, T. M. (1997). Experience and expertise:  The role of memory in planning for opportunities. In P. J. Feltovich, K. M. Ford & R. R. Hoffman (Eds.), Expertise in context:  Human and machine (pp. 101-123). Menlo Park, CA: AAAI Press.
29. Turnbull, J., Turnbull, J., Jacob, P., Brown, J., Duplessis, M., & Rivest, J. (2005). Contextual considerations in summative competency examinations:  Relevance to the long case. Academic Medicine, 80, 1133-1137.
30. van der Vleuten, C.P.M. (1996).  The assessment of professional competence: Developments, research, and practical implications.  Advances in Health Sciences Education, 1, 41-67.
31. van der Vleuten, C.P.M. & Swanson, D. (1990).  Assessment of clinical skills with standardized patients: State of the art.  Teaching and Learning in Medicine, 2, 58-76.
32. Wass, V., & van der Vleuten, C. (2004). The metric of medical education: The long case.  Medical Education, 38, 1176-1180.
Williams, R. (2004). Have standardized patient examinations stood the test of time and experience? Teaching and Learning in Medicine, 16, 215-222.
Williams, R., Tafesse, E., Sinacore, J., & Valaski, M. (1998). Are standardized patient ratings influenced by patient perceptions of the preceding examinee. Teaching and Learning in Medicine, 10, 201-206.