A.	 Specific Aims
The Competency Project seeks to foster foundational inquiry into the cognitive and interactional elements of competent clinical performance.  Our interest is in unpacking what is actually measured in consequential practical exams widely used in medical education and licensure.  Becoming more articulate about what we mean by clinical competence may lead to suggestions for improving both professional development and its assessment. Under the scope of this R03 proposal we request funding to achieve the following two goals:
1) To expand our existing collection of graded professional practice by (a) developing a new case in psychiatry, (b) training a standardized patient (SP), and (c) running a set of subjects.
2) To evaluate several promising frameworks (i.e., Cognitive Protocol, Interactional, Cognitive Discoursee) for analyzing this pilot data.  
B.	Background and Significance
The National Advisory Mental Health Council report “Bridging Science and Service” (NIMH, 1999) defines practice research as inquiry into “how to improve treatment or service delivery” by examining “how and which treatments or services are provided to individuals within service systems.”  Improvements in medical technology, skills, and therapies will not in and of themselves lead to improvements in patient care absent an adequately prepared and competent workforce of health care providers. Further, the potential losses associated with medical error are not only financial, but include the highest human costs—pain, suffering, and life itself (Institute of Medicine, 2000).  A series of more recent reports (Institute of Medicine, 2001, 2003) call for the development of new competencies for health care.  Competency is treated conventionally as a stable trait of an individual.  It is generally assumed to be measurable using appropriately designed instruments and that it is in some way linked to the possession of a relevant inventory of skills and disciplinary knowledge.  When there are calls for change in professional practice (such as those recently issued by the Institute of Medicine), the relevant inventory of skills and disciplinary knowledge change and, in this way, what counts as competent performance will also be subject to change.
Medical schools and licensing agencies are confronted daily with the task of evaluating and certifying professional competence.  The practices and technologies for doing so have evolved over time.  They include conventional paper and pencil tests (though now more frequently administered on a computer) as well as newer “performance-based assessments” (Newble et al., 1994; van der Vleuten, 1996; van der Vleuten & Swanson, 1990; Wass & van der Vleuten, 2004).  The National Board of Medical Examiners, for example, has begun testing using standardized patients as part of Step 2 of the United States Medical Licensing Examination (USMLE) (NBME, 2005).  This same exam, USMLE Step 2 CS, is now used for licensure of foreign medical graduates (ECFMG, n.d.). This means that all physicians now entering practice in the U.S. must take this exam regardless of how and where they received their training. Given the growing importance of these assessments, it is important that we understand how they work and what they measure.
In performance-based assessments, there are two basic ways of evaluating competency.   One is to establish prior to testing a set of criteria for adequate performance for a particular case.  These might consist of a list of questions that the subject should ask in taking a history or particular physical exam items that should be performed.  Alternatively, the criteria might include specific tests that should be ordered or particular disorders that should appear on the subject’s differential diagnosis.  These criteria are case-specific and are represented as a check-off list.  Alternatively, competency may be assessed globally.  In this approach, a trained observer watches the subject work up the case and then forms a judgment of the performance taken as a whole.  Neither evaluation directly identifies the specific components of competency.  They do not, for example, tell us why asking one question and not another is wrong or why asking a question in a particular way might be considered a form of bad practice or what a test taker would need to do in order to appear ‘competent’.  To answer such questions we would have to look elsewhere.
One reasonable place to seek answers might be in the cognitive science literature focusing on expertise and particularly expertise related to medical diagnosis. Most of the literature on diagnostic reasoning has focused on the kinds of presenting complaints one commonly encounters in primary care medicine (e.g., chest pain, shortness of breath). Relatively little attention has been directed to the more complex problems of psychiatric practice where diagnoses can seldom be settled with a single laboratory test. Research into diagnostic reasoning has largely focused on two issues:  how (and what) basic science knowledge is brought to bear on clinical problems (Boshuizen & Schmidt, 1992; Patel, Evans, & Groen, 1989) and the basic reasoning processes employed (Arocha & Patel, 1995; Elstein, Shulman, & Sprafka, 1978).   The translation of a set of signs and symptoms into a case is not a purely cognitive task, however, but is instead a collaborative achievement worked out interactively in a practical setting (Patel & Arocha, 2001).  There is a need, therefore, to extend research on medical expertise into the complex, interactive situations in which medical problem solving is actually done (Patel, Kaufman, & Magder, 1996).  
One place where one could find information on the interactional aspects of professional competency is in the sociological literature on doctor/patient communication (c.f., Maynard & Heritage, 2005; Heritage & Maynard, forthcoming).  Past research on medical communication has suggested that the conversational organization of the clinical encounter can shape clinical findings.  Beckman and Frankel (1984), for example, demonstrated that key information for constructing a case may be lost depending on how the doctor poses questions to the patient.  Just as an assessment of competency that neglects the interactional facets of case building would be incomplete, however, an assessment of competency that focused exclusively on interaction to the exclusion of medical content would be equally incomplete.  An analysis of clinical competence must, therefore, incorporate an appreciation for both the cognitive and interactional aspects of clinical performance.
The Competency Project seeks to foster foundational inquiry into the components, cognitive and interactional, of competent professional performance. We will apply standard methods of assessment based on current state-of-the-art practices and technologies for performance-based testing to construct a collection of graded instances of professional performance. Our goal in creating this data set is to study in detail just how competency is evaluated.  We will approach this task using a variety of different analytic frameworks in order to explore which of these frameworks might be most useful for producing new understandings of competency.  Though it is beyond the scope of the proposed work, our hope is that new ways of discussing and conceptualizing competency may eventually be developed based on analytic findings of this project.  Such conceptualizations may direct us not only toward new ways of assessing competence, but also new and better ways of preparing newcomers for entry into professional practice.
C.	Preliminary Studies
An operational model of how competency is currently assessed in medical education and licensure was developed in previous work.  A description of its components is provided in the section entitled “Standard Methods of Assessment.”  A sample testing case (“Shirley Naylor”) was created involving a patient who presented with a diffuse collection of complaints (headache, tingling in legs, itchy skin).  A later laboratory test revealed that she suffered from polycythemia vera.  A standardized patient was trained to present this case.  Four clinical faculty members did a test run with the standardized patient and then data were gathered for 7 test subjects — 2 second year medical students (MS2s), 1 fourth-year medical student (MS4), 3 first-year residents (PG-1s), and 1 third-year resident (PG-3).
D.	Research Design and Methods
Case Development.  A new case (“Mary Watkins”), taken from psychiatric practice, will be used in this phase of the project.  This case was based on an actual patient seen and treated by one of the clinical faculty members.  It will be presented in a way that, to whatever degree possible, reproduces circumstances under which the patient was originally seen.  Such cases afford the opportunity to observe how clinicians and pre-clinicians at various levels of training interact with patients and reason through a clinical encounter.  The case selected for use in this study will be constructed and delivered with the same high fidelity to clinical practice.  In order to capture reasoning at the level of problem solving, cases will be selected in which findings are encountered that are unexpected or ambiguous if one were using pattern recognition as the diagnostic mode (cf., Arocha & Patel, 1995).  This will allow investigators to look at how subjects handle incongruous data, whether through problem-solving, ignoring the new data or discounting it, and whether there are differences across levels of training and expertise.  
In the “Mary Watkins” case, the initial patient workup suggests a psychiatric working diagnosis.  Additional data, however, should trigger the possibility that the psychological symptoms are secondary to a primary organic lesion.  In addition to looking at patient-doctor interaction and clinical reasoning, such a case will allow for investigating whether bias held by non-psychiatric clinicians would predispose them to overlook subtle or equivocal neurological signs.  The case involves a 24 year-old female elementary teacher who is referred by her primary care physician.  The patient, who has a history of depression, complains of headache and recurring depression.  A CT scan workup by the primary physician revealed no lesions.  The patient spontaneously reports no associated symptoms.  However, on direct questioning via the neurological review of systems, the patient admits to some difficulty swallowing, a few incidents of choking on food, and an occasional feeling of being “off balance” over the past 3-4 months.  Final diagnosis via MRI:  cerebellar glioblastoma. The differential diagnosis in this case is likely to weigh heavily toward psychiatric conditions because of setting (psychiatrist’s office), referral following a presumably thorough neurological work-up, and the patient’s previous history of mental illness.   The case and setting allow a rich field for investigating clinical reasoning. A preliminary run with this case will be done with four clinical faculty members.
Subject Selection.  We will seek paid volunteers of three types: second-year medical students (MS2s), fourth-year medical students (MS4s), and residents (PG-1s, PG-2s, PG-3s).  We will recruit a minimum of 8 subjects and ensure that we have at least two representatives of each type.  We will employ a policy of over-sampling, if necessary, to match proportions of minority and female participants to the pools from which they were drawn (see “Human Subjects”) and will continue running subjects until we meet our initial recruitment goals (i.e., minimum of 4 female subjects and 2 subjects from any category other than White/Non-Hispanic).  For pilot work, our sample will be small, but we consider it important to build meaningful variability into the data corpus from the outset.
Components of the Clinical Encounter.  A three-step protocol will be followed.  This protocol has been previously piloted to insure a high degree of fidelity to clinical practice and to establish the adequacy of the produced data for the various planned analyses.  SIU School of Medicine has excellent resources for conducting evaluations of this type.  Many of the techniques of performance-based assessment were actually developed here.  Faculty and staff have over twenty years experience developing cases and conducting performance-based exams.  There are also excellent facilities for carrying out the components of the clinical encounter (see “Resources”).
Step 1: Patient encounter. Subjects will be asked to take a history and perform a physical exam on an SP in the Professional Development Laboratory (see “Resources”), a clinic simulation suite equipped with a one-way mirror and two video cameras. The facility was designed to have the look and feel of a conventional clinical examination room.  The encounter with the SP will be limited to 20 min (with a 5 min warning).  A pair of high-quality, synchronized digital recordings will be produced of the encounter with the SP.  
Step 2: Construction of a SOAP note.  After working up the patient, subjects will progress to an adjacent computer lab and will be given 30 min. to compose a SOAP note and to order laboratory tests. The SOAP note is a standardized format for communication among healthcare professionals and consists of the subject’s interpretation of Subjective (patient reported findings) and Objective (observed findings) information gathered from the patient, as well as their Assessment of the significance of these findings and their Plan for further investigations and/or management. This represents information that would ordinarily go into the patient’s chart so the SOAP note might be thought of as a generic chart note for the patient visit.  For the purposes of this project, the SOAP note will be constructed using a program developed by DxR Consulting Group (http://www.dxrgroup.com). The program is web-based and will store the subject’s input data to a secure SQL server.
Step 3: Debriefing interview.  After completing the SOAP note, subjects will progress to a debriefing interview where they will be questioned about their preferred diagnosis and supporting reasons. There is no time limit for the interview.  There is a pre-established set of questions and the discussion will go on until all of the questions have been asked.  This questioning will potentially uncover information not observable during the subject’s SP encounter, and not included in the SOAP note.  Interviews will be conducted by either Nicole Roberts or Timothy Koschmann.  It will have four distinct phases:
Phase 1:  Case presentation. The interview will begin with the subject presenting the case according to standard case presentation form widely used to facilitate communication among healthcare professionals.  Within the form’s parameters, the presentation constitutes an un-cued summary of the subject’s understanding of the significant patient findings, their assessment of these findings and their plan for further investigations and management.  Following the un-cued presentation, the subject’s SOAP note will be retrieved from the server and used as a vehicle for focusing the interview which will include questions as to working (preferred) diagnosis, significance of findings, findings that are inadequately accounted for by the data or unelicited findings that would be expected given the working diagnosis, and actions the subjects on reflection wished they had taken.  Subjects will also be asked to identify deficits in their knowledge that have become apparent.   Finally, subjects will be asked to rate their confidence in the patient work-up to that point.
Phase 2: Reviewing laboratory findings. Following the first phase of the interview, the subject will be given a copy of the requested laboratory findings and/or specialist consultant reports.  These reports will be presented via the DxR program.  The interview will continue with the same basic questions brought forward to include the newly acquired information.  Questions will pertain to the diagnosis, relevant clinical findings, unaccounted for findings, expected findings not elicited, knowledge deficits, how they plan to proceed with the case and confidence in the work-up thus far.  This repeated interview following the incremental addition of information will allow the comparison of reasoning outcomes.  How do subjects adapt their reasoning given new information?  Does the response vary depending on confirming vs. contradictory information?  Do subjects update their diagnoses with new information, or stick to the original, potentially inadequate conclusion?
Phase 3: Reviewing expert report. After presentation of the ordered test results and consultant’s reports, subjects will be given a copy of a SOAP note prepared by an expert physician.  The interview will then resume with the same set of questions brought forward to include the subject’s check of his work-up against that of the expert.  In addition to identifying knowledge deficits, the subject will be asked to explain the underlying pathophysiology of the case. Further debriefing questions will get at the subject’s awareness of, recognition of, and response to error.  Is the subject surprised by information that came from investigation results, consult reports, or expert opinion?  Are subject’s confidence ratings of the adequacy of the work-up influenced by “expert” reasoning?  Subjects will be asked for additional thoughts about the case, knowledge deficits, related cases in memory, and problems in its diagnosis.
Phase 4: General questions. The final phase of the interview is to assess knowledge difference as a potential explanation for performance differences by specifically probing knowledge of physiology, pathophysiology and psychopathology underlying the case. 
In earlier pilot work, the debriefing interviews took 30-60 min to complete.  A digital recording will be made of the debriefing interview.  
Standard Methods of Assessment.  There are two general methods of doing performance-based assessment, the long case method and so-called OSCEs (Objective-Structure Clinical Examinations).  Long cases involve an extensive clinical encounter (often with a real patient) followed by some sort of post-encounter report.  OSCE exams, on the other hand, consist of multiple stations each set up to test a particular skill.  Some, but not all, stations may involve the use of standardized patients.  The psychometric properties of both types of exams have been studied extensively (van der Vleuten, 1996; van der Vleuten, & Swanson, 1990; Wass & van der Vleuten, 2004; Turnbull et al., 2005).  In prior work we designed a set of standard testing methods based on current practices and technologies of assessment.  Our methods employ a modified long case format that incorporates many of the features of contemporary high-stakes tests such as the USMLE Step 2.  Specifically, we will use three different instruments to produce three judgments of competency for each protocol.
SP checklist.  A special checklist will be constructed for each testing case.  This will consist of a list of relevant questions and examination items provided by the case author.  A scoring rubric, also provided by the case author, will be used to assign a pass/fail judgment on the checklist form.  In addition to the case-specific components there will also be some general items that allow the SP to grade things such as question skill, information-sharing skill, and professional manner.  Space will be provided for the SP to offer free-text comments on the encounter.  This form gathers information similar to that collected for the data gathering component of the “Integrated Clinical Encounter” (ICE) and the “Communication and Interpersonal Skills” (CIS) scores of the USMLE Step 2 CS (NBME, 2005).
SOAP note grading. Performance-based assessments frequently include a written component.  In the USMLE Step 2 CS, test takers are asked to create a “patient note” (NBME, 2005).  Subjects tested within the Competency Project will be asked to produce a SOAP note.  For reasons described below, these SOAP notes will be graded by a computer.  There are descriptions in the literature of automated grading of student-generated essays (Landauer, Foltz, & Laham, 1998). Automated grading has also been used in measuring student performance in tutoring, comparing a student answer to a tutor’s question with an expert’s answer (Graesser, Wiemer-Hastings, Wiemer-Hastings, Kreuz, & Tutoring Research Group, 1999). Latent semantic analysis (LSA) is a statistical, corpus based, technique for representing world knowledge. It computes similarity comparisons for terms and documents by benefiting from the fact that particular words occur in particular documents. LSA takes quantitative information about co-occurrences of words in documents (paragraphs and sentences) and translates this into an N-dimensional space. What is so special about LSA is that the semantic relatedness is not (only) determined by the relation between words, but also by the words that accompany a word (see Landauer & Dumais, 1997; Kintsch, 1998). In other words, terms like consciousness and mind will have a high cosine value (are semantically highly related) not because they occur in the same documents together, but because words that co-occur with one equally often co-occur with the other (see Baeza-Yates & Ribeiro-Neto, 1999, Kintsch, 1998; Landauer & Dumais, 1997; Landauer, Foltz & Laham, 1998). Recently, LSA has been successfully used to identify idiolect and sociolect (Louwerse, 2004), to classify genres and registers (Louwerse, McCarthy, McNamara & Graesser, 2004) and embodied representations like space and time (Louwerse, forthcoming).  There are two phases to the process of comparing text samples using LSA.
1) Computing the semantic space.  To use LSA to grade free-response text one must first construct a semantic space to represent meaning relationships in the domain of interest by analyzing a substantial body (or corpus) of text (Landauer & Dumais, 1997; Landauer, Foltz, & Laham, 1998). The analyzed text corpus may be a textbook, an encyclopedia, or some other large example of language use topically-related to the text samples to be graded.  First, the set of corpora are segmented into documents, generally paragraphs and entered in LSA. Next, LSA creates a large term-document matrix from the input.  Singular Value Decomposition (SVD) is used to reduce noise so that only the most important dimensions corresponding to the larger singular values are kept. Each word then becomes a weighted vector on K dimensions. The dot product (cosine) between two vectors can be seen as an estimate of the semantic relationship between the words. Following this operation, the "semantic space" is approximated by a matrix with each row once again representing a unique word type and roughly 300 columns corresponding to the contexts in the original matrix that capture the greatest variance.
2) Comparing text semantics.  Once the semantic space has been created, SOAP notes can be automatically scored. The performance of the subject would be evaluated by comparing the produced SOAP note to a golden standard.  The gold standard in this case will be a chart note prepared and maintained by the case author.  As described earlier, this traditional LSA comparison approach has proven to be extremely valuable in automated essay grading as well as comparing student answers to an ideal answer in intelligent tutoring systems.
There are several reasons for using LSA to grade the SOAP notes, the most obvious being that it will reduce some of the work of evaluating subject records.  It will also enable us to control the consistency of the grading.  Further, by preserving a copy of the gold standard used in grading a SOAP note, we can produce an audit trail that will display how assessments of competence change over time.  Finally, we anticipate that the medical certification industry will eventually automate its grading of free-text responses, so by employing LSA to score SOAP notes we are anticipating future assessment practice.  Because the semantic space will not have been constructed in the pilot phase of the project, scoring of SOAP notes during this period will be done by members of the Clinical Evaluation Panels (see below for further information about these panels).  
Global Assessment of Competency.  After testing is completed, three-member Clinical Evaluation Panels will be convened to produce global assessments of each protocol.  Members of these panels will have access to the video from the SP encounters and the debriefing interviews along with the subjects’ SOAP notes.  The panel must arrive at a consensus pass/fail judgment with regard to each protocol.  Unlike the other two measures described earlier, this assessment will be sensitive to experience.  The graders will be instructed to evaluate whether the performance is acceptable at their current level of training.  The expert panel meetings will be convened in the Collaboration and Learning Lab (see “Resources”). The video will be presented in such a way that the panelists will be able to rapidly locate the segments of greatest interest. We estimate that it will take approximately 30 min per protocol to make these assessments and we would schedule several panel meetings of approximately 2 hours each to grade all of the pilot data records collected under this grant.  The work of serving on these panels will be shared among the project team members with clinical expertise (see “The Project Team Members and Their Responsibilities”).  A digital audio recording of the panel’s deliberations will be made and it will be linked to the protocol in the data corpus. 
Constructing the Data Corpus.  Our data corpus will be constructed by employing standard methods of assessment to produce a collection of records. The data gathered from the “Mary Watkins” case will complement the data previously gathered for the “Shirley Naylor” case.  Once we move past the pilot stage, we hope to collect a large number of data records based on multiple cases and a substantial number of subjects, some of whom may be repeatedly tested over the course of their training.  Digital video will be captured in Raw DV format.  Storage of the data in this format, however, would be prohibitive in terms of space and the files would be too large to stream over the web or transport conveniently.  All timeline-based data, therefore, will be stored online in a compressed format.  It may, in fact, prove convenient to store some or all of the timeline-based data in multiple formats for different uses.  Annotations are data objects that are linked to and describe some fragment of a timeline-based object.  They can be nested, can overlap, and can, in some cases, apply to multiple timeline-based objects.  Examples would include transcripts (working and the more detailed CA variety), task models (expert and student), reasoning traces, and computed LSA cosine values.  Analytic narratives in the form of research reports and external publications are also a form of annotation. The corpus will be designed to comply with NIH data sharing policy (NIH, 2003).  
Assessment rubrics for a testing case should reflect current best practices of diagnosis and treatment.  If best practices change during the course of the project, then the assessment rubrics must be modified to reflect this. This signifies that the same “case” may be scored in different ways in different years, but it does not invalidate the records in the data corpus provided we know which rubric was applied to each protocol when it was evaluated.  The SP checklist is in a sense self-documenting.  If the standards of practice change, the form may be changed, but the checklist form stored with the protocol will reflect the criteria used at the time that it was evaluated. Automated scorings of SOAP notes and global assessments of competency do not have this self-documenting property since they depend on resources external to the protocol.  For the SOAP notes, however, we can retain a record of the “gold standard” used to grade any particular note and, with regard to the global assessments, we will also preserve with each protocol an audio recording of the Clinical Evaluation Panel’s deliberations with respect to that protocol. 
The Analytic Frameworks.  The stated goal of the project is to develop appropriate analytic frameworks for studying the foundations of competency.  In the proposed pilot work we will examine three candidates: an analysis of reasoning patterns (Cognitive Protocol Analysis), an analysis of the interactional competencies that participants (i.e., subjects, SPs, interviewers) exhibit in their collaborative work of creating a case, and an analysis that blends the study of cognition and interaction while focusing on knowledge structures (Cognitive Discourse Analysis).  Each rests on different assumptions concerning what constitutes competency and how it is expressed.  In the proposed work we will apply the three analytic frameworks described below to the data from the “Shirley Naylor” and “Mary Watkins” cases.
Cognitive Protocol Analysis.  The extensive literature on diagnostic reasoning in medicine has centered on formal decision-making approaches (c.f. Arkes & Hammond, 1986).  While these approaches continue to address important issues in medical reasoning, they do not model the flow of cognitive processes occurring moment by moment during a patient encounter.  Other work has addressed cognitive processes in medicine by studying reasoning about complex diagnosis in presented cases (c.f. Patel, Arocha & Kaufman, 1994). A Cognitive Protocol Analysis, as described here, however, differs from these earlier approaches by examining problem solving within a processual framework. This analysis provides information about whether relevant domain knowledge is applied in specific circumstances deemed appropriate by experts. Pólya  (1957) postulated problem solving as four step process: Formulating the problem, devising a plan, executing the plan, and verifying the results. The behavioral evidence from a clinician’s handling of a case can be analyzed using this framework.
1) Problem formulation.  Classic and contemporary studies of problem solving identify the initial stage of problem formulation, where a problem description is interpreted, as key to successful solution (Kohler, 1969). A great deal of information, verbal and nonverbal, is presented in a patient encounter, and not all of it is relevant to diagnosis. In laboratory studies of word problem solving, subjects quickly identify the forty percent of the presented problem that is relevant to solution, and ignore the rest (Loman & Mayer, 1983). In addition, the source of information may appear irrelevant while actually quite informative, and vice versa.  For example, the tone of the small talk may reveal potential depression or even stroke in an elderly patient, while the patient’s response to questions about symptoms may be misleading (e.g., conventional responses to questions, such as “I’m doing okay.”) Brooks, LeBlanc, & Norman (2000) showed that even “textbook” examples of disorders are not consistently decoded as specific symptoms. Failures in initial problem formulation may result in perseverance on inappropriate solution paths (Kaplan & Simon, 1990). Are subjects able to identify the relevant features of a case, and does this ability correlate with eventual success in diagnosis? In order to determine whether a subject’s clinical performance is competent, we must see that the diagnostic problem is similarly understood. The results will provide cross-sectional data enabling a close study of developmental changes in the components of clinical expertise, knowledge and reasoning that were revealed using the models of experts for each clinical case.  An important piece to this analysis is the collaboration with the effort on interpersonal communication revealed in the video of the encounters.  Having the goal to ascertain knowledge is not adequate; instead, communication with the patient must occur in a way that elicits the information needed.  Cognitive goals cross over into communicative goals as questions are generated and answers understood.  The multidisciplinary approach will give us greater leverage in addressing whether an individual failed to acquire knowledge due to a cognitive or to a communicative difficulty.
2. Devising a plan. In an interactive case, new information gives rise to goals to acquire additional information; for example, excessive thirst may suggest diabetes, and this suggests looking for other signs of the disease. Given the presence of some features, do subjects ask the appropriate questions to collect related information, and to help them evaluate the evidence? People often stop seeking information well before they have enough to draw a clear conclusion (Wason, 1968). Wason (1968) also found that people had difficulty generating tests that could potentially show their reasoning is incorrect.  Studies of hypothesis testing in the laboratory have identified a confirmatory bias in reasoning (Nisbett & Ross, 1980), where people tend to seek out information that is consistent with their hypotheses rather than looking for evidence that might disconfirm them (Klayman & Ha, 1989). In medical reasoning, this step of looking for disconfirming evidence may be more salient because of the differential diagnosis task structure. This style of reasoning is well-rehearsed in medical training, and we expect to find physicians are better able to test their hypotheses while looking for both positive and negative evidence. Another finding in laboratory studies is the tendency to choose a solution plan quickly, and then fail to recognize evidence that the approach cannot succeed (Wason, 1968). In the natural setting, the physician may be led towards diagnoses based on the order of information presentation. Roter and Frankel (1992) found that patients began their report of problems with less significant complaints, leading up to major problems.  However, physicians had settled on their diagnoses within five minutes of the start of the interview.  So, it is particularly important for subjects to consider revising their beliefs about the case over time. The three-step interview will allow us to compare subjects’ responses when information is presented that is unexpected.  Will they accommodate the information, or reject it for the beliefs they have already established? 
3. Executing the plan.   Consistency and the “flow” of execution of the selected plan can be assessed by examining the sequencing and time of intended actions based on the plan, and the intrusions of actions not intended.  Work on cognitive skill acquisition has developed a well-tested model of how learners move from explicit, declarative knowledge to automatic, procedural execution (Fitts, 1964; Anderson, 1982). Procedural rules specify conditions to watch for in the clinical setting, and when matched, indicate specific actions to follow (Anderson, 1996). The procedural rules allow the expert to make rapid, efficient decisions based on access to past patterns of features. Rikers, Winkel, Loyens, & Schmidt (2003), for example, found that inexpert clinicians emphasized the same case components as experts, but lacked speed and accuracy in processing them.  Experienced clinicians, after years of practice, possess rich and diverse sets of well-encoded procedural rules. Physicians can make decisions using these highly-practiced patterns quickly and seamlessly and with minimal deliberate thought.  It will be possible to compare time stamps on actions as a general measure of the automaticity in executing procedures in the cases.  While prior research provides a set of hypotheses about skill execution, it is important to note that this will be the first study to examine these questions as clinical cases rather than written, non-interactive cases. 
4.  Verify the results.  A capacity for reasoning about existing knowledge and possible biases is required in order to learn to solve problems in complex domains.  Ideally, one is able to reflect on whether needed information is missing, whether the evidence matches conclusions, and whether biases have been influential.  If so, learners would have an advantage in attempting to correct for problems, and to “check” their work to avoid error.  However, people vary in how successfully they can detect these problems, and prior studies have found large differences in the accuracy with which people can characterize their reasoning success (Ericsson and Simon, 1984).  In the proposed work, reflections can be collected through comments in the encounter, along with final interview data.  This will allow us to compare the reasoner’s report of the problem solving experience with their actions at the time.  An ability to reflect on the quality of one’s solution process may be predictive of solution quality.  If so, training about what to notice could assist reasoners in detecting problems with their own solutions. Fragments of data, both from SP encounters and debriefing interviews, will be taken up in project team meetings to better understand the potential problems of reasoning evidenced in the data.
Work on expertise by Ericsson and Charness (1994) suggests expertise requires extensive “deliberate practice,” where one reflects on the aspects of performance that need improvement and practices those aspects in a thoughtful manner.  However, the nature of the cognitive processes involved in the type of practice that leads to expertise is unknown (Ericsson, Krample, Tesch-Romer, 1993).  The proposed work will contribute to the development of the theory of expertise by identifying the cognitive processes involved in metacognition about the problem solving process. Differences in reasoning will be analyzed in the transcripts of the post-encounter debriefing.  Though retrospective reports on cognition must be considered with caution (Ericsson and Simon, 1984), the four-component interview protocol will reveal how the subject dynamically responds to consistent and inconsistent information. It will also allow us to directly ask subjects about their reasoning, though the questions themselves will alter the reasoning observed.  This interview data will complement the straight observational data obtained through the videotaped patient encounter. This methodology allows the observation of the actions taken by the test subject, along with the questions asked and comments made during their patient encounter, supplemented by post interviews.
Analyzing the interactional competencies of case-building  As a designed simulation of an actual encounter with a patient and the patient’s problem, the SP encounter has the character of a ‘mock up.’  It is a mock up  “in the very way that it provides for an accurate representation of features in the actual situation, and in the very way it provides for an accurate representation of some relationships and some features in the observable situation, it also makes specifically and deliberately false provision for some of the essential features of that situation” (Garfinkel & Sacks, 1970, p. 363).  Various kinds of “interactional competency” (Psathas, 1990) can be located in the data record.  The test subject, the SP, and the debriefing interviewer all treat the ‘case’ as if it were an existent phenomenon, a real entity. Its existence, however, is a joint accomplishment brought about through and within their sequentially unfolding interaction.  How is this done?  Ethnomethodology (EM) is the branch of sociology that concerns itself centrally with questions of this sort.  It starts from the assumption that all social activity is ordered and asks, how is this order accomplished in actual produced concrete action?  Its objects of study are the methods of practical reasoning that members (i.e. participants within a social scene) employ in making sense of their own actions and those of others (Garfinkel, 1967, 2002; Heritage, 1984).  A number of specialized literatures have developed out of this central question.  The body of empirical work conducted as Conversation Analysis (CA) focuses on how basic aspects of talk (e.g., turn taking, selection of next speaker) are negotiated interactionally (Heritage, 1984; Sacks, 1995; Psathas, 1995).  Other research has focused on how science is produced within laboratory work (Lynch, 1993), how instruction is done (Hester & Francis, 2000), and how institutions are constituted through talk and action (Drew & Heritage, 1992; Luff, Hindmarsh, & Heath, 2000).  In the latter category, there has been considerable work focusing specifically on the organization of interaction between doctors and patients (e.g., Heath, 1986; Heritage & Maynard, forthcoming). This literature includes research on the organization of the medical interview (Heath, 1981), on the conversational work involved in taking a history (Cicourel, 1993), on performing a physical examination (Heath, 1986), and on aspects of delivering diagnoses and test results (Maynard, 2003; Pomerantz & Rintel, 2004).
Ethnomethodologically-informed research of this type relies upon a well-formulated set of assumptions and requirements (see Garfinkel’s [1967] policies; also Koschmann, Zemel, & Stahl [forthcoming]).  Because of the prominent role of talk in these recordings, Conversation Analysis (CA) methods will be employed in carrying out this research.  The primary focus of CA studies is the organization of conversational sequences; that is, how participants interact with one another. Participants in ordinary conversation use sequences to accomplish social activities: questioning/answering; making proposals/accepting or declining proposals; offering explanations/assessing explanations, etc. Medical interactions are also accomplished through doctors’ and patients’ use of these sequences. When used to analyze conversations between doctors and patients, CA can produce precise documentation of the conversational sequences through which the medical interview proceeds, such as the question-answer sequences in the history-taking portion of the medical encounter (Robinson, forthcoming; Heritage & Maynard, forthcoming; Beckman & Frankel, 1984).  CA research reveals not only the structure of these sequences, but also their interactional consequences and ramifications. For example, as mentioned earlier, relevant information can be lost when physicians ask questions that restrict the range of possible answers, or when physicians focus on patient’s initially-presented problem rather than allowing patients to express their full range of concerns. 
The analysis will be primarily based on the video recordings of the SP encounter and the debriefing interview. For the recordings of the SP encounters, the primary analytic question will be how does prior research on doctor/patient communication inform our understanding of the interaction between the test subject and the SP (and what new insights into medical communication are offered this data)?  For the debriefing interviews an alternative question will be pursued.  The analytic question here will be how do participants (the test subject and the interviewer) orient to and make relevant "the case" as a pre-existent reality?  The analysis of the SP encounters and the debriefing interviews will obviously be closely inter-connected, however, and will inform each other.
Cognitive discourse analysis.  Related to the investigation of the reasoning used in building a case, is inquiry into the representations used in constructing a patient case.  Cognitive discourse analysis (CDA) is one approach to studying the representations (Frederiksen, 2001). Research on discourse comprehension in the 70’s and 80’s established that understanding involves a process of building semantic representations within concrete situations of cognition and action (Frederiksen, 1975, 2001).  These cognitive representations are related to one’s long-term store of conceptual and relational knowledge.  Semantic representations and knowledge can be modeled in the form of semantic networks (i.e., conceptual graph structures). Natural language discourse represents such information in the form of linguistically encoded meanings (i.e., propositions) that a speaker or writer asserts in describing a situation or knowledge for a listener or reader (Frederiksen & Breuleux, 1990). By this theory, a listener constructs propositions from the linguistic expressions and uses them to infer the relevant cognitive representations of the situation or knowledge being described. CDA begins by constructing models of experts' procedural and declarative knowledge in particular domains of problem solving.  The propositional content of subjects’ discourse contributions can then be mapped with reference to the expert task model so constructed as a means of both capturing the emerging case structure and of diagnosing possible deficiencies in the subjects’ preparation or understanding.  This has been done in tutorial dialog (Frederiksen & Breuleux, 1990; Frederiksen, 1999), but has yet to be attempted in a clinical problem solving situation.  There are three phases to this form of analysis consisting of developing an “expert model” for each testing case and modeling the performance of individual students in particular case encounters. These will each be described in turn.
1) Creating an expert model.  The first requirement for performing a Cognitive Discourse Analysis (CDA) in this domain is to develop theoretical model of professional knowledge and clinical competency that includes models of experts’ case knowledge, case representations (causal models of cases), clinical reasoning, diagnostic problem solving, and interaction with patients. These “expert models” will be based on analysis of the SP encounters and the debriefing interviews with input from clinical faculty.  Data will be cataloged using the NVivo program (http://www.qsr.com.au).  The purpose for applying CDA methods to these data is to develop a complete model of clinical competency as exhibited in the SP encounter. The components of this model include models of: (a) procedures used, e.g., to take the patient history, conduct the differential diagnosis, develop the treatment plan; (b) explanation of the causes of patient symptoms for each diagnostic hypothesis; (c) reasoning using evidence (including lab test data) to select a particular diagnostic hypothesis; (d) the treatment plan and its forecast effects; (e) interaction with the patient, e.g., in obtaining the patient history, explaining the diagnosis, lab tests, treatment plan, and forecast effects; and (f) the SOAP Note.  Expert models will be developed for each case using CDA to analyze the propositional content of the discourse and actions (including explanations provided to the students) that occur in the patient encounter, the SOAP note, and in response to questions in the interview. Once the propositional content has been identified, semantic network models will be developed of the case models, and network models of the experts’ procedures, reasoning, and the treatment plan and its forecast effects will be constructed. An expert model was developed for the “Shirley Naylor” case (Frederiksen, Donin, Koschmann, & Kelson, 2004).
2) Modeling the performance of study subjects.  The second phase of this approach is to use these expert models to investigate professional knowledge and clinical competency of medical students and residents (including their case representations, reasoning, problem solving, and interaction processes with patients). The simulated patient (SP) methodology provides controlled situations in which to examine the development of clinical competency in medical students and residents. The results obtained for different groups of medical students and residents will be used to investigate changes associated with the progression from novice to more expert levels of competency. This component of the research also will help establish “benchmarks” or norms that could be used in assessing levels of clinical expertise. 
Analyses of the data from patient encounters (i.e., video and transcript of the patient encounter, the SOAP notes, transcript of the patient interview) will be facilitated using the expert models developed in the previous component. The student and resident data will be analyzed by comparing their performance and discourse to the expert models. In this way student models will be developed for each case based on the expert models for that case. Because of the volume of data that will have to be analyzed, scoring templates and procedures based on the expert models will have to be developed. Scoring systems will be developed for each case based on the expert models of the components of clinical competency identified above. These will be pilot tested with a small subset of subjects for each case, and the scoring models will be refined before applying them to the entire sample for each case. The student model developed for each student (for each case) will include: (a) a trace of the procedures applied in producing a differential diagnosis, (b) the subject’s causal explanation of the patient symptoms for each hypothesis considered, (c) the reasoning the subject uses to select a particular hypothesis, (d) the subject’s treatment plan and its forecast effects, (e) the subject’s interaction with the patient at each step of the diagnostic interview, and (f) analysis of the SOAP note in terms of how it reflects these aspects of the student model. One such student model has been constructed for one of the subjects in pilot testing (Frederiksen et al, 2004).
The primary forum for sharing findings across the three analytic perspectives will be the biannual project team meetings.  Here we will present analytic findings for critical evaluation by other members of the team.  The ultimate standard of adequacy for any analysis, however, is publishability in the literature (i.e., cognitive science, research on language and social interaction) relevant to the framework in question.  In addition to the presentation of analytic findings, we will conduct data sessions as a means of bringing diverse analytic perspectives to bear on brief fragments of previously unanalyzed data (Jordan & Henderson, 1995).  These sessions will be audio recorded and preserved for later reference.

Project Timeline.  

Year 1, 1st quarter:	Develop collection and use protocols (as described in “Human Subjects”) and submit them for approval to the local IRB. 
Year 1, 2nd quarter:	Complete design of the psychiatric testing case and train SP.  Conduct ‘dry-runs’ with clinical faculty.  Convene first project team meeting.
Year 1, 3rd quarter:	Recruit residents and medical students (as described in “Subject Selection”) for doing pilot testing with the new case.  Run subjects with the psychiatric test case.
Year 1, 4th quarter:	Convene Clinical Evaluation Panels to grade the SOAP notes and clinical encounters.  Conduct second project team meeting.
Year 2, 1st quarter:	Compete transcripts and DVD sets for the data records for the “Mary Watkins” case.  Distribute same to project team members.
Year 2, 2nd quarter:	Conduct third project team meeting.
Year 2, 4th quarter:	Finalize semantic space for LSA grading of SOAP notes.  Conduct final project team meeting.

Project Team Members and Their Responsibilities.  The PI has extensive prior experience working with large video corpora.  He will administer the project, participate in data collection, and assist in the construction of the database.  Responsibility for conducting the debriefing interviews will be shared by Nicole Roberts and the PI.  Linda Morrison directs the Professional Development Laboratory at SIU.  She will be responsible for all SP training. Reed Williams (Surgery) will serve as a consultant on medical evaluation methods. The following clinical faculty from the School of Medicine have been recruited to participate in this project: Drs. J. Albers (Family and Community Medicine), D. Resch (Medicine/Psychiatry), S. Soltys (Psychiatry), C. Todd (Internal Medicine), S. Hull (Medical Humanities), and T. Lower (Pediatrics).  These faculty will author new cases, participate in ‘dry-run’ testing of the cases, and serve on the evaluation panels that produce the global assessments of competency.  The LSA work will be done at the University of Memphis by Max Louwerse and colleagues.  The Cognitive Protocol Analysis work will be done by Colleen Seifert at the University of Michigan. The analysis of interactional competence will be jointly done by Timothy Koschmann and Richard Frankel.  The work on Cognitive Discourse Analysis will be done at McGill University under the direction of Carl Frederiksen.  Brian MacWhinney comes to the project with extensive experience managing shared data corpora.  He developed the CHILDES database for use by the language development researchers (MacWhinney, 1995) and was a PI for the NSF-funded Talk Bank Project (MacWhinney & Bird, 2000).  He will make recommendations with regard to data management and will participate in the analytic work of the project.

E. Human Subjects
Protection of Human Subjects.  An IRB protocol and set of consent forms were approved by the local board overseeing research involving human subjects in December 2002.  This protocol allowed us to do the initial pilot work.  It has now been terminated.  When funding is available for doing additional pilot work, a new protocol will be submitted to the IRB.  It will be based on a newly developed model for research involving large aggregated data collections including video of human subjects (video-based corpora).  Structured loosely on the protocol arrangements used in tissue banking studies, the model employs two types of protocols—collection and use protocols.
A collection protocol is used to build a collection or data corpus.  It has one or more associated informed consents forms (ICFs).  The collection protocol specifies how data will be collected, who will do the collecting, and the constraints on ultimate use of data as specified in the ICFs.  The collection protocol must be reviewed and approved by the local IRB and must be kept active for the full period of data collection.  (The corpus itself may continue to exist after the collection protocol has been terminated.) Use protocols are always dependent on a previously approved collection protocol.  Use protocols have no associated ICFs, but are bound by the terms of the ICF(s) of their parent protocol.  Like collection protocols, use protocols must be reviewed and approved by the local IRB of the institution that owns and maintains the data corpus.  An approved use protocol must be in place for the full period during which some responsible party has access to some or all data residing in the corpus for the purposes of carrying out some form of approved research.  The use protocol must specify what data will be accessed, who will have access, how access will be implemented and controlled, specific plans for publication, and plans for data destruction when the study is completed. 
In the Competency Project, there will be two approved consent forms, one for SPs and one for study subjects (they differ in terms of the description of the consenter’s involvement, description of benefits, etc.). Specific safeguards are built into the consent agreement to avoid creating recordings that could potentially embarrass a subject (or SP) or be damaging to their reputation.  If, for whatever reason, a participant does not wish to have a video record produced, the participant may demand to have recording discontinued at any point in the protocol.  Furthermore, subjects may request to have tape containing their visage destroyed anytime up to 72 hours following completion of the taping.  These consent forms were used for all pilot work done to date.  
Inclusion of Women, Minorities, and Children.  Our recruitment methods are explained in the section on subject selection in “Research Design and Methods.”  Subjects will be recruited without regard to gender or ethnicity using over sampling, if necessary, in order to ensure participation representative of the pools from which the subjects will be drawn.  Subjects will be recruited from three respective pools:  second-year medical students (MS2s), fourth-year medical students (MS4s), and residents and fellows (PG-1s, PG-2s, PG-3s) in the non-surgical residency programs based in Springfield, IL.  The Medical School complies with a non-discrimination policy with regard to admission to the undergraduate program and all residencies.  This policy statement is as follows:

Southern Illinois University School of Medicine does not discriminate on the basis of race, color, religion, age, sex, disability, or national origin in administration of its educational policies, scholarship and loan program or other School-administered programs.

This policy governs the constitution of the pools (medical students and residents) from which we will draw our subjects.  The actual gender and racial/ethnic composition of these programs are summarized on the following tables:

								
Population:  Class of 2008 Medical Students				
 	American Indian or Alaskan Native	Asian or Pacific Islander	Black, non-Hispanic	White, non-Hispanic	Hispanic	Total		
Female	1	3	3	33	2	42		
Male	0	1	4	27	1	33		
Total	1	4	7	60	3	75		

Population:  Class of 2009 Medical Students				
 	American Indian or Alaskan Native	Asian or Pacific Islander	Black, non-Hispanic	White, non-Hispanic	Hispanic	Total		
Female	3	4	9	24	1	41		
Male	0	4	2	26	2	34		
Total	3	8	11	50	3	75		

Population:  Non-Surgical Residents and Fellows 				
 	American Indian or Alaskan Native	Asian or Pacific Islander	Black, non-Hispanic	White, non-Hispanic	Hispanic	Total		
Female	0	33	2	26	3	64		
Male	0	42	0	30	4	76		
Total	0	75	2	56	7	140		

Because the nature of the research, there will be no children involved in this project. 

Concerns were raised by reviewers of an earlier version of this proposal with respect to using medical students in this research.  Medical students fall within a protected class of research subjects. To study how competency develops over time, however, it is necessary to evaluate competency at different stages of training, both pre- and post-graduation.  This will necessitate recruiting students from the medical school. Their involvement in this study will be extra-curricular and entirely voluntary, however.  Students are not obligated to participate in the study, will not be coerced to do so by faculty, and will participate strictly as paid volunteers.

F. Vertebrate Animals
Not applicable.

G. Literature Cited
1. Anderson J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369-406.
2. Anderson J. R. (1996) ACT: A simple theory of complex cognition. American Psychologist, 51, 355-365.
3. Arkes, H. R., & Hammond, K. R. (1986).  The scientific study of clinical judgment in psychology and medicine.  In B. Kleinmuntz (Ed.), Judgment and decision making: An interdisciplinary reader (pp. 551-567). Cambridge, U.K.: Cambridge University Press. 
4. Arocha, J. F., & Patel, V. (1995). Novice diagnostic reasoning in medicine: Accounting for evidence. Journal of the Learning Sciences, 4, 355-384.
5. Baeza-Yates, R., & Ribeiro-Neto, B. (Eds.) (1999). Modern information retrieval. New York: ACM Press.
6. Beckman, H., & Frankel, R. (1984). The effect of physician behavior on the collection of data. Annals of Internal Medicine, 101, 692-696.
7. Boshuizen, H. P., & Schmidt, H. G. (1992). On the role of biomedical knowledge in clinical reasoning by experts. Cognitive Science, 16, 153-184.
8. Brooks, L. R., LeBlanc, V. R., &  Norman, G. R.  (2000). On the difficulty of noticing obvious features in patient appearance. Psychological Science, 11(2), 112-117.
9. Cicourel, A. (1993). Hearing is not believing:  Language and the structure of belief in medical communication. In A. D. Todd & S. Fisher (Eds.), The social organization of doctor-patient communication (2nd Ed., pp. 49-66). Norwood, NJ: Ablex.
10. Drew, P., & Heritage, J. (Eds.). (1992). Talk at work: Interaction in institutional settings. New York: Cambridge University Press.
11. ECFMG (n.d.).  Step 2 CS—General information.  Retrieved February 2, 2006 from http://www.ecfmg.org/usmle/step2cs/index.html.
12. Elstein, A., Shulman, L. S., & Sprafka, S. A. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press.
13. Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49(8), 725-747.
14. Ericsson K. A., Krample,  R. T., & Tesch-Romer,  C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406.
15. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA, US: The MIT Press.
16. Fitts, P. M.  (1964). Perceptual motor skill learning. In A. W. Melton (Ed.), Categories of human learning. New York: Academic Press.
17. Frederiksen, C. (1975).  Representing logical and semantic structure of knowledge acquired from discourse.  Cognitive Psychology, 7, 371-458.
18. Frederiksen, C.H. (1999).  Learning to reason through discourse in a problem-based learning group.  Discourse Processes, 27, 135-160.
19. Frederiksen, C.H. (2001).  Propositional representation in cognitive psychology. In N.J. Smalser & P.B. Baltes (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Amsterdam: Elsevier Science.
20. Frederiksen, C.H. & Breuleux, A. (1990).  Monitoring cognitive processing in sematically complex domains.  In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 351-392).  Hillsdale, NJ: LEA.

21. Frederiksen, C., Donin, J., Koschmann, T., & Kelson, A. (2004, August). Investigating diagnostic problem solving in medicine through cognitive analysis of discourse in clinical situations. Fourteenth Annual Meeting of the Society for Text & Discourse, Chicago, IL.  Avail. at:
http://www.mcgill.ca/edu-acsrg/research/competence/
22. Garfinkel, H. (1967). Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall.
23. Garfinkel, H. (2002).  Ethnomethodology’s program: Working out Durkheim’s aphorism.  Lanham, MD: Rowman & Littlefield Publishers.
24. Garfinkel, H., & Sacks, H. (1970). On the formal structures of practical actions. In J. C. McKinney & E. Tiryakian (Eds.), Theoretical sociology: Perspectives and developments (pp. 337-366). New York: Appleton-Century-Crofts.
25. Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & the Tutoring Research Group. (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51. 
26. Heath, C. (1981). The opening sequence in doctor-patient interaction. In P. Atkinson & C. Heath (Eds.), Medical work:  Realities and routines (pp. 71-90). Westmead, England: Gower.
27. Heath, C. (1986). Body movement and speech in medical interaction. New York: Cambridge University Press.
28. Heritage, J. (1984).  Garfinkel and Ethnomethodology. Cambridge: Polity Press.
29. Heritage, J., & Maynard, D. (Eds.) (forthcoming). Practicing Medicine: Structure and Process in Primary Care Consultations. Cambridge, U.K.: Cambridge University Press.
30. Hester, S., & Francis, D. (Eds.). (2000). Local educational order: Ethnomethodological studies of knowledge in action. Philadelphia, PA: John Benjamins.
31. Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. Journal of the Learning Sciences, 4, 39-104.
32. Institute of Medicine (2000).  To err is human: Building a safer health system.  Washington, D.C.: National Academies Press.
33. Institute of Medicine (2001).  Crossing the Quality Chasm: A new health system for the 21st Century.  Washington, D.C.: National Academies Press.
34. Institute of Medicine (2003).  Health professions education: A bridge to quality.  Washington, D.C.: National Academies Press.
35. Kaplan, C. A., & Simon, H. A.  (1990).  In search of insight.  Cognitive Psychology, 22, 374-419. 
36. Kintsch, W. (1998).  Comprehension: A paradigm for cognition.  NY: Cambridge University Press.
37. Klayman, J., & Ha, Y. (1989).  Hypothesis testing in rule discovery: Strategy, structure, and content.  Journal of Experimental Psychology: Learning, Memory, & Cognition, 15, 596-604
38. Kohler, W.  (1969).  The task of Gestalt Psychology.  Princeton, N.J.: Princeton University Press. 
39. Koschmann, T., Zemel, A., & Stahl, G. (forthcoming). The video analyst’s manifesto (or The implications of Garfinkel’s policies for studying practice within design-based research). In R. Goldman, B. Barron, S. Derry & R. Pea (Eds.), Video research in the learning science. Mahwah, NJ: Lawrence Erlbaum Associates.
40. Landauer, T.K., & Dumais, S.T. (1997).  A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge.  Psychological Review, 104, 211-240.
41. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259-284.
42. Loman, N. L., & Mayer, R. E. (1983) Signaling techniques that increase the understandability of expository prose.  Journal of Educational Psychology; 75, 402-412 
43. Lord, C. G., Ross, L., & Lepper, M. R.  (1979).  Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence.  Journal of Personality and Social Psychology, 37. 2098-2109.
44. Louwerse M.M. (2004). Semantic variation in idiolect and sociolect: Corpus linguistic evidence from literary texts. Computers and the Humanities, 38, 207-221.
45. Louwerse M.M. (forthcoming). Iconicity in amodal symbolic representations.  In T. Landauer, D. McNamara, S. Dennis, & W. Kintsch (Eds.), LSA: A road to meaning.  Mahwah, NJ: Lawrence Erlbaum.
46. Louwerse, M.M., McCarthy, P.M., McNamara, D.S. & Graesser, A.C. (2004). Variation in language and cohesion across written and spoken registers. In K. Forbus, D. Gentner & T. Regier (Eds.), Proceedings of the twenty-sixth annual conference of the Cognitive Science Society (pp. 843-848). Mahwah, NJ: Erlbaum.
47. Luff, P., Hindmarsh, J., & Heath, C. (Eds.). (2000). Workplace studies: Recovering work practice and informing system design. New York: Cambridge University Press.
48. Lynch, M. (1993). Scientific practice and ordinary action: Ethnomethodology and social studies of science. New York: Cambridge University Press.
49.  Maynard, D. (2003). Bad news, good news: Conversational order in everyday talk and clinical settings. Chicago, IL: Univ. of Chicago Press.
50. Maynard, D., & Heritage, J. (2005). Conversation analysis, doctor-patient interaction and medical communication. Medical Education, 39, 428-435.
51. MacWhinney, B., & Bird, S. (2000).  Talk Bank: A multimodal database of communicative interaction.  Proposal to National Science Foundation.  Retrieved May 16, 2003 as:  http://www.talkbank.org/resources/proposal.pdf
52. MacWhinney, B. (1995).  The CHILDES project: Tools for analyzing talk (2nd Ed).  Hillsdale, NJ: Lawrence Erlbaum Associates.
53. NBME (2005).  2006 Step 2 Clinical Skills (CS) Content Description and General Information.  Philadelphia, PA: National Board of Medical Examiners.  Retrieved February 2, 2006 as: http://www.usmle.org/step2/Step2CS/Step2CS2006GI/2006Step2CS.pdf
54. Newble, D. et al. (1994).  Guidelines for assessing clinical competence.  Teaching and Learning in Medicine, 6, 213-220.
55. NIH (2003).  NIH data sharing policy and implementation guidance.  Bethesda, MD: Office of Extramural Research, National Institutes of Health.  Retrieved May 30, 2003 as: http://grants.nih.gov/grants/policy/data_sharing/data_sharing_guidance.htm
56. NIMH (1999).  Bridging science and service.  Bethesda, MD: National Advisory Mental Health Council Report.  Retrieved June 25, 2005 as: http://www.nimh.nih.gov/publicat/nimhbridge.pdf
57. Nisbett, R. E., & Ross, L. (1980).  Theory maintenance and theory change.  In R.E. Nisbett (Ed.), Human Inference: Strategies and Shortcomings in Social Judgment (pp. 167-192).  Englewood Cliffs, NJ: Prentice-Hall.  
58. Patel, V., & Arocha, J. F. (2001). The nature of constraints on collaborative decision making in health care settings. In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum.
59.Patel, V. L., Arocha, J. F., & Kaufman, D. R. (1994).  Diagnostic reasoning and medical expertise.The Psychology of Learning and Motivation, 31: 187-252. 
60. Patel, V., Evans, D. A., & Groen, G. J. (1989). Biomedical knowledge and clinical reasoning. In D. A. Evans & V. Patel (Eds.), Cognitive science in medicine: Biomedical modeling. (pp. 49-108). Cambridge, MA: MIT Press.
61. Patel, V., Kaufman, D. R., & Magder, S. A. (1996). The acqisition of medical expertise in complex dynamic environments. In A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports and games. (pp. 127-165). Mahwah, NJ: Lawrence Erlbaum.
62. Pólya, G. (1957). How to solve it. Princeton, NJ: Princeton University Press.
63. Pomerantz, A., & Rintel, E. S. (2004). Practices for reporting and responding to test results during medical consultations: Enacting the roles of paternalism and independent expertise. Discourse Studies, 6, 9-26.
64. Psathas, G. (1990). The organization of talk, gaze, and activity in a medical interview. In G. Psathas (Ed.), Interaction competence (pp. 205-230). Lanham, MD: University Press of America.
65. Psathas, G. (1995). Conversation analysis: The study of talk-in-interaction. Thousand Oaks, CA: Sage.
66. Rikers, R., Winkel, W. T., Loyens, S., & Schmidt, H.  (2003). Clinical case processing by medical experts and subexperts. Journal of Psychology: Interdisciplinary & Applied, 137(3), 213-223.
67. Robinson, J. (forthcoming). Soliciting patients' presenting concerns. To appear in J. Heritage and D. Maynard (Eds.) Practicing Medicine: Structure and Process in Primary Care Consultations. Cambridge: Cambridge University Press.
68. Roter, D. L., & Frankel, R. (1992). Quantitative and qualitative approaches to the evaluation of the medical dialogue. Social Science & Medicine, 34, 1097-1103.
69. Sacks, H. (1995). Lectures on conversation. Oxford, U.K.: Blackwell.
70. Turnbull, J., Turnbull, J., Jacob, P., Brown, J., Duplessis, M., & Rivest, J. (2005). Contextual considerations in summative competency examinations:  Relevance to the long case. Academic Medicine, 80, 1133-1137.
71. van der Vleuten, C.P.M. (1996).  The assessment of professional competence: Developments, research, and practical implications.  Advances in Health Sciences Education, 1, 41-67.
72. van der Vleuten, C.P.M. & Swanson, D. (1990).  Assessment of clinical skills with standardized patients: State of the art.  Teaching and Learning in Medicine, 2, 58-76.
73. Wason, P. C.  (1968).  "On the failure to eliminate hypotheses…" a second look.  In P. C. Wason & P. N. Johnson-Laird (Eds.), Thinking and reasoning.  Harmondsworth, Middlesex: Penguin.
74. Wass, V., & van der Vleuten, C. (2004). The metric of medical education: The long case.  Medical Education, 38, 1176-1180.

H. Consortium/Contractual Arrangements
Not applicable.
I. Consultants
Letters are attached from representatives at the following collaboration sites:

Richard Frankel, Ph.D. (Indiana University)
Carl Frederiksen, Ph.D. (McGill University)
Max Louwerse, Ph.D. (University of Memphis)
Brian MacWhinney, Ph.D. (Carnegie Mellon University)
Colleen Seifert, Ph.D. (University of Michigan)