The orthopedic surgery preparatory exam course: Measuring residents’ satisfaction and perceptions through video ethnomethodology and survey analysis

INTRODUCTION

Saudi orthopedic training in Saudi Arabia has changed the way board-eligible residents are assessed and evaluated. At present, the delivery of the board exam has shifted to using scenario cases instead of real patients during the assessment. There were some indications that this change was not received well by the residents. From our work with residents, we heard repeated verbal reports on their experience with the new method of using scenario examples in the exam. They reported their experience of the scenario cases as somewhat odd and generally preferred using the previously used real cases method, which they were trained on in medical school time. The residents also (verbally) expressed their unfamiliarity with the new setup and expressed a need to improve the delivery of scenario cases and a need to be oriented with this new approach, preferably through an unbiased structured course. Unfortunately, there are very few courses that are conducted to address orthopedic board exam preparation concerns specifically. One example of a course that is designed to address these issues is the Canadian Orthopedic Forum in Calgary, Canada. Another example is the course studied in the paper; the Orthopedic Surgery Preparatory Exam Course (OSPEC), which is currently conducted in Saudi Arabia.

It is known that assessment drives learning to a great extent.^[1] To improve teaching and learning, one option is the use of exams. The preparatory exam courses have proven to significantly improve exam scores by addressing exam concerns.^[2] A study by de Virgilis et al.^[3] has shown that reading assignments on a weekly basis conjoined with weekly preparatory examinations, improved overall American Board of Surgery In-Training Examination (ABSITE) scores. Godellas and Huang^[4] concluded that various factors participate to residents’ successful performance. These include the amount of study, previous performance in similar exams, probationary status, amount of sleep, and conference attendance. These factors improve the examination and knowledge application performance, both during residency and throughout their career. The OSPEC is considered a “previous performance” experience for students attending the course.

In a comprehensive review, Klein et al. concluded that the performance on the Orthopedic In-Training Examination (OITE) was proven to be highly projective of achieving well on the American Board of Orthopedic Surgery Part 1 Examination (ABOS1). The study examined the residents’ performance on OITE, and the results were similar to the ABOS1 that the residents had undertaken.^[5] The OSPEC reinforces the acquired knowledge while pinpointing the deficiencies found in the examinee. The course examined both knowledge and attitudes. It was a 3-day course during which knowledge content was delivered and assessed by a mock exam, and other skills were assessed as well. Derossis et al. concluded that the amount of dedicated study and conference attendance correlated significantly with resident performance and, thus, success rate.^[6]

Ethnomethodology research is the study of the everyday practice people engage in for the procreation of social order.^[7] Ethnomethodology’s research focuses on documenting the methodic practices through which society’s members produce their world as sensible. Unlike other forms of quantitative and qualitative research paradigms, ethnomethodology does not provide an exact “method.” There is no clear-cut set of procedures, although this approach involves numerous critical analytical processes, which have been used in other empirical studies.^[7] One confusing issue for newcomers to ethnomethodology is that it lacks a formally stated theory and a formal methodology.^[8] These issues never prevented ethnomethodologists from doing ethnomethodological studies and bringing important literature of “findings.”^[7] David Scott has noted the “off-stage role of theory,” in ethnomethodological writings, “the concern that there is nowhere in the ethnomethodological corpus a systematic theoretical statement that would serve as a touchstone for ethnomethodological inquiries.”^[8,9]

The current study aimed to investigate the perceptions of residents and their satisfaction with the OSPEC course and to analyze the strengths and weaknesses of this course.

MATERIALS AND METHODS

The OSPEC was delivered through presentations, focus groups, and video-recorded pre- and post-course mock exams with professional feedback. The residents who attended this course went through many educational session cycles. This included the simulated (mock) examination room for each participant. Throughout the session cycles, each participant’s focus was directed toward case-relevant knowledge and skills related to the examination, including the exam environment, tips, and the communication skills required while conducting mock exam cycles. Each resident had one mock exam on the 1^st day of the course (pre), then expert feedback on every subject’s performance. On the ^3rd day of the course, each resident had another mock exam (post) and received expert feedback (from communication skills experts and other mock examiners).

All course activities were video recorded as part of the course process. Recording the course events was an important part of the educational process for the course attendees. During the OSPEC course, every effort was made to simulate a real exam experience by providing an environment that is as close to the real exam environment as possible and a professional atmosphere to the course. The study was conducted in Jeddah, Saudi Arabia, after preparing a setup similar to actual board examination rooms.

The subjects selected in this study were board-eligible residents (i.e., the subjects have finished their training in the residency program). They were 5^th-year residents (R5) from all the regions of Saudi Arabia. This course was done in Jeddah over 3 full days, starting at 0800–1700 h. All of the residents who participated in this OSPEC course later sat the Saudi board exam after they attended the OSPEC course.

The subjects were aware of the study before it was conducted, and they were given an information sheet [Appendix 1] about the purpose of the study and why they were selected for it. In addition, the subjects were informed of their option to withdraw from, or not participate in the study. After the course, they were consented to and invited to fill up the web-based questionnaire [Appendix 2]. This was sent to the subjects’ e-mails after the date of the actual board exam.

RESULTS

DISCUSSION

The study investigated the orthopedic residents’ perception and satisfaction with the OSPEC course as a structured study plan prior to the exam. Durak and his associates have found that instructive case-based exams followed by case discussions seemed to be a high-potential and motivating teaching tool for 6^th-year students in the clinical problem-solving domain.^[1] Within the OSPEC, mock exams are conducted, followed by expert discussion and feedback.

Miyamoto et al.^[13] showed that several factors, including structured study habits and the use of specific study materials, contribute to residents’ successful OITE performance. They concluded that the adaptation of these findings by current orthopedic residents might positively impact OITE performance. Furthermore, Barnes concluded in her study that the oral examination represents an additional tool that may contribute to developing a student’s connection between knowledge, effective communication, and pharmacy practice.^[14]

Booth TC et al. concluded^[15] that “Efforts to help reduce anxiety levels in the oral component would improve perceived fairness.” One of the main aims of the OSPEC is to reduce the candidates’ anxiety levels, which was addressed in the study.

As Pletch et al. highlighted the importance of having a structured study plan, as residents might have different studying attributes, which might hinder their preparation for the ABSITE.^[16] These findings are in accordance with a study published in 2013, which found that residents who emphasized self-assessment examination before OITE as a preparation tool had high scores, unlike their peers.^[14] De Virgilio and Stabile concluded that assigning residents weekly reading assignments and assessments improved the mean ABSITE test scores compared to the previous years.^[3] Using practical techniques in medical education like digital transformation, virtual reality, augmented reality, mixed reality, and extended reality, and three-dimensional medical images would maximize the effectiveness of patient care in the field of surgery while maintaining the educational needs of residents.^[17-19]

The OSPEC seemed to be of benefit, as verbal reports from orthopedics residents were repeatedly received each time after the OSPEC course was conducted. They generally portrayed good feelings and positive perceptions about the course. They voiced overall satisfaction with the course content, its procedures and how it positively affected their performance during their real examination afterward. However, these perceptions have not been documented through previous empirical research. Such feedback and our experience with the OSPEC course over 3 years gives some indication that such a course can positively impact resident training and their confidence; and warrants further research.

Most participants shared their gratitude for integrating the course as part of their training in orthopedics. The results showed that the OSPEC course was more of a constructed plan in preparing for the board exam. The evidence also showed that the critique repeatedly received following the course was of great help in utilizing and enhancing the knowledge and skills they acquired while addressing the deficiencies they might have exhibited during the course.

CONCLUSION

The participants generally perceived the course to be useful in helping them to prepare for their board exam by familiarizing them with the exam environment. They were generally satisfied with the course. The participants reported the strength of the course as being able to provide a realistic simulated experience. On the other hand, the participants viewed the weakness of the course in its duration and generally requested more days to be added to the course.

AUTHORS’ CONTRIBUTIONS

OB and TJ conceived and designed the study. OB, TJ, and AO conducted research, provided research materials, collected and organized data, and wrote the initial draft. MA and TJ wrote the final draft of the article and provided logistic support. All authors have critically reviewed and approved the final draft and are responsible for the manuscript’s content and similarity index.