Simulation-based education (SBE) supports early-stage psychiatric doctors to bridge educational and clinical practice while encouraging reflective practice. Research comparing the efficacy of in-person and online mental health SBE is limited. In a large-scale comparison study, we assessed for significant course evaluation measure differences between in-person and online participants attending an SBE course for early-stage psychiatric doctors.
A full-day in-person course was adapted for online delivery over a half-day. It focused on developing confidence and clinical skills relevant to early-stage psychiatric doctors. In-person (n = 228) and online (n = 90) participants were early-stage psychiatric doctors based in South London mental health trusts. Pre- and post-course quantitative data using the Human Factors Skills for Healthcare Instrument (HuFSHI) and Course Specific Questions (CSQ) measures were compared across the two delivery formats. Data collected from previous in-person deliveries were compared with online delivery data.
Paired-samples t-tests comparing pre- and post-course HuFSHI and CSQ scores indicated significant improvements across both measures for the two delivery formats. Large and very large effect sizes, respectively, were observed for HuFSHI and CSQ scores in both delivery formats. Participants reported greater benefits from in-person delivery across CSQ measures and from digital delivery across HuFSHI measures. Independent-samples t-tests used to assess for significant differences between online and in-person delivery formats for HuFSHI and SCQ scores indicated no significant differences in scores favouring either in-person or online delivery.
The data suggest online mental health SBE potentially represents an effective adjunct or alternative to in-person delivery. Further research is required to better understand these differences to support learners, educators, and commissioners.
What this study adds
Simulation-based education (SBE) supports early-stage psychiatric doctors to bridge the gap between educational and clinical practice. It enables this through exposure to a variety of clinical presentations and a safe space to hone communication and de-escalation techniques, whilst also encouraging reflective practice [1,2]. Achieving key learning objectives during the initial stages of psychiatric training presents inherent challenges. Opportunities for learning can be limited to patient crises or when interacting with patients with severe mental illnesses without close supervision by senior colleagues (such as out-of-hours). Moreover, these patients may be made anxious by early-stage trainee doctors [3].
COVID-19 required simulation faculties to provide online alternatives to in-person training. Online SBE offers psychiatric trainees the opportunity to continue to access high-fidelity experiential learning irrespective of location or shielding status. In this paper, we define high-fidelity SBE as the development and deployment of immersive scenarios that exclusively employ professional actors and offer a high degree of physical, environmental and psychological realism for our participants. Beyond the pandemic context, online SBE ostensibly also presents several further advantages in terms of equity of access for trainees in remote areas or smaller training schemes, in addition to potentially reduced costs for providers.
Across medical sub-specialties, however, there is a heterogeneous evidence base in terms of the comparative efficacy of online SBE compared to in-person training. Comparisons of both modalities with medical student participants have shown higher knowledge and self-confidence scores for in-person SBE; however, both in-person and online SBE were emphasized as effective methods of course delivery [4–6]. The effectiveness of both modalities has been demonstrated in nursing training and both have been deemed suitable and effective options for delivering simulation to off-site participants when considering social distancing, and for assessment, including virtual objective structured clinical examinations [7–9].
With respect to psychiatric training, there are limited large-scale research studies comparing the efficacy of online and in-person SBE. Despite the many recent innovations in online learning, educators have become increasingly aware of some potential negative aspects of online delivery. A prominent example highlighted in the recent literature is the inherently fatiguing effect of prolonged videoconferencing (VC) [10,11]. As universities, training bodies, and healthcare organizations seek to develop their simulation capacity, further research comparing and evaluating online delivery is necessary in terms of determining its strengths, most appropriate applications, and indeed, in identifying and understanding the nature of any such consistent limitations. This evidence will be vital in optimizing course design. It will also support commissioners and training bodies to appropriately integrate online SBE into their training such that learners are afforded high-quality and educationally effective experiential learning.
During the pandemic, our organization pivoted fully to large-scale online delivery of mental health SBE: defined as simulation training delivered entirely via a VC platform to a participant group remote from one another and the simulation team. As such, we are well-positioned to evaluate and elucidate its strengths and limitations which has now become a pressing need for learners, educators and commissioners. By comparing human factors skills self-efficacy in addition to course-specific measures, we aimed to assess for significant differences across a large sample of early-stage psychiatric doctors attending in-person and online versions of an SBE course.
Based in South London, our organization has several years’ experience of training early-stage psychiatric doctors through its regular delivery of Practicing Psychiatric Competencies 1.0 (PPC 1.0), an in-person simulation course focused on developing confidence and skills in psychiatric history-taking; mental state examination; and risk assessment and formulation; meeting the relevant learning outcomes of the Royal College of Psychiatrists [12].
Scenarios for PPC 1.0 were originally developed by our faculty team comprised of nurses, psychologists and psychiatrists with experience across a range of sub-specialties within mental healthcare. Our simulation faculty are trained in simulation scenario development and design courses in collaboration with professional actors. In scenario development, we relied on Kolb’s experiential learning theory as the conceptual framework [13]. This theory allowed our faculty to develop scenarios to maximize participants’ learning capabilities through the simulation experience. We worked with professional actors from a specialist medical role-play agency, trained for participating as patients in mental health simulations. Our SBE practice relies primarily on a communication processes-focused debrief. The multidisciplinary faculty undergo dedicated debrief training and peer debrief reflective practice sessions. In addition to comprehensive experience in the delivery of high-fidelity SBE, our team has extensive experience in its evaluation [14–17]. The existing full-day PPC 1.0 in-person course was adapted for online delivery over a half-day.
Regardless of delivery format, the PPC 1.0 course followed the same structure and principles. Within each course, the simulation scenarios were preceded by an introductory session. This encompassed ice-breaker sessions to engage participants, information on key concepts related to SBE such as psychological safety, and operational and scheduling plans for the session.
The scenarios introduced participants to common mental health conditions and with a focus on the de-escalation of acutely agitated patients in different clinical contexts (see Table 1). Each scenario was followed by a structured debrief using a modified version of Pendleton’s model [18]. The simulation training day finished with a summary of the learning, and an opportunity for feedback and reflection on the course.
| Summary | Selected learning objectives to guide debrief | |
|---|---|---|
| 1 | Setting: Home visit (in-person version) or remote consultation (online). Context: A 36-year-old junior doctor has been referred by her GP. A health visitor doing a recent 6-week check on her new baby encouraged her to seek help for low mood. She is showing signs of postnatal depression. Task: The participant must begin to take a relevant history. |
• To demonstrate effective skills in managing difficult dynamics with relatives in consultations. • To discuss the differences between risk factors for harm to self in the perinatal period, versus the non-perinatal period. • To analyse how certain personal attributes, e.g. health professional as a patient, can impact the dynamics of a consultation. |
| 2 | Setting: Inpatient psychiatric unit. Context: A 29-year-old woman was admitted overnight following a crisis. She is now requesting to leave. Task: The participant must attempt to come to an understanding of her presentation and risk and begin to collaborate with her on a management plan. |
• To outline and define a framework for risk assessment with personality disorders. • To review and demonstrate skills in managing a personality disorder, including the maintenance of both boundaries and empathy. • To evaluate the impact of stigma in mental health. |
| 3 | Setting: Inpatient psychiatric unit. Context: A 32-year-old man has recently returned from unescorted leave in an agitated state. Task: The participant must assess his mental state and demonstrate appropriate verbal de-escalation skills and situational awareness. |
• To review protocols and demonstrate both technical and non-technical skills in the assessment and management of violence and aggression in an inpatient psychiatric setting, including the management of personal safety. • To demonstrate the ability to work effectively with colleagues, including team working. • To recognize and identify the rapid tranquillization protocol. |
Both delivery modalities featured two trained facilitators, one of whom was a higher psychiatric trainee. At least one simulation technician assisted in all deliveries regardless of format. They provided operational, moderator (online format) and general technical support. Different faculty members led this training throughout the year based on the organization’s yearly faculty rotation schedule.
The in-person course was delivered at our simulation centre in South London. The original full-day in-person course comprised 12 high-fidelity live 10-minute simulation scenarios. The active participants entered an immersive environment. Live footage of this scene was broadcast to a debrief room containing the non-active participants and simulation faculty.
Zoom was our VC platform of choice [19]. Each online session was preceded by a half-hour platform orientation session delivered by our technicians who supported the delivery throughout. Actors, participants and faculty were fully remote and relied on in-built webcams and microphones to participate. Participants accessed the course primarily via personal or work computers and a minority used tablets. The half-day online course comprised of seven simulation scenarios (five live and two video-based scenarios) of 10 minutes each. Non-active participants observed the scenarios with their cameras off, to limit distraction for the active participant.
The PPC 1.0 course content was the same in both delivery formats, except for the number of scenarios. In both delivery formats, each scenario was followed by a 15-minute structured debrief that encouraged participants to explore consultation dynamics, with a strong emphasis on communication and human factors skills.
The maximum number of participants for each PPC 1.0 course delivery was 12, in both in-person and online delivery. In-person (n = 228) and online (n = 90) participants comprised of early-stage psychiatric doctors (at either core psychiatric trainee, or GP trainee level) based in mental health trusts in South London (i.e. South London and Maudsley NHS Foundation Trust, and South West London and St George’s Mental Health NHS Trust). In terms of recruitment, this course was part of participants’ mandatory induction for psychiatric placements. In total, 318 junior doctors attended the PPC 1.0 course in-person between September 2018 and December 2019, and online between February 2020 and April 2021.
Ethical approval for this project was provided by the Psychiatry, Nursing and Midwifery Research Ethics Subcommittee at King’s College London on behalf of the Health Research Authority (ref. PNM1314173).
Quantitative pre- and post-course human factors skills for healthcare [20] (HuFSHI) and course-specific questions (CSQs) self-report survey measures were collected from participant groups across both delivery formats, i.e. in-person and online.
The HuFSHI is a validated 12-item tool for assessing interprofessional learning across healthcare practice settings. It was developed amongst a group of healthcare professionals working in both acute and community care settings and attending acute care and mental health simulation training courses in London. It has been demonstrated to be a reliable and valid method of assessing trainees’ human factors skills self-efficacy across acute and mental health settings with good internal consistency and sensitivity to change. We deemed this scale to be applicable to our training and participant group given the overlap in its development and delivery contexts.
CSQs were developed internally by the simulation faculty to assess the course-specific learning objectives. This included 14 items designed to measure participants’ confidence, skills and knowledge about the course content on a 10-point scale ranging from totally disagree to totally agree (see Table 2 for examples).
| No. | Item |
|---|---|
| 1. | I am able to assess a patient’s risk of harm to themselves and others. |
| 2. | I am able to manage risk in my surroundings when working with patients with mental illness. |
| 3. | I can collaborate effectively with multidisciplinary colleagues to support people experiencing mental illness. |
| 4. | I can demonstrate effective communication with patients with severe psychiatric presentations. |
| 5. | I feel confident in de-escalating a situation that might involve violence and aggression. |
Previously collected in-person delivery data alongside newly collected online delivery data were used to compare the new course format. Quantitative measures were completed immediately before and after both course formats. In-person data were collected using tablets distributed by simulation technicians, and online data were collected via an online survey platform.
Completed response rates were greater than 90% for both modalities, except for CSQs for in-person deliveries which was relatively low. This may partly have been due to data loss and difficulties in completing this measure using tablets distributed during in-person training.
Statistical analyses were conducted using SPSS [21]. The pre- and post-data were screened using Mahalanobis Distance [22] to identify participant outliers which indicated there were four degrees of freedom, equating to a chi-square value of 1.83 (p = 0.127). Cases (n = 3) with a distance score exceeding this value were excluded. The final sample size for further statistical analysis was 315 participants. Levene’s test confirmed that there was no violation of the assumption of homogeneity for HuFSHI pre-course scores, F(1, 307) = 0.66, p = 0.417, HuFSHI post-course scores, F(1, 304) = 0.62, p = 0.430, CSQ pre-course scores, F(1, 139) = 1.68, p = 0.197 and CSQ post-course scores, F(1, 145) = 1.36, p = 0.245.
Paired-samples t-tests were conducted to assess for significant changes in HuFSHI and CSQ scores pre- and post-course, based on the course delivery format. Independent-samples t-tests were conducted to assess for significant differences between online and in-person delivery formats for HuFSHI and SCQ scores.
For the HuFSHI scores, there was a significant improvement between the pre-course scores (M = 78.33, SD = 13.18) and post-course scores (M = 89.30, SD = 14.55), t(218) = 12.44, p < 0.001. Likewise, for CSQ scores, a significant improvement was observed between the pre-course scores (M = 91.62, SD = 15.45) and post-course scores (M = 119.96, SD = 14.46), t(54) = 16.63, p < 0.001 for in-person delivery.
For the HuFSHI scores, there was a significant improvement between the pre-course scores (M = 76.72, SD = 12.70) and post-course scores (M = 89.40, SD = 13.48), t(80) = 9.08, p < 0.001. Likewise, for CSQ scores, a significant improvement was observed between the pre-course scores (M = 90.12, SD = 14.22) and post-course scores (M = 115.71, SD = 17.12), t(84) = 16.28, p < 0.001 for online delivery.
A large effect size was observed for the HuFSHI scores in both online (d = 0.96) and in-person delivery (d = 0.79). A very large effect size was observed for the CSQ scores in both online (d = 1.62) and in-person delivery (d = 1.90).
For the HuFSHI scores, there was no significant difference in scores between online and in-person delivery modalities, t(298) = 1.02, p = 0.851, 95% CI = [−1.59, 5.02]. However, participants scored slightly higher in the online delivery (M = 12.58, SD = 12.57) compared to in-person delivery (M = 10.96, SD = 13.05).
Similarly, for the CSQ scores, there was no significant difference in scores between online and in-person delivery modalities, t(138) = 1.15, p = 0.263, 95% CI = [−7.48, 1.96]. However, participants scored slightly higher in the in-person delivery (M = 28.35, SD = 12.64) compared to online delivery (M = 25.59, SD = 14.49).
With respect to a comparison between in-person and online SBE in this context, what remains unclear is whether one is superior in relation to reported self-efficacy in human factors skills and meeting specific course learning objectives. Higher means were observed across both measures (HuFSHI and CSQ) in addition to larger effect sizes across formats in favour of in-person delivery for the CSQ, and in favour of online delivery for HuFSHI. Large and very large effect sizes, respectively, were observed for HuFSHI and CSQ scores in both delivery formats.
Given these results, our data suggest both participant groups benefitted from the training in terms of self-efficacy in human factors skills and meeting course-specific learning objectives. Interestingly, these data suggested the benefit was greater in terms of course-specific learning objectives for in-person participants, and greater in terms of self-efficacy in human factors skills for online participants. However, we found no evidence that participants found one delivery modality to be significantly better than the other.
Previous research supports our findings, with studies reporting high participant satisfaction rates after attending a remote simulation training programme [23], and significantly increased subject knowledge post-online simulation training [24]. A study comparing the outcomes of switching from in-person to online delivery of an educational course (combining lectures and simulation-based training) found similar results to ours, wherein there was no clear preference in favour of either in-person or online simulation training [25]. Rather, learners identified merits to both training modalities and showed a preference towards one or the other modality based on specific aspects of the course. Our finding of no significant differences in post-course scores between the two delivery modalities further strengthens support for the hypothesis that online delivery may represent a viable adjunct to in-person delivery.
It is noteworthy that whilst significant efforts were made in terms of technical orientation, course design and the use of breaks, the potential latent effects of VC fatigue may have impacted the participants’ overall comfort and ability to concentrate over the online sessions’ duration [26]. The inherent constraints of the VC platform limit interactivity, fluency and engagement [27], and in turn, may account for the higher means and larger effect sizes for in-person delivery in terms of the CSQ measure to some unknown degree. In view of the growing understanding in this area, there may be a need for a reliable measure of perceptions of VC fatigue for online training.
In terms of strengths and limitations for each delivery modality, one obvious difference and, indeed advantage, of online simulation is that it offers the active observer a far richer sensory experience. During in-person deliveries, observers typically collectively view the live scenario on a television screen in a separate debrief room. By contrast, during online SBE, participants are far more immersed and benefit from improved visual and audio inputs with enhanced views of facial expressions and body language, for example, through their individual devices. We remain cautious, however, not to extrapolate beyond our data in terms of what factors may be facilitating higher scores in the online format. Additionally, online SBE offers extremely high-fidelity learning opportunities with respect to virtual consultations. An obvious limitation of online simulation is that debriefing can prove demanding and debriefers experience a high cognitive load online, with potentially adverse implications for performance and learner outcomes [28]. In-person delivery offers its own unique advantages in that it affords participants the opportunity to meet and interact without the inherent constraints of VC platforms. This takes place at a specialist centre which represents an ‘away-day’ for staff removed from their usual working environment. As a result, participants spend the break periods together rather than alone as with online delivery. The nature of in-person delivery renders inter-participant engagement within the debriefing phase more fluent. We argue that this potentially enhances their cohesion as a learner group and adds to psychological safety. However, in-person deliveries may limit access to those that are geographically remote or have disabilities, in a way that is easily overcome through the online modality.
The COVID-19 pandemic has fast-tracked digital health innovations and heralded the widespread adoption of virtual consultations. The National Health Service (NHS) Long Term Plan [29] includes the mainstream adoption of digitally enabled care, which online SBE supports by uniquely offering staff the ability to train in remote consultations with high fidelity. There are numerous positive implications for learners such as increased access to training and greater flexibility. For example, healthcare professionals in rural regions would ordinarily have to take prolonged time away from clinical practice to attend training in a larger urban area. Another benefit of increased accessibility for trainees is the ability to meet more specialized learning objectives, e.g. perinatal mental health, by accessing expertise remotely. The UK postgraduate training system is known worldwide, attracting doctors from all over the world. Through online SBE, there is scope for international medical graduates to gain experience and build confidence by familiarizing themselves with the NHS prior to arrival, which would support the recruitment and retention of healthcare professionals as set out in numerous government policy documents [29,30].
This study benefited from the use of a human factors instrument validated for the context in which we worked and with a uniquely large sample size with the final sample including pre- and post-course data from 315 participants. To the authors’ knowledge, there are no known comparison studies using such large samples. Data were collected systematically across both measures before and after each delivery and maintained on a database. Data were collected from each cohort, i.e. in-person and online, over periods of 15 and 14 months, respectively: this spanned several training in-takes. As several different faculty members delivered this training, this relatively prolonged data collection window mitigated any biases that otherwise may have been more prominent in relation to the facilitation styles and patterns of engagement associated with individual staff members. It is our view that this variation in facilitators over time, as such, represents a key strength of this study, in addition to its overall considerable scale.
Additionally, our debriefers delivered weekly online simulation from July 2020 onwards. As such, they were able to quickly develop confidence and fluency in using this modality. In this new operational context, we continued to pay extremely close attention to the development, maintenance and protection of a high degree of group psychological safety [31]. We plan to undertake further research to better understand the various challenges faced by debriefers delivering online SBE.
Both the HuFSHI and CSQ are self-report measures which limits their objectivity. Increased scores signify increased human factors self-efficacy and subject matter competence, respectively, by self-appraisal, as opposed to through an objective assessment. It is also plausible that the Dunning–Kruger effect [32] was at play, biasing our relatively early-stage participant group to overestimate their capabilities and give themselves higher scores in the post-course survey. Additionally, the CSQs are not a validated measure, and individual items were specific to the course content. Low response rates for CSQs noted on the in-person deliveries may have been due to data loss and difficulties in completing this measure using tablets distributed during the sessions. Lastly, this study relied solely on quantitative data. It may be the case that supplementing this with qualitative data could yield a richer understanding of the differences between the chosen modalities.
Further research is required to better understand the differences between these modalities. Specifically, it will be important to understand what was driving higher HuFSHI scores in online delivery and if this effect persists across multiple studies. Whilst online SBE may offer scope for improved self-efficacy in human factors skills, there are likely to be limitations in terms of practical skills acquisition. The different development processes of group cohesion online represent another key area of future research for online SBE, and this has relevance for interactive experiential online learning more broadly. We also recognize that the incorporation of qualitative data into future studies should be considered by researchers.
Our understanding of the educational differences between in-person and online mental health SBE remains at an early stage. Using a relatively large data set for the field, our self-report data suggest that online mental health SBE potentially represents an effective adjunct, or even alternative, to in-person delivery. For psychiatric training schemes covering large geographical areas – or where funding or resources are limited – online SBE may present an attractive option. To our knowledge, this is the first comparison of delivery formats for mental health SBE for psychiatric trainees to be made on so large a scale. Further evidence is needed, but it is anticipated that this will be an interesting area of educational innovation research given the wider shift to both hybrid higher education and, indeed, workplaces.
None declared.
DB and OPO’S (joint first authors) conceived the project and analysed the data, in addition to leading on manuscript drafting. These stages were supported by HI. OPO’S, NT, JP, AB and SP were involved in project design, delivery and data acquisition. HI was responsible for project design and final approval of the manuscript.
This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.
None declared.
Ethics approval: PNM1314173.
The authors have no conflicts of interest to declare.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
Improved self-efficacy in human factors skills in early-stage psychiatric trainees following online simulation: a quantitative comparison study with in-person training
Twitter
Facebook
WhatsApp
LinkedIn
Tweets