Recently, simulation-based education (SBE) has been evidenced as an effective form of pedagogy in mental health and care settings, through consistent improvements in self-efficacy and technical and non-technical skills. A key component of SBE is post-simulation debriefing. Debriefing involves educators turning into facilitators and guiding participants through reflective discussions; however, there is no single debrief model used across simulation training. Debrief models have been previously evaluated, but not directly compared. This paper investigated whether there is a significant difference between self-efficacy scores of participants debriefed using the Diamond model and a modified Pendleton’s during SBE.
Participants included 751 healthcare professionals who attended various simulation training courses between September 2017 and August 2019. Participants completed pre- and post-course questionnaires using the Human Factors Skills for Healthcare Instrument. Pre- and post-data were screened using Mahalanobis distance and Levene’s test and data were analysed using paired-samples t-tests.
Significant differences in human factors scores were found for the Diamond debrief model only. No significant improvements were found for the Pendleton’s model.
Results suggested a benefit to using the Diamond model over the Pendleton’s model during simulation debriefs, due to a significant improvement in self-efficacy scores. These findings contribute to the gap in literature around direct comparison of debrief models and support studies where the Diamond model has yielded significant improvements in human factors skills previously.
What this study adds
• This research contributes to the gap in literature around comparing different debrief models within simulation training to analyse impact on participants’ self-efficacy.
• It provides support for simulation training as an effective teaching method to improve healthcare professionals’ technical and non-technical skill sets.
• It lays the foundations for further longitudinal or explanatory research to unpack why the Diamond debrief model is more effective than other models.
In recent decades, our understanding of mental health needs for the general population has developed significantly, particularly regarding the overlap between mental and physical health [1]. The events of the COVID-19 pandemic, for example, further emphasize the ever-changing nature of mental healthcare requirements [2]. Pre-pandemic evidence highlighted that people with mental health needs not only receive poorer medical care, but also experience the most stigma from healthcare staff [3], in turn also implicating subsequent treatment [4]. Post-pandemic research has further highlighted the urgence for implementing more effective and preventative mental health services [5]. Ultimately, there is a call for high-quality, informed and updated mental health training for all healthcare professionals.
Recently, simulation-based education (SBE) has emerged as a powerful tool facilitating experiential learning and the acquisition of both clinical and reflective skills, generating various positive outcomes in healthcare [6,7]. In a mental healthcare context, SBE involves equipping clinical and non-clinical professionals with experience in the form of simulated scenarios with ‘patients’, where a skill set in mental healthcare should be used. This experience is reflected upon during debrief, allowing participants to acknowledge both the gaps and strengths in their current skills. While research on the application of SBE has often focused on medical aspects of healthcare training, recently, studies have provided strong support for SBE in mental health training and care settings. These have demonstrated consistent and promising results in the improvements of both technical and non-technical skills, as well as self-efficacy, in a variety of contexts [8–10].
Referred to as one’s belief in their ability to perform activities and tasks successfully [11], self-efficacy is an important indicator that trainees have fully integrated knowledge into their practice. It is documented that self-efficacy is an essential predictor of learning and knowledge conversion [12]. In particular, SBE plays an important role in addressing healthcare workers’ self-perception, self-report, well-being and confidence in their clinical skills, with research showing improvements in self-efficacy as a result [13,14]. While self-efficacy is not a direct measure of clinical outcomes, it is a useful tool in gaging knowledge consolidation and perception across healthcare practice settings. As self-efficacy has been found to be closely correlated with improved performance, in both academic and work-related contexts [15,16], it acts as a good proxy for clinical skill, where we cannot directly measure this.
Moreover, self-efficacy has been noted as a key component in major behaviour change models [11,17]. For example, the Theory of Planned Behaviour suggests that a person’s behaviour is determined by their intentions, which are developed around a combination of their attitude, subjective norms and their overall ‘perceived behavioural control’ [17]. This is defined in terms of their feelings of self-efficacy around performing certain tasks. As such, improvements in self-efficacy impact a person’s behavioural intentions and subsequent behavioural change. Social cognitive theory posits that self-efficacy is a key motivational force underlying behaviour change [11]. Self-efficacy acts as a personal factor, which when combined with environmental factors such as resources, opportunities and social support, can facilitate overall behavioural change. It is especially significant that self-efficacy is believed to have situation specificity, rather than being a general factor; hence, it is useful to use it as a measure across different environments and skills. By noting improvements in self-efficacy specifically in non-technical skills, SBE can be viewed as promoting positive outcomes on healthcare workers’ development needs, evoking positive behavioural change, and an effective environment to improve clinical skills.
Key to effective healthcare and clinical proficiency is the cognitive and social skills required to manage the demands of different clinical and high-pressure situations and work collaboratively within these. Such skills include decision-making, leadership, situational awareness and teamwork, often referred to as non-technical skills. It has been argued, however, that these skills termed as strictly ‘non-technical’ differ across different healthcare sectors and disciplines. Communication, for example, particularly in mental healthcare settings, is a mediator for diagnosis and treatment, suggesting its role as both a technical and non-technical skill. As such, Reedy et al [16] use the term ‘human factors skills for healthcare’ to describe and encompass the range of key social and cognitive skills contributing to clinical expertise. Self-efficacy in these skills can be measured using the Human Factors Skills for Healthcare Instrument (HFSHI) [18]. This instrument has been developed and shown to provide a reliable and valid way of measuring training participant’s human factors skills self-efficacy across both acute and mental healthcare settings [16].
Much of the literature has identified the most important component of SBE to be post-simulation debriefing [6,19,20]. Indeed, in a systematic review looking into the importance of debriefing in SBE, 51 studies identified educational feedback as the most critical component for effective SBE [21]. Debriefing involves educators turning into facilitators and guiding participants through a reflective discussion of the simulation experience. This allows participants to consolidate their experiential learning, thereby enhancing workplace practice, patient care delivery and experience [22]. Ryoo and Ha [23] have shown that those debriefed after SBE also had improved technical and non-technical skills, relative to those without debrief. It is the improvement of non-technical skills that is reflected in higher HFSHI scores post-SBE, which is seen across clinical [18] and non-clinical contexts [24].
However, there is no single debrief style used across SBE. Two popular models include Pendleton’s debrief model [25], and the Diamond model [26]. Pendleton’s model focuses on giving balanced feedback to participants, in terms of what was done well, and what could be improved; essentially, Pendleton and colleagues aimed to counter the problem of error-focused feedback, which has been shown to negate the effectiveness of debriefing [27]. As Pendleton provides general rules, there is no singular Pendleton model; most are modified when applied to SBE. Conversely, the Diamond model is a more structured process based on ‘description’, ‘analysis’ and ‘application’, in that set order. The Diamond model ensures that evaluation of performance is not immediate, by starting with an objective description of the simulation [26] before building the conversation around learning experience and implementation. The effectiveness of diamond-structured debriefing has been attributed to its interactive and reflective nature [28].
Both the Diamond and Pendleton models have been proven effective at improving non-technical skills, though they have not been directly compared [29,30]. Furthermore, Gantt et al [31] have shown that different debriefing techniques can affect learning outcomes, in the sense that using facilitated debriefing is more effective for knowledge retention than self-debrief. This supports notions that debrief styles and models can affect participants’ learning in different ways.
This research paper aims to begin addressing this gap in the literature, where these models of debriefing have not been overtly directly compared. We aim to answer the following research question:
Participants consisted of 792 clinical and non-clinical professionals who attended various simulation training courses conducted in South London between September 2017 and August 2019.
Demographic information: A general range of demographics was collected including profession, age, self-identified gender and career stage.
HFSHI [18]: This instrument is a questionnaire using a validated 10-point Likert scale compiled of 12 different items intended to measure self-efficacy of human factors skills for a wide range of healthcare professionals. These include both social and cognitive aspects such as situational awareness, leadership, teamwork, communication and decision-making [18]. The instrument was chosen as it was deemed an appropriate method of measuring self-efficacy of human factors skills pre- and post-simulation training.
A quantitative research design was adopted using pre–post measures in the form of a survey with a validated tool to collect data.
The simulation training courses that participants attended were related to a wide variety of mental healthcare topics, knowledge and skill sets. Depending on the course, these could last for between 1 and 4 days and consisted of specific learning objectives. The majority of training courses consist of a full 1-day session. As the training courses and dates were pre-pandemic, all training was held in Maudsley Simulation’s training centre.
At the beginning of each training session, all participants were briefed on the process of simulation and the structure of the day ahead. Following this, any questions were answered, and relevant notice was given to pre-warn for any distressing content that may be covered during scenarios and discussions throughout the day. Each training day consisted of numerous clinical simulated scenarios (between 6 and 10) using simulated patients (SPs), and each lasting for around 10 minutes. Each of these scenarios were outlined in a manual that all facilitators and SPs follow, were related to specific learning objectives and were designed to focus on certain skills or knowledge areas. The SPs were also briefed before the course and received specialist, comprehensive training from Maudsley Simulation’s quality assurance course, as well as guides and rough scripts for each of their roles. These briefs were produced by course facilitators comprising of multi-professional faculty and were designed to ensure SPs had the appropriate knowledge and skills to portray the patients they were representing accurately. The SPs would usually portray between 1 and 3 patients throughout different scenarios during each training day. Facilitators were generally consistent across all training courses and trained to the same standards in debriefing to ensure they each delivered the same quality of teaching.
Training sessions and scenarios covered a multitude of topics and needs reflective of the course title; for example, some courses targeted specific professions and their training needs (e.g. Mental Health Crisis in the Emergency Department Simulation training), while other courses were designed for a large variety of staff across different settings who may encounter someone who has mental healthcare needs (e.g. Perinatal Mental Health Simulation training). Each session and scenario was created and developed by a team of clinical experts in the field including doctors, psychiatrists, clinical psychologists, mental health nurses, subject matter experts and lived experience individuals. Please see Appendix A for an example of a simulated scenario faced by the participants. Courses followed a mixed structure of didactic teaching led by clinical experts or facilitators, and simulated scenarios consisting of interactions between the SP and training participants. Course facilitators would assign 1–2 participants to each scenario to allow them to play their role as they would in their professional setting. Those not involved in the scenario would observe using a live audiovisual link.
Each simulation scenario would then conclude with a 40-minute in-depth debrief, involving all the training delegates and the course facilitators. The types of debrief used included both an adapted version of Pendleton’s [25] and of the Diamond debrief model [26] in order to unpack experiences of the scenario and learning needs and outcomes. Please see Appendices B and C for the guidance followed by facilitators for each debrief method. For both debriefs, the duration of the scenarios was the same.
For data collection and evaluation purposes, ahead of each training course a detailed information sheet was shared with all participants outlining the use of any data collected for research, and written informed consent was obtained. All participants were also asked to complete a pre-course and post-course questionnaire. These included questions around course learning objectives, course-specific questions and human factors questions using the HFSHI scale. Data were collected for individual, internal course evaluations, which is standard practice for all participants engaging in training courses.
A paired-samples t-test was conducted to investigate whether any significant changes in self-efficacy scores pre- and post-training could be noted based on the type of debrief model used (the Diamond model or Pendleton’s model).
A total of 792 participants started the pre-course survey. Of these, 780 specified their gender; 70.8% were female (N = 552), 29.0% were male (N = 226), with less than 1% indicating their gender as other (N = 2).
| Age range | <20 | 20–24 | 25–29 | 30–34 | 35–45 | 45–55 | 55< |
|---|---|---|---|---|---|---|---|
| N | 1 | 66 | 234 | 144 | 182 | 118 | 46 |
| Job role | Nursing | Midwifery | Medicine | Allied health professional | Other non-clinical professional | Other clinical professional |
|---|---|---|---|---|---|---|
| N | 352 | 23 | 254 | 74 | 51 | 38 |
| Career stage | Qualified | Student | No response |
|---|---|---|---|
| N | 664 | 90 | 38 |
Seven hundred and ninety-two clinical and non-clinical professionals attended various simulation courses conducted in South London between September 2017 and August 2019 and completed the pre- and post-course questionnaires. From this, participants (N = 34) were excluded due to incomplete responses on the HFSHI pre- and post-course questionnaire.
The HFSHI sum difference between pre- and post-scores was screened using Mahalanobis distance to identify participant outliers. There were two degrees of freedom, which equated to a chi-square value of 34.61 at p < 0.001. Cases (N = 7) had a distance score exceeding this value and were subsequently excluded. Thus, the final sample size for further statistical analysis was 751 participants.
Prior to any statistical analysis, Levene’s test was conducted to ensure the assumption of homogeneity was not violated, F(1, 751) = 0.84, p = 0.359.
There was a minimum change of −48 and a maximum of 50 (M = 2.89, SD = 13.73) in HFSHI scores across all participants.
While we recognize and have considered the risk of potential bias with excluding participants, screening for participant outliers where participants may have misread questions, for example, and excluding participants who did not complete questionnaires was deemed a necessary measure in order to accurately match participant data pre- and post-course.
A paired-samples t-test was conducted to determine if there were any changes in self-efficacy pre- and post-training based on the debrief model used. For the Diamond model, analysis showed that there was a significant difference in the pre-course scores (M = 96.83, SD = 12.19) and post-course scores (M = 101.08, SD = 14.70), t(545) = 7.33, p < 0.001. A small–medium effect size of d = 0.31 was observed for this model. Conversely, no significant differences were found in the pre-course scores (M = 92.03, SD = 12.28) and post-course scores (M = 91.16, SD = 14.82) for the Pendleton’s model, t(204) = 0.85, p = 0.399. Both pre- and post-course scores were higher for the Diamond model compared to Pendleton’s. Note that the scores of participants are within the typical range observed when using the HFSHI [18].
| Debrief model | Pre-course | Post-course | p-value | Effect size (d) | ||
|---|---|---|---|---|---|---|
| Mean | SD | Mean | SD | |||
| Diamond | 96.83 | 12.19 | 101.08 | 14.70 | <0.001 | 0.31 |
| Pendleton’s | 92.03 | 12.28 | 91.16 | 14.82 | 0.399 | N/A |
Results indicate that mean HFSHI scores were significantly improved after simulation training amongst the participants that were debriefed using the Diamond model only; there was no significant improvement found for the Pendleton’s model. A significant increase in HFSHI scores for participants who experienced the Diamond model was shown to have a small–medium effect size. Given the limited existence of literature comparing debrief models, particularly within simulation, this result does suggest it is worth exploring further. Our results imply an underlying benefit of utilizing the Diamond model over the Pendleton’s model due to the improvement in self-efficacy scores. Although the finding is not a replication of anything in the current literature, as the direct comparison of debrief models is an exploratory piece of research, it does support research where the Diamond model has been used and significant improvements in Human Factor skills have been reported quantitatively and qualitatively [32].
In addition, the implication of these findings is congruent with literature where the Pendleton’s model has been criticized for its lack of deeper analysis, explanation [33] and positivity bias [34]. Timmis and Speirs [33] note that lasting behavioural changes are unlikely where reasonings behind actions are not discussed in debrief. This places Pendleton’s model at a disadvantage, as the focus on balancing positive with negative feedback means that deeper analysis and explanation, as seen in the analysis component of the Diamond model, could be missed. Additionally, the Pendleton model has been criticized for forcing initial positive feedback, which has been described as patronizing when participants are expecting more negative feedback, and lacking authenticity and productivity through its rigid style [31]. Thus, these factors may contribute to a lack of significant difference observed with the Pendleton’s model.
Nonetheless, it is important to note that the sample size of each group differed greatly, with the Diamond debrief group having over twice as many participants than the Pendleton’s group, which may have an impact on the power to detect change. A potential reason for this imbalance, however, may be related to the criticisms noted in the use of Pendleton’s model as a debriefing technique and an underlying preference towards the more structured and in-depth process of the Diamond model for SBE. It is important to note, however, that for both groups, the duration of scenarios did not change, and both groups covered similar courses including both clinical and non-clinical staffing groups, of different stages of qualification.
However, the vast sample size generally puts our research at an advantage in the literature; not only was the sample representative of a large age range working within healthcare, but a vast range of healthcare professions and mental health training topics were also represented in the data. Improvement in self-efficacy when the Diamond model was used was observed across a range of simulation courses that have dissimilar learning objectives, indicating that the model is effective regardless of course type or topic. This also suggests that the Diamond model style of debrief can be applied to various healthcare workers training and learning needs. However, it would be beneficial to also explore the efficacy of the model in non-healthcare SBE.
There are other limitations to this research that must also be considered. As discussed previously, self-efficacy is not a direct measure or predictor of behavioural change, but only a factor which can influence this and so while our results do evidence that the Diamond debrief model yields higher self-efficacy scores, we cannot assume a direct causal link to behaviour change. While we justified self-efficacy as a proxy for perceived skill, caution must be taken with directly applying these findings. Indeed, future research is needed to explore longitudinal aspects of our findings to assess whether the model has any long-term impacts on learning, behaviour change and perceived self-efficacy. Moreover, explanatory research looking into debriefing specifically would be beneficial for further unpacking why the diamond model is effective in improving self-efficacy scores. While we can infer that the improvements are likely down to the description, analysis, application structure of the model whereby the reflective nature of the model builds on learning experiences, more research is needed to confirm and elaborate on this.
It is also important to recognize that this study was an opportunistic secondary analysis of data collected from simulation courses run previously. Therefore, groups were not able to be matched, and there was increased potential for confounding factors. Factors including previous attendance at a simulation training and more exposure to debriefing amongst some participants may have impacted improved scores. For those who have attended more training, have higher levels of expertise, knowledge or experience, they may already have a higher perceived self-efficacy, or contribute more to debriefing, also potentially impacting scores. Previous research has explored the possibility of gender differences impacting HFSHI scores [35]; however, this is unlikely to be a major limitation because of the little difference found in HFSHI scores due to characteristic differences. Lastly, while a strength of this paper is its large sample size, there could be a respondent bias where not all participants completed both pre- and post-course questions and so our results are not wholly representative of all participants who attended the training courses.
Despite these limitations, our findings make an important contribution to the gap in SBE literature where debrief models are not extensively compared. For the wider simulation community, and within mental health simulation training, these results indicate the benefits of using the Diamond debrief over other potential debrief models on the improvement of non-technical skills.
None declared.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Data are available on request.
Ethical approval was awarded by the Psychiatry Nursing and Midwifery Research Ethics Subcommittee at King’s College London, PNM 13/14-179.
The authors declare no conflict of interest.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
Simulation Scenario in ‘Return to Work’ course
| Communication & Taking a History | Summary: | Setting: GP, Neurology Clinic |
| Scenario | Actor Instructions: | Requires: 1 actors |
| Jemma Jacobs is a 30-year-old female, who has a history of tonic clonic seizures since childhood. Her seizures have worsened recently, which neurologists have worked up & feel to be secondary to non-epileptic seizures, so they want to refer her to a psychiatrist. The work up has included an MRI & 24-hour video EEG which showed the patient having these episodes without seizure activity. | From course: 1 participants | |
| She developed seizures as a teenager. She loses consciousness and then feels tired afterwards. She was started on medication (Lamotrigine (pronounced la-mot-ra-gine) 100 mg once daily) and the seizures were well-controlled on this and has not had a seizure for a number of years. Then a few months ago, she started to have seizures, they were a little different and felt as if she was separated from the world, but she managed to control some of the seizures and she can communicate with people during it. Sometimes she falls to the ground. These are happening multiple times a day and impact on the care she provides for her mother. | Set Up & Props: GP Letter Neurology w/video EEG results MRI report |
|
| The patient has been back to see her neurologist who has stated that these new ‘attacks’ do not sound the same as epilepsy, which, in his opinion, remains well controlled. | Participant task: The task for the participant is to begin to take a history of presenting complaint. | |
| She currently does not work as she had to stop to look after her mother who is unwell at the moment with an early onset dementia. She is also married with two young children, ages 6 & 8, a boy and girl. Her husband is very busy, works as a management consultant in the City. The mother now has carers twice day, but the patient had moved her mother into her home to help feed her and look after her between carer visits. The mother can manage some things herself, but the patient is having to do more for her than before. She still recognizes the patient, but the patient fears the point when she cannot. The patient has a sibling who is helping out now that the patient’s seizures are getting worse, but has not previously been helping out up until now. Her husband also helps out a bit more with household responsibilities and kids when the patient is unwell. The patient claims that she is happy to care for her mother as well as the kids and does not recognize that this is stressful. She previously worked as an office manager. She had a difficult time at school and was bullied as an adolescent. | ||
| She is sleeping poorly, as she is worried about her mother, appetite is gone, but her intake is fine as she eats with her mother. | ||
| She does have some friends whom she sees occasionally but not much as she you look after her mother and is quite isolated. | ||
| Unsure what to look forward to as only looking after her mother most of the time. | ||
| She stays away from all drugs and alcohol b/c her mother used to drink a lot when the patient was a child. | ||
| She tends to be almost emotionless, detached from her feelings of stress. | ||
| Overall, the patient doesn’t really agree that this is psychological, but rather that this is something medical & that she requires further medical investigation. She is resistant to psychological suggestions & brings the discussion back to medical interventions. | ||
| End when you hear voice saying scenario has ended. | ||
| Debrief Discussion: Learning objectives & Handouts: Taking a functional history |
Modified Pendleton’s model, as instructed in facilitator training
The Diamond debrief model, as taught to facilitators
The impact of debrief models on self-efficacy within mental health simulation training: a quantitative analysis
Twitter
Facebook
WhatsApp
LinkedIn
Tweets