During the height of the COVID-19 pandemic, healthcare systems globally experienced immense strain and uncertainty. Preparedness was essential to manage the worst-case scenario of overwhelmed critical care capacity and potentially prevent having to choose which patients should receive life-saving critical care. The developed critical care triage (CCT) framework was evidence-informed and provided a framework to guide and execute systematic clinical decisions; however, the operationalization of the CCT framework presented several challenges. Hence, proactive testing of the CCT framework was essential. Simulation and human factors teams collaborated with provincial clinical experts in critical care to develop online facilitated scenarios and identify potential latent safety threats. Using simulation-based education and human factors in the novel methodology of testing a CCT framework revealed key insights and learnings, which were subsequently embedded into the iterative updates following the simulation. The outcomes from these simulations informed organizational learning on the highest risk and highest impact recommendations to be prioritized.
As the COVID-19 pandemic entered its fourth wave in the fall of 2021, healthcare systems globally were straining to meet the surging demand of patients who required critical care resources . The increased stress experienced by the healthcare system, coupled with the level of uncertainty present during the pandemic, brought the reality of having to potentially enact critical care triage (CCT; in Alberta represented by the CCT framework) [2,3]. As a result, it became crucial to be prepared for the worst-case scenario of overwhelmed critical care capacity and being able to ensure the most patients benefited from the life-saving critical care resources available .
It is essential to have planned and predetermined approaches for healthcare professionals to guide responses when allocating resources and ensuring the most equitable process is consistently followed [5,6]. The CCT framework provides a guide to execute clinical decisions; however, the conceptual operationalization of the CCT framework and preparation presents several challenges. While using evidence-based triage frameworks is pragmatic and provides a systematic approach to decision-making , there is uncertainty about its application from theoretical conceptualization into practice.
While many current articles describe the content of CCT frameworks [8,9], they have not described (a) the value in beta testing a CCT framework through online facilitated simulation prior to implementation to identify potential latent safety threats, and (b) have not had to consider the implications for time-sensitive decision-making in testing the framework across the geographical span of both rural and urban centers, ensuring equitable distribution of resources and access to critical care, regardless of patient location in a healthcare system. In addressing this imminent need, the simulation and human factors teams collaborated with provincial clinical experts in critical care to develop online facilitated scenarios, designed to identify potential latent safety threats that may have occurred.
The goal of this article is to describe the additive benefits of combining both a simulation and human factors methodology demonstrated through a quality improvement case study. Furthermore, the sharing of insights into how these unique approaches can be used for simulation-based testing, and providing recommendations for simulation practitioners who would want to design and enact similar exercises using online facilitation for broader organizational learning.
This quality improvement case study took place in Alberta Health Services (AHS), the largest Canadian provincial fully integrated health system encompassing 106 acute care hospitals and 21 dedicated adult intensive care units that provide population-based care to an estimated population of 4.4 million spread across a large geographical area [10,11]. Within the provincial healthcare system, the Strategic Clinical Networks™ (SCNs) exist to advance improvements in specific areas of health and work provincially to develop integrated, sustainable solutions to complex, multidisciplinary challenges including one that focuses on critical care. The SCNs™ collaborate with patients, families, clinicians, operations, communities and researchers to address the most pressing healthcare challenges facing our organization [12,13].
In the spring of 2020 with the declaration of the COVID-19 pandemic, the AHS CCT framework was developed to ensure that triage decisions are ethical, fair and transparent, using a pre-developed and objective framework to guide healthcare professionals in the unthinkable task of CCT. As all individuals have equal moral worth, the principle of formal equality is used when several patients have an equal likelihood to benefit from critical care. Anchoring the triage process is the Capacity to Benefit: the best action when demand for scarce critical care resources exceeds supply, with the goal to save the greatest number of lives possible. Please see supplementary material for a summary of the AHS CCT triage framework . The testing of the framework was further compounded by the provincial approach taken to support the vast geography contained under a single healthcare system. Equitable access to care and resources could not be limited by patient location. Leadership and clinicians were required to coordinate across large provincial services. Logistically, the unique requirements of using online facilitated simulation had to provide a modality where engagement of participants respected COVID-19 physical distancing and allowed for individuals to participate from all five zones of the province, including all levels of urban, rural and remote areas.
To ensure consistency and provincial perspective to CCT, the Critical Care SCN™ engaged and collaborated with the AHS provincial simulation and a provincial human factors program. Proactive testing of a CCT framework using simulation-based education and human factors is an emerging methodology that remains limited in the literature . Human factors are best described as a body of knowledge about human abilities, human limitations and other human characteristics that are relevant to organizations and workplaces. When combined with simulation-based activities, applications can focus on the design of tools, systems, new processes, strategic planning and optimization of an inherently complex healthcare system [14,15]. In 2022, Kochel and colleagues used simulation alone to evaluate the acceptability, feasibility and reliability of an institutional triage framework based on German guidelines during the COVID-19 pandemic. They found the framework to be realistic and acceptable, while emphasizing the importance of interdisciplinary cooperation and education prior to having to use the framework . Figure 1 visually shows how the triage framework would entail complex decisions through a team-based approach, using predetermined processes for selection and prioritization of adult patients into critical care beds across the province.
Due to the expediated timeframes required to complete these simulation exercises, a modified methodology to the development and planning phases was implemented, incorporating elements from frameworks developed for Systems Focused Simulations and known project management approaches [16–21]. The project management plan started with the development phase, where an intake meeting included a needs assessment, clarifying the goals, objectives, scope, timelines and key stakeholders. Brainstorming sessions in subsequent meetings helped refine anticipated concerns, potential gaps and process issues that might arise and included these for testing [16–21].
The overarching objectives of the CCT simulation exercises were threefold: (1) assess and define current resources, communication pathways, algorithms, cognitive aids and checklists, (2) identify gaps and mitigation strategies to better inform the operationalization of CCT and (3) develop familiarity and comfort with the process for CCT and validate role clarity and responsibilities. Specific detailed objectives for each scenario, which were determined by facilitators, are listed in Table 1. However, an unintended outcome was the learning that occurred from session to session as to how to best utilize and leverage the technology and approach to online facilitated simulation.
|Scenario 1||Scenario 2||Scenario 3|
|Simulation objectives||Enact the CCT consultation for admission
Familiarize stakeholders with their roles and application of frameworks using CCT documentation
Explore decision points for triaging patients: how to decide and manage the queue
Communication and testing the process for electronic queue and patient prioritization
|Educate and test the Triage Dispute Team:
• Convening process
• Agenda for meeting
• Process for dispute resolution
• Communication back to stakeholders
Allow Triage Coordinator to present case to the Decision Committee
|Allow teams to work an end-to-end scenario using the documentation (Eligibility, Exclusion Criteria) and making triage decision
Triage Coordinator Role
• Communication between frontline providers and patient placement team
• Presentation of case to Decision Committees
• Evaluate role education received prior to simulation
Dispute resolution process – Convene Decision Committee
Re-test bed allocation process from first simulation
Due to the constraints of urgency and the looming threat of having to enact the framework, the simulation was confined to the adult population only as COVID-19 at the time was primarily affecting adult patients. The pediatric inclusion criteria and response outlined in the protocol were considered out of scope. As well, the exercise would not be able to fully explore moral and ethical distress of both patient and staff and was excluded.
Timelines from the initial request for assistance to the first exercise were less than 1 week (17 September/23 September 2021). The second exercise occurred 4 weeks later, on 21 October 2021, with the final end to end exercise occurring on 18 November 2021 (see Figure 2).
The key stakeholders both developed the framework, held a fundamental role in its operationalization and were crucial in the initial planning stages. For the simulation exercises, stakeholders fulfilled roles as facilitators, participants and observers. Facilitators included representative from the Critical Care SCN™, eSIM (Educate, Simulate, Innovate, Motivate) Provincial Simulation program, Human Factors (HF), and Referral, Access, Advice, Placement, Information & Destination (RAAPID), AHS’s single point of contact for physicians and healthcare providers to access appropriate and timely advice, referral, admission, repatriation and consultation for patients.
Participants in the simulation exercises represented their professional role within the system, and included operational leaders, Integrated Operations Command Centre members, frontline clinicians (physicians and nurses), RAAPID staff, along with patient and family advisors. One week prior to the simulation exercises, each participant received an education package that included the CCT Framework, standard operating procedures, educational videos, documentation and resources pertinent to their role. To be fully prepared, there was an expectation of these participants to have reviewed all these resources prior to the simulations. Familiarity and understanding of the protocols and resources varied amongst the groups, as some aspects of these documents were in use as part of clinical practices (sequential scoring) and some participants had been involved in the design of the framework. Triage Coordinators attended two 1-hour online pre-education training sessions which included case scenarios intentionally scheduled at intervals prior to the simulation sessions that mimicked proposed timelines for activation of CCT.
Across all three simulation sessions, greater than 60 observers attended including Clinical Ethics, Emergency Disaster Management and stakeholders who contributed to the development of the CCT framework. Observers were used to identify and record system issues. Their feedback was elicited through the debriefing and post-simulation evaluation feedback forms. Table 2 provides an overview of the stakeholders involved in simulation exercises by role and role description.
|Facilitators (N = 13)||Participants (N = 33)||Observers (N = >60)|
|• Developed critical care process/content expertise
• Developed patient scenarios and debriefing outline
• Lead the simulation (SCN)
• Monitored chat boxes
• Supported the debriefing
• Documented outcomes
|• Scenario 1: Have intimate knowledge of CCT framework and played a role realistic to their clinical background
• Scenario 2: Content experts in bed allocation eligibility
• Scenario 3: Participants who were not involved in the development of CCT framework
|• Provided clinical expertise
• Medical legal implications
• Ethical considerations
• Patient and family perspective
|Role description||Role description||Role description|
• Human Factors Specialists
• Critical Care Leadership (Educators, Management)
• RAAPID Management
|• Critical Care Leadership (Physicians, Nurses, Management Zonal and Provincial Executive)
• Emergency Department physician leads and nurses
• CC Triage Nurses
• RAAPID leads (nurses, EMS)
|• Critical Care Leadership
Site Leadership for Critical Care and Emergency Departments (physicians and nurses)
• Patient Family Centered Care Management
• Legal Representatives (AHS)
A vital step to pre-testing the simulation was to conduct a process walkthrough of the entire CCT framework prior to the implementation phase. The objectives of this were to (a) obtain clarity on the scope and breadth of the content being tested during each of the simulations and (b) pre-emptively identify gaps in the framework prior to running simulations. Guided by the key critical care stakeholders, decision algorithms and newly developed processes were reviewed with the simulation and human factors teams. The walkthroughs helped inform the development of patient scenarios to ensure that all components of the CCT framework were tested and known challenges were incorporated into the scenarios. Based on this information, the entire CCT framework was divided into three separate online facilitated simulation sessions. The walkthrough also provided insight into the specific pre-education that would be necessary for participants who did not have intimate knowledge of the CCT framework prior to the session.
An online facilitated tabletop simulation format was selected to test the CCT framework due to the geographic spread of participants over a large area (i.e. across the province) and constraints on in-person meetings during the pandemic. An online facilitated simulation is a method that involves end users working through clinical patient scenarios used to assess plans, policies and procedures by applying relevant documents and processes on a technical communication platform , in this case ZOOM™.
Given the complexity of the framework and sheer volume of potential if-then scenarios, it was determined that the testing of the CCT framework would be split into three separate simulation exercises of varying lengths (approximately 2 hours each). All three simulation exercises began with a structured pre-briefing outlining the purpose, objectives, limitations and scope of the simulation session followed by introductions of facilitators, participants and observers. A debriefing for facilitators was held after each simulation session, and key takeaways with respect to coordination, delivery of technology, data collection as well as facilitation/debriefing strategies were discussed and revised for each subsequent simulation based on the learnings from the previous simulation. In this way, the iterative process resembled a Plan-Do-Study-Act (PDSA) cycle , and the online facilitated tabletop was refined and improved from each simulation session.
Each of the scenarios was designed with specific context and objectives that informed each other through an iterative approach. To facilitate and allow for movement of patients within the exercise, a dashboard was developed to allow visibility of the de-identified waitlist (i.e. no patient names used) and available resources across the province. Validity testing of the CCT framework and clinical decision-supporting documents was needed and achieved through a series of scenarios. The following section describes the scenarios developed to test the CCT framework.
Scenario 1 involved presenting an existing list of patients waiting to be admitted and allocated a bed with corresponding CCT scores, while adding new cases to the list and asking participants to allocate beds accordingly. Specific to this scenario was the inclusion of one patient who would not meet eligibility requirements to ensure the CCT framework was effective at correctly prioritizing bed allocation. This simulation allowed participants to focus on bed allocation, use the dashboard to sort and prioritize patients.
Scenario 2 focused on resolving triage disputes (i.e. a frontline team could not agree on eligibility) by escalating the decision to the triage dispute committee. The Triage Coordinator activated the triage dispute team and presented the relevant patient information and the committee then asked clarifying questions. The committee then worked through the process for dispute resolution and review inclusion/exclusion criteria. Specific to this scenario was the threshold at which the triage dispute committee is engaged, the process to convene and review the patient case and the communication pathways involved.
Scenario 3 provided the opportunity for participants to simulate the CCT framework from beginning to end and involved two patients. It included the added challenges of rural medicine as well as other complexities related to limited resources in a non-urban setting. The communication and process for bed allocation were also tested to ensure important discoveries from previously run simulations were incorporated.
The facilitators focused on the identification of latent safety threats and guided participants through the predetermined objectives. Debriefings were conducted with systems lens and learnings were captured using the Systems Engineering Initiative for Patient Safety (SIEPS) 2.0 human factors framework categories: assessing the tasks, the processes, the support systems (organization), interpretation of documents (tools) and integration of clinical decisions (people) . By simulating the integration of these systems, potential issues were identified and addressed before they could occur in the real-world setting, improving patient safety and care quality [25–28]. Within-Simulation debriefings were held at predetermined transitions throughout the scenarios. Debriefing models built on the recent work from Fatimah and Clapper using micro-debriefings and reflective pauses took place during certain customized checkpoints of the simulation. [29,30]. These frameworks were adapted to improve the flow of the debriefing, including targeted Plus Delta questions on ‘what went well’ as well as reflecting on challenges that may have arisen in the scenario. Facilitators used strategically placed probing questions during transitions where anticipated obstacles required further discussions with the larger participant and facilitator teams. In addition to participants and facilitators, observers were also given the opportunity to give feedback and perspective on the framework during debriefing discussions. Outcomes were collected, collated and themed based on the debriefing discussion, chat box comments and evaluations from participants and observers. Finally, a ‘parking lot’ of additional recommendations that were out of scope for the exercise or could not be addressed given the tight timelines and experts present, were kept, and taken forward by the CCT team.
The iterative process facilitated by human factors and simulation, allowed the SCNs to refine the CCT framework and documents by using them in a realistic manner with clinicians. Challenges with the documents were observed and addressed, and end-user feedback was incorporated into both the design of the documents and process itself. Critical dashboard design changes included the clear differentiation of patients who were ineligible for beds, as well as the addition of a free text status column to provide details such as transport requirements and site changes. This ensured greater situational awareness across the province.
Examples of additional reference documents included a visual aid for all provincial site-based critical care sites, outlining ICUs that offered specialty services such as Cardiac, Neurosurgery and Trauma. A CCT Quick Reference document and a Frequently Asked Questions tool provided supplemental resources for leadership and frontline clinicians. Communication issues in the handover processes from the triage coordinator to the eligibility review team became more structured with the addition of a formalized huddle agenda and handover tool with embedded scripts. Condensed workbooks were created to address the difficulty noted with scoring triage eligibility and variances in the frontline clinicians’ understanding of the Sequential Organ Failure Assessment (SOFA) score. This change allowed the Point of Care Triage team to better navigate through the ineligibility criteria. Additionally, inserting calculation tools into the assessment document facilitated a shared understanding of the patient’s condition. By explicitly labeling roles and responsibilities in the reference documents, team members and leadership gained an understanding of who would have decision-making power. In total, 50 safety threats were identified, 4 design changes were made to the dashboard displaying patient queue information, and 6 additional reference documents and 3 new documents were created to support end users completing CCT.
The sequential nature of the simulations allowed for an iterative design process to occur with the simulation process as well. The online facilitated simulation process was adapted to ensure that the simulation unfolded in a realistic manner. Data and dashboards were updated in real time based on participants’ decisions. Facilitators needed to be able to converse amongst each other, collect and collate participant and observer feedback, adapt the simulation and identify debriefing discussion themes, all as the simulation unfolded. To do this, these roles were spread amongst more facilitators: one to update dashboards in real time, one to organize observers’ feedback and multiple facilitators to keep the simulation going in a realistic manner. In the first simulation, one facilitator was assigned to run both the dashboards and provide patient information verbally as part of the scenario. This was not a manageable workload and the team transitioned in real time to have another facilitator take on the role of updating the dashboards and other visual presentation of data. The following sessions incorporated more equal division of labour among facilitators. Furthermore, in the first simulation the Zoom meeting chat function was used for both observers to enter comments and questions and to provide information relevant to the simulation. This caused some confusion and missed information relevant to the scenario. In future simulations, a separate messaging application was utilized for facilitators to communicate with each other. Observers were encouraged to share their findings in the debriefing checkpoints and on the observer, checklist submitted at the end (see Figure 3).
In summary the extensive data collected on the gaps and latent safety threats that were identified (highest risk and greatest impact) enabled validity testing for frameworks and documents, provided feedback on the education and training delivered to ensure the CCT framework met the highest ethical and patient care standards.
The COVID-19 pandemic is a crisis that has significantly impacted the Canadian healthcare system . As of March 2023, the virus has not been contained, and while vaccinations are underway in Canada, future logistical and distribution challenges mean COVID-19 is an ever-present concern . Preparing for the unknown is extremely difficult given the unpredictability of multiple waves of the COVID-19 pandemic . Valuable insights from this facilitated exercise highlighted the critical need for testing and validating the CCT tools within a strained healthcare system. Healthcare simulation has emerged as a valuable tool in preparing these teams for pandemic responses and potentially other emergency situations in a safe and controlled environment. The addition of human factors expertise augmented the simulation exercise by applying a human error lens to help identify gaps in the process and potential for confusion in the CCT documentation and tools.
The global pandemic emphasized the importance of leadership and staff preparedness in times of crisis  and the value of testing the CCT framework prior to activation as a strategy to support emergency management (EM). While expertise in the creation and development of CCT framework, design, frameworks and resources has been previously well described [34–36], there has been scarcity of literature describing examples of applications of simulation and human factor approach to test and evaluate the real-time execution of newly developed CCT frameworks. The literature, however, highlights how EM methodology plays a key role in preparing Canadian healthcare organizations for disasters and emergencies . During the COVID-19 crisis, healthcare systems formed integrated operations committees, fashioned to mimic EM structure . These committees helped to maintain plans while ensuring logistics and communication systems functioned effectively in times of crisis. Much like emergency disaster preparedness exercises [33,34], the online facilitated scenarios were designed to prepare clinical operational leaders, with frontline medical experts making complex care decisions through a team-based approach. This was accomplished by using prioritization documents and clear processes that informed revisions to the framework and education modules, development of cognitive aids and the need for additional re-testing following those revisions. Regrettably, empowering and training healthcare leaders in EM techniques can be problematic, as they typically lack the specific knowledge, experience and preparedness in managing large-scale crises effectively . Once trained, keeping the mastery level of knowledge of disaster EM, wanes in the time that occurs from the last time it was practiced or enacted. Recommendations from this facilitated exercise would include future annual exercises for the CCT, to maintain leadership preparation, familiarity of the protocol and system readiness.
The COVID-19 pandemic has forced many healthcare systems to adopt online solutions. This case study had to pivot and adapt to the challenges of a provincially integrated system, across multiple units, departments, hospitals and geographic zones. Simulation can be adapted to these online environments, providing healthcare providers with similar levels of training and preparation as in-person simulation exercises. With technology, these simulations exercises can be accessed from anywhere, allowing healthcare providers to participate in training programs even during periods of lockdown, quarantine and geographical barriers. Our online facilitated simulations build on the approach of Dogan et al. (2021), that have used ‘Visually Enhanced Mental Simulation’ (VEMS) where teams rehearse all the steps of scenario in an interactive environment without physically doing it, using both a ‘think aloud technique’ and team decision-making approach . Using the VEMS technique, our participants were able to enact what a CCT response would look like without having the pressures of real patient decisions. Applying these complex decisions across the geographical span of both rural and urban centres, ensured equal distribution of resources and access to critical care, regardless of patient location in the province. Through the beta testing of the CCT framework using online facilitated large-scale simulation and engagement of stakeholders across the entire healthcare system, the team was able to develop recommended actions (fix, change, improve) for time-sensitive decisions based on the highest risk and highest impact to the healthcare system.
Overall, healthcare simulation is a crucial tool for preparing healthcare teams for pandemics and other emergency situations, developing leadership skills and facilitating online facilitated learning. As the world continues to navigate the challenges of the COVID-19 pandemic and beyond, simulation will remain a vital component of healthcare training and preparedness.
This study is subject to some identified limitations. Only 33 participants and 60 observers took part in the simulations, thus limiting access to data and generalizability to all relevant clinical groups across Alberta. The results of the simulations exercises have been generated through a set of clinician responses and assumptions with one Canadian provincial healthcare system; therein, care should be taken in generalizing these findings to other settings and contexts.
Further, the quality improvement project used a cross-sectional design and included a convenience non-probability sample which may have resulted in sampling and selection bias of participants and feedback on the CCT framework. All simulations sessions were held remotely, which may have influenced the rapport developed between the facilitators and participants and potentially influenced the data obtained in the debriefing. Also, participants may have had some degree of recall bias, recalling either only incredibly positive or extremely negative experiences during the simulation session. In addition, the limitations to the realism of the online facilitated simulation exercise could not fully replicate what clinicians in the real world would have to face, balancing the care of current patients while also being involved having to treat for incoming and current critically ill patients. Limitations of the scope of the study were due to the constraints of urgency and the looming threat of applying the framework. These confines included not being able to assess the pediatric component of the CCT due to COVID-19 primarily effecting adult patients and fully exploring moral and ethical distress of both patient and staff. Further studies and research into CCT frameworks could explore using simulation to evaluate pediatric inclusion criteria and decision-making. Another limitation of the simulation exercise design was the potential for confirmation bias concerning the appropriate level of educational preparation received on the CCT framework. At the time of the testing, many leaders, educators and clinicians in the participant roles engaged in the development of the process and familiar with the roles, responsibilities and documents. (Prior experience with CCT framework: Scenario 1 N = 50%, Scenario 2 N = 83%, Scenario 3 N = 50%). Pressures on the frontline prevented the team from pulling human resources away from the bedsides of real patients. The design of the exercise was also constrained by having to condense the timeframe into a 2-hour exercise, so may or may not have accurately reflected real-time decision-making.
Finally, a key assumption of this case study was that the changes made to the framework and documents following the simulations could have direct impact on the framework. As these items were never operationalized in a real context because of declining system stress, we can only extrapolate that the themes and latent safety threats identified in the three simulations could improve clinician ease, understanding and application of the CCT.
In the event of a pandemic, preparedness is crucial. No one has been trained for a situation in which our healthcare system is strained to meet the surging demands of patients who require the resources of critical care. The overall pattern of the coronavirus pandemic so far has been a series of COVID-19 waves: surges in new cases followed by declines, new variants of concern and no predictable end in sight. Therefore, it is imperative that healthcare professionals are supported to meet the decision-making challenges they may face.
The bridging of both simulation and human factors teams brought expertise in preparing systems for real-time testing of the CCT framework, resources, efficacy and usability. It is anticipated that the organization’s learnings from these novel simulations give a more realistic appreciation of what clinicians would experience if the CCT framework became operational, allowing for a proactive approach to ensure healthcare system preparedness and patient safety. Furthermore, this experience has informed simulation sessions that use online facilitated simulation and future large-scale simulations will benefit from the learnings from this project.
The following groups participated in exercise design and subject matters experts from Alberta Health Services: RAAPID, Provincial Critical Care Triage Committee, Critical Care Strategic Clinical Network™, eSIM Provincial Simulation Program, Health Profession Strategies, and Patient Relations.
All authors contributed to manuscript conception and design. Material, preparation, data collection and analysis were performed by SB, SM, JS, KR, CS, JO, SS and AK. The first draft of the manuscript was written by SB and all the authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sector.
Ethics approval and consent to participate is not applicable as this was a non-research project. The project followed the successful completions of the ‘A Project Ethics Community Consensus Initiative ARECCI’ screening tool (https://arecci.albertainnovates.ca/ethics-screeningtool/). This decision support tool identified the primary purpose of the project as quality improvement/program evaluation and that the project involves minimal risks: therefore, review by the Research Ethics Board was not required.