Healthcare systems improvement using simulation and debriefing is an increasingly employed, yet underutilized quality improvement tool to enable user-centred design. This approach allows users to experience real-life systems and processes through simulation and then provide feedback on how a system supports them within their role. Understanding this interaction of people and their systems is critical to safe, quality, reliable and efficient care and bridges the gap between how we think a system is working and how it is working.
This novel project was collaboratively developed and led by simulation, human factors and patient safety experts and used existing organizational safety data to target further high-risk safety threats surrounding administering, cross-checking and labelling blood products for transfusion. A system-focused simulation-based approach was used to identify system issues for a large healthcare organization’s transfusion policy redesign. A Failure Mode and Effects Analysis (FMEA) was then used to apply a risk score to the findings from the simulation user feedback to inform a large high-risk policy redesign.
Multiple recommendations were provided to the participating units and policy and procedure redesign teams surrounding environmental issues, standards, interpretation and usability of the policy.
Our collaborative patient safety, simulation and human factors project was successful in proactively identifying both active and latent factors contributing to adverse events and identifying recommendations using FMEA methodology to improve patient safety, including revisions to the physical space within the lab, and the provincial blood transfusion policy and procedure.
• An interprofessional approach to address system failures.
• Using simulation to identify potential failure modes.
• Analyzing simulation results using Failure Mode and Effects Analysis.
• Enabling user-centred design to inform policy.
Healthcare organizations continue to move towards increased standardization of policies and protocols. It is critical that a user-centred design approach is utilized to ensure that they will work for the people and teams required to use them [1]. Many policies and protocols are designed without testing, evaluating and collecting feedback from the system users, both during design and after implementation, in an applied environment that is realistic (i.e. considering human factors elements). The Systems Engineering Initiative for patient safety (SEIPS) 2.0 [2] model provides a conceptual framework for the interrelationships of a complex healthcare system noting that the many components of the work system could impact processes and outcomes in significant ways. Thus, the impact of ineffective or poorly designed policy and procedure may be significant in a complex system.
The premise that ‘work as imagined’ is not ‘work as done’ is essential to consider when designing and evaluating these processes and their impact on care delivery [3]. These policies and protocols may not meet the users’ needs by lacking the necessary detail, language or steps required to guide clinicians in their typical day-to-day work, which can add confusion to critical patient safety care processes and increase the chance of error due to poor design.
The field of human factors focuses on how human interacts with all elements of a system. This is critically important in complex systems, including healthcare, where humans fill numerous roles within the system and where errors can have devastating consequences [4]. It is crucial to examine how the different roles work simultaneously to achieve a goal and use this information to design policies and protocols to ultimately improve patient safety.
Simulation for systems integration (SIS) is a growing quality improvement tool to improve the systems and processes of care delivery [5–11]. SIS can uncover systems issues, including latent safety threats related to process steps within any protocol or procedure. System-focused debriefing (SFD) has many applications. It allows facilitators to effectively obtain feedback after a simulation experience (i.e. re-creation of a real-to-life experience within a system) to guide reflection and uncover systems issues [12]. Essentially, SFD enables a better understanding of how work is done.
The outcomes from SIS/SFD can be categorized and rated using the Failure Mode and Effects Analysis (FMEA) methodology [13]. FMEA provides a systematic and proactive approach that can be used to evaluate new or current processes for potential failures. Once these failure modes are identified, action can be taken to mitigate the risks before something goes wrong. FMEA allows healthcare organizations to identify processes with the greatest potential for patient harm, prioritize resources accordingly and proactively intervene by introducing various levels of controls to assist with the detection and prevention of harm before it occurs. FMEA looks at all the potential failures and the effects of the failures within each step of a process. A risk score is calculated for each possible failure considering severity, the frequency of occurrence and how detectable it would be before it happens. Our large provincial healthcare organization collaborated on a focused project between the patient safety, simulation and human factors teams to inform and improve a high-risk blood transfusion safety and policy redesign. Using existing safety data, including Quality Assurance Reviews (QARs) and voluntary Reporting and Learning System (RLS) reports, the project team used combined simulation, human factors and patient safety methodologies, including FMEA, to target the process steps of administering, labelling and cross-checking blood products for inpatient units.
Our work builds on a closely related study by Campbell et al. [14]. That study describes the use of in situ simulation in an operating room environment to uncover latent safety threats related to blood transfusion. Although a formalized FMEA risk rating system was not used as described in our methods, our collective publications emphasize the value of using simulation as a powerful means to uncover system-level patterns and gaps in policy and across different types of complex healthcare environments and organizations.
There is otherwise a paucity of literature on the use of FMEA in combination with SIS and SFD for redesigning high-risk protocols in healthcare, such as blood transfusion. When a patient receives an incorrectly matched blood product due to a cross-checking or labelling error, the results can be fatal. While FMEA has been used previously to improve the safety of blood transfusion [15], the purpose of this paper is to describe a novel quality improvement project approach to inform and improve a high-risk blood transfusion policy redesign for a large provincial organization using a human-centred design through SIS, SFD and FMEA.
Our project used existing patient safety data to target the approach and identify opportunities for system improvement. The organization tracks both QAR (reviews of safety-critical events to identify system issues and determine opportunities for improvement) and RLS reports (voluntary reports from staff that include hazards, close calls and adverse events to learn about and improve patient safety that is subjectively coded and classified). Our project was deemed quality improvement based on criteria presented in the ‘Project Ethics Community Consensus Initiative ARECCI’ screening tool; therefore, ethics approval was not required [16].
A typical organizational review of the QAR records and RLS reports identified common themes of transfusion-related events and reports. This led to the formation of an interdisciplinary project group, including representatives from Patient Safety, Simulation and the Human Factors team to investigate these errors.
The project group conducted a retrospective analysis of transfusion-related QARs conducted between January 2011 and March 2019 that revealed failures within the verification process (i.e. nurses cross-checking prior to administration) as the most common contributing factor (45%). Similarly, a review of the organization’s reported RLS events involving blood components or products intended for transfusion over two years (January 2018–December 2019) revealed that the two most common categories for these reports were administering (i.e. the nursing role from cross-checking blood to actual transfusion) (27%), and preparing, dispensing or packaging blood and blood products in the lab (21%, see Figure 1). It is important to note that reporting events into RLS is voluntary, and, therefore, the numbers of reports do not reflect the total number of events that may have occurred.
Based on the QAR and RLS data, the project team focused on evaluating the processes related to administering/verification blood transfusions, and the lab processes of preparing, dispensing and packaging blood and blood products intended for transfusion.
A multimodal approach was used to identify the root causes contributing to blood administration, labelling and dispensing errors. First, in situ (i.e. within the clinical environment) simulations were facilitated in both a haematology/oncology inpatient unit and a laboratory Transfusion Medicine department. The objective was to identify active and latent system-level contributing factors and develop recommendations to mitigate patient safety risks. A secondary objective was to use the findings to inform the review and revision of the Transfusion of Blood Components and Products Policy and Procedure.
Our core project team consisted of a simulation consultant, a human factors specialist and two provincial patient safety specialists and executive sponsors. In addition to the core team, two clinical educators from a haematology/oncology inpatient unit and three Transfusion Medicine department members acted as content experts for the simulations to answer any unit-based questions from staff during the debriefings. Frontline clinical staff, as described below, participated in simulation sessions in the respective areas.
This project was conducted from October 2019 to February 2020 at a large regional hospital. Two different in situ (i.e. an unoccupied regular clinical room and laboratory dispensing area) spaces were used for the simulations with each respective team. Both environments, being in situ, were functioning that day as they normally would with the same flow of people traffic in the environment. Although an unoccupied patient care clinical room was used, on other days, this room was a regular patient care space. The haematology/oncology inpatient unit was equipped with a patient bed, an intravenous pump, all necessary supplies to set up a blood product for administration, a laptop for participants to review patient information and physician orders on the electronic health record. A simulated blood bag was available in the unit’s actual fridge for retrieval, coloured blue and marked for simulation only. All working staff were made aware, and this was done to ensure that patient safety was of the highest priority and to anticipate any potential safety implications of using simulated blood in a live clinical environment. The debrief sessions were conducted in an adjacent room, and an administrative assistant took notes during the debrief sessions. The in situ laboratory environment was used for the Transfusion Medicine team using a similar set-up.
Simulation scenarios were co-designed with the input of the project team and clinical educators to ensure the process steps and clinical context aligned. The scenario for the nursing unit focused on a scenario handover of a common patient history and daily situation on the unit (49-year-old female with a haemoglobin of 75g/L requiring 1 unit of packed red blood cells with stable vital signs provided) requiring an imminent blood transfusion at the start of a new shift. The appropriate order was available in a lab IT training environment (an exact replica of the live order entry environment) and ready for checking the order. The participant was then to go through the usual and frequent process steps, including the cross-checking of blood products with another team member, the initiation (priming/line preparation) and administration of blood to a ‘patient’ (i.e. an embedded actor), the ability to locate and follow the provincial blood transfusion policy, the recognition of a transfusion reaction and the ability to locate and use the transfusion reaction chart and post documentation. The embedded actor was a project team member who was familiar with the unit and patient population and was scripted to ask common questions and provide cues (e.g. How long does this usually take? And later once the transfusion had started: ‘I’m feeling really, really cold and uncomfortable’), with a perceived normal level of emotion and response in conversation during the process.
The lab transfusion medicine session focused on two scenarios, one targeting a ‘day in the life’ blood dispense, followed by a more complex (but routine) scenario of multiple simultaneous dispenses, including orders for repeating massive transfusion protocol packs. The first scenario, a routine single blood dispense within the lab environment, was a common occurrence at the start of a shift. The second scenario, also typical to this environment, was designed to test the blood transfusion during the ‘routine emergent times’ with multiple massive transfusion orders and immediate or rush orders arriving simultaneously and the routine distractions in the live environment. Usual distractions were enabled such as one embedded participant patient arriving at the counter to pick up their blood product and interrupting the dispenser once, as well as a phone call from the emergency and operating room to ask about the orders. All other routine lab operations continued in the same environment.
Participants were encouraged to identify any other challenges or uncertainties related to blood transfusion and its policy during the simulation session.
Each session ran for approximately 60 minutes (a 20-minute simulation followed by a 40-minute debriefing), beginning with a pre-briefing by a simulation specialist or clinical educator trained in simulation-based methods. The project’s purpose was explained, including the focus on uncovering system and process challenges, improving understanding of what was working well related to blood transfusion and opportunities to improve the associated policy. Participants were also informed of the project’s goal, which was to improve patient safety during blood transfusion by identifying opportunities to improve the policy and procedure.
After each simulation, a scripted debriefing was used following the PEARLS for systems integration (PSI) debriefing framework [12]. PSI debriefing uses common debriefing strategies (plus/delta, focused facilitation and directive feedback) to identify systems issues and threats, and maximize improvements in patient safety and quality. Key pre-determined objectives (as listed above) were explored, and debriefing feedback was anonymized, collected via an administrative assistant typing real-time, and later themed into systems issue categories.
The haematology/oncology inpatient unit was chosen due to its high volume of blood transfusions (e.g. self-reported average of three transfusions administered/day/nurse). In total, 68 nurses participated in over 12 simulation sessions. This was all of the nurses who could be scheduled during the sessions while still maintaining clinical care on the unit. The inpatient simulation sessions included a structured debriefing, targeting the highest risk and highest impact workflow steps surrounding blood administration, the current blood transfusion policy, general practice on the unit and any other concerns brought forward by participants.
In the Transfusion Medicine department, a total of eight lab technologists participated over two simulation days who were all full time lab technologists working in this lab. Each group of participants completed two scenarios. The lab simulation sessions also included a similarly structured debriefing to obtain user feedback from the participants.
Once the simulation sessions were completed, debriefing data from both areas were categorized and rated for risk using the FMEA methodology.
FMEA is a systematic, proactive method for evaluating where a process, such as administering blood, might fail and what the impact of the failure might be, including the impact on the patient [13]. The primary goals of an FMEA are to identify potential failures before they occur and either prevent the failure from reaching the patient or reduce the impact on the patient if or when the potential failure occurs.
All of the information collected during the simulation sessions and in the subsequent debriefings was transcribed, sorted and classified in relation to the process steps. Relevant failure modes were then identified from these data for the FMEA. All potential failure modes identified were scored using the FMEA Risk Matrix Tool according to (i) severity or impact of the failure, (ii) frequency of the failure and (iii) the likelihood of detection using a scale of 1 to 5 resulting in an overall criticality score, which is calculated by multiplying the severity, frequency and detectability scores (Table 1).
Severity of Failure | Frequency of Occurrence (number of times this could occur) | Detectability (Likelihood of detecting the failure or effect before it occurs) |
---|---|---|
1-No Effect: not noticeable, no effect on the patient. | 1 – Yearly | 1 – Always |
2-Slight: minor effects on patient, provider and process. No injury or increase in the level of care. | 2 – Monthly | 2 – Likely |
3-Moderate: performance loss or increased level of care to patient (i.e. hospitalization or increased length of stay). | 3 – Weekly | 3 – Unlikely |
4-Major: high degree of performance loss, permanent impact on patient, reduced function, surgical intervention. | 4 – Daily | 4 – Never |
5-Severe or Catastrophic: death or major, permanent loss of function. | 5 – Hourly |
Scoring of the risk components was done using established FMEA methodology definitions for each component and nominal group technique (structured discussion/brainstorming) to reach a consensus on scores based on the expertise and knowledge of the team. Patient safety data, such as Reporting and Learning System (RLS) and quality assurance review reports, also supported severity and frequency estimates. Detectability scores were determined by identifying existing controls that increase the likelihood of preventing or detecting the failure mode.
Failure modes with a severity level of 5 (regardless of overall criticality score) or a criticality score of 15 or greater, based on a consensus discussion and review with the project team and clinical leads, served as the threshold for risk mitigation and action.
The system issues identified during the FMEA were further categorized into Policy and Procedure, Clinical Practice and Process and Labelling in the Lab Environment-related issues.
Validation of scores occurred with clinical nurse educators and others experienced in the process (nurses, lab techs) and was also confirmed with participants during simulation, either through direct observation or through debrief. Without going into details of the FMEA process (as the clinical team members were not familiar with FMEA, and that was not the purpose), the clinical stakeholders were presented with the process steps and risks identified by the staff during the simulation debriefs. They were then asked to identify other risks not captured within each step, the risks representing the greatest threat to patient safety and the potential recommendation to address and mitigate the identified risks. There was an overall agreement between the stakeholders and the FMEA results regarding the most severe patient safety risks. The project team used these data to develop system issue categories and resulting recommendations for improvement work.
Table 2 illustrates the failure modes identified during the inpatient unit simulation and subsequent debriefing sessions, including their associated FMEA criticality score, as well as the possible causes of each failure mode, the potential safety threats and the resulting recommendations for consideration.
Failure Mode (Criticality Score) |
Possible Cause of Failure | Potential Safety Threats | Recommendations |
---|---|---|---|
Incorrect or incomplete verification of orders (nursing) (32) | –Process of matching product to order in policy and procedure leaves room for interpretation –Variation in nursing practice |
–Patient receives incorrect blood product –Increased risk of necessary steps being missed, and errors not being caught |
Review and revise the blood transfusion policy and procedure to provide clarity and step-by-step instruction (where applicable) to sections identified as troublesome for end users. |
Incorrect patient identification (32) | –Best practice for cross-checking patient identification unclear –Variation in the interpretation of policy/procedure |
||
Patient receives pre-medication for potential transfusion reaction when not required (24) | –Variation in physician approach and in ordering practices –Nursing unclear of expectations, as no standardized practice |
–Patient could react to pre- medication when it wasn’t needed –Pre-medication could mask a transfusion reaction if given when not required |
–Consider the use of standardized blood administration order sets based on best practice guidelines to reduce variability in practice. |
Patient does not receive pre-medication when required (30) | –Patient experiences unnecessary reaction | ||
Incorrect priming/ preparation of lines (27) | –Priming intravenous lines outside of infusion pumps, thus bypassing built-in safeguards (i.e. pre-programmed drug library for blood administration) –Procedure does not provide specific instruction related to this step –Staff unaware of ability and/or necessity to use drug library for blood administration |
–Using the pump outside of the drug library increases the risk that the blood will be given at an inappropriate rate | Review and revise the blood transfusion policy and procedure to provide clarity and step-by-step instruction (where applicable) to sections identified as troublesome for end users. Pump should never be run out of drug library when blood is connected, including for any priming or flushing needed. Consider repeating simulation sessions to evaluate the effectiveness of revisions made to the policy and procedures prior to implementation or as part of the evaluation process post implementation to inform future revisions. Review the education and training provided to nursing staff to ensure ongoing support for continued competency in blood administration with emphasis on high-risk and/or error-prone process steps known to contribute to patient safety events. Identify and implement strategies to increase awareness and ease of access to transfusion reaction resources. Develop standardized escalation of care policies, procedures or guidelines with organization wide implementation and education. |
–Correct process to follow when infusing multiple units in succession not clear | –Back-to-back transfusions may cause a reaction that could be missed; it may be unclear which unit caused the reaction | ||
Transfusion of expired products (16) | –Policy unclear for allowable storage time outside of fridge for different blood products/ components | –Transfusion of an expired product –Wasted product –Delay in treatment –Increased workload |
|
Unrecognized or delayed response to transfusion reaction (24) | –Unclear/complicated decision support tools –Policy/procedure not easily accessible –Difficulty reaching the most responsible physician |
–Adverse reactions are not investigated –Delay in treating adverse reaction –Patient at risk for future transfusion reactions if reactive unit not discovered |
|
Transfusion reaction not reported/documented (24) | –Variability in documentation of transfusion reaction –Process ambiguous in policy/ procedure |
Significant findings were a lack of policy clarity in several critical steps of the blood transfusion process. The absence of a clear step-by-step procedure for the double-checking process and requirement for verification by two healthcare providers resulted in wide variation in nursing practice. This increased the risk of patient safety events. Other findings included trouble with access and location of the blood transfusion policy, related resources (e.g. transfusion reaction chart) and a lack of standardized guidelines related to prescribing medications for prevention and managing of potential transfusion reactions. All of these and other findings increased the risk of treatment delays, errors and patient safety threats.
Table 3 illustrates the FMEA results identified during the transfusion medicine lab simulation and debriefing sessions. Within the lab, interruptions and distractions were identified as the most common precursor to processing or labelling errors. The functional layout of the blood dispensing area (i.e. clutter, distractions and space limitations) contributed to the increased level of risk. The need for formal documented processes and clear roles and responsibilities was identified as another opportunity for improvement.
Failure Mode (Criticality Score) |
Possible Cause of Failure | Potential Safety Threats | Recommendations |
---|---|---|---|
Order not processed or incorrectly processed (16) | –Unclear roles and responsibilities of team in times of extreme workload –Variability in work processes –Physical space limitations –Interruptions and distractions |
–Increased risk of orders being missed or processed incorrectly, or dispensed to the wrong patient, due to variation in workflow processes –Errors due to interruptions, distractions, clutter and congestion |
Review current team roles and responsibilities, and related policies or procedures, to ensure standardized processes and provide role clarity. Investigate opportunities within the existing infrastructure to redesign the dispensing space to reduce clutter, limit distractions and improve workflow. |
Incorrect product selected/prepared (32) | –Interruptions and distractions by other team members –Lack of separate work area to complete this step makes concentration difficult, interruptions frequent –Task interruption by Porters coming to pick up products |
||
Product incorrectly labelled (15) | –Variability in work processes –Interruptions and distractions –Physical space limitations |
||
Dispensing errors (45) | –Complexity of current electronic system and need to move between multiple windows and screens –Limited error proofing functionality within current technology –Interruptions and distractions |
–Increased risk for an incorrect blood product to be dispensed | Explore options to utilize technology to incorporate automated solutions (e.g. alerts, forcing functions), that address high-risk and/or error-prone process steps, within the electronic information system. |
Despite completing our project prematurely due to the pandemic, multiple outcomes and trends were reported following project completion. In the lab, the physical space was modified in multiple ways, including a half wall along the front of the lab, with a whole permanent Plexiglas wall above it to decrease noise and distractions from people walking by. Staff were more aware of the issues and reported fewer distractions. The printer was rerouted to reduce traffic flow through the dispensing area from the rest of the lab. Overall, fewer dispensing errors occurred in a four-month period after the changes (17 errors) compared with the same period prior to the changes (24 errors). None of the 2021 dispensing errors had reported the potential for significant impact.
The health authority took several actions, including redesigning the blood administration policy and procedure to combine them into one document, targeting the many associated resources that caused confusion (e.g. human factors review of the blood transfusion reaction chart and redesign) and specifying steps in the policy, such as the specifics of always needing a two-person check. Finally, work is underway to modify the transfusion website and change the transfusion blood tag so that a nurse can see the label and bag simultaneously to reduce error. All of these improvement measures are significant to reduce the potential for harm in blood transfusion.
Our project scope was constrained due to shifting priorities as a result of COVID-19 pandemic response activities. It was planned to repeat simulations on an inpatient unit with very few transfusions to validate the required improvement further. However, by conducting the simulations with nurses very experienced in blood administration, it is reasonable to conclude that nurses with less experience would also struggle with these same aspects, thereby making the findings from this project generalizable to other units.
Despite previous patient safety and quality improvement efforts, the administration of blood and blood products continues to be a high-risk activity across large health care organizations, as indicated by our safety reporting and quality assurance reviews. Our collaborative patient safety, simulation and human factors project was undertaken to proactively identify both active and latent factors contributing to adverse events and identify recommendations using FMEA methodology to improve patient safety, including revisions to the provincial blood transfusion policy and procedure.
The premise that ‘work as imagined’ does not equal ‘work as done’ relative to policy, procedure, practice and environment was reiterated throughout this project. For example, the assumption that the policy and procedure had enough detail and were usable for the well-trained staff was identified as problematic on multiple occasions. Ensuring human factors principles and methods such as SIS/SFD to test and implement new and existing protocols is a key to ensuring that they are effective and safe. We observed a wide variation in practices and processes through simulation, and although some variation is expected and even necessary in healthcare delivery, unwarranted variation can lead to uncertainty in following a best practice, staff confusion and patient safety risks. These variations were debriefed with the teams, which in turn informed the improvement work needed.
In the lab setting, interruptions, a high number of distractions and limitations related to existing infrastructure and current technology contributed to increased patient safety risk. Utilizing simulation to uncover systems issues increased awareness and recognition of potential inefficiencies related to workflow and challenges. This included those associated with not only the physical layout of the space but also highlighted areas of high traffic flow resulting in multiple distractions. The interruptions to workflow included both lab and other hospital personnel, such as porters picking up blood and other environmental distractions (e.g. accessibility of printers, the use of various phone ring tones).
When examining the clinical practices and environments through simulation and debriefing, it is important to take a systems approach to probe the system to understand issues and latent safety threats. Ideally, ‘systems thinking’ will ensure project team members and clinicians alike search for root causes not directed at the individual’s performance but rather how the system elements supported them, or not, in their daily work. For example, it was not that nurses lack knowledge of administering blood or blood products on an inpatient unit, but ambiguous elements of a policy or procedure, variable process steps, information technology issues and environmental issues that can lead to errors.
Using the FMEA methodology was a novel approach to support a rigorous analysis of debriefing data. It supported the identification and categorizing of the highest risk issues but was time-consuming to ensure thorough categorization, rating discussions and validation with input from the system users. The FMEA process identified the highest risks and patient impact issues, which were further validated by stakeholders unaware of the FMEA process. FMEA is a valuable tool to use when time and resources allow and when deemed highly beneficial for higher-risk protocols such as this one.
Our collaborative project used existing safety data along with simulation, patient safety and human factors methodologies, combined with FMEA, which resulted in a targeted approach to understanding system-level issues surrounding safe blood transfusion and labelling of blood and blood products. Several user-informed recommendations improved the policy and procedure redesign, the clinical environment and practices surrounding transfusion safety for a large health organization.
The authors would like to thank all participating units and team members for supporting this work, and Andrea Faid of the eSIM Program of Alberta.
All authors contributed to writing, editing and reviewing this work. TM collated past RLS reports and data to inform the work. MD and AR completed referencing. KO lead the data collection and all authors contributed to data analysis and reporting.
There has been no funding for this work.
None.
The project followed the successful completions of the ‘A Project Ethics Community Consensus Initiative – ARECCI’ screening tool (https://albertainnovates.ca/our-health-innovation-focus/a-project-ethics-community-consensus-initiative/arecci-ethics-guideline-and-screening-tools/). This decision support tool identified the primary purpose of the project as quality improvement / program evaluation and that the project involves minimal risks; therefore, review by the Research Ethics Board was not required.
MD is faculty for Healthcare Systems Simulation International, which provides consulting and education services on systems-focused simulation and debriefing for quality and safety staff, simulation enthusiasts and others in health care. There are no competing interests.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.