T Walsh, P Beatty
alarms, anaesthesia, anesthesia, anesthesiology, complications, critical care medicine, intensive care medicine, regional anesthesia, trauma
T Walsh, P Beatty. Development Of A Scale Of Severity For Clinical Situations. The Internet Journal of Anesthesiology. 1999 Volume 5 Number 1.
Recent research has documented the inadequacies of auditory alarms in both the Intensive Care Unit and Operating Room (1). 'Urgency mapping', whereby the objective urgency of the situation is mapped to the subjective urgency induced in the hearer, is one recommendation that has been made for improving auditory alarms in anaesthesia. The paper reports an initial attempt to extend this principle to a variety of possible effectors of anaesthetist actions, by establishing a list of clinical situations ranked by anaesthetists. A questionnaire was designed to assess subjective ratings of severity, urgency, anxiety and attention. Anaesthetists' assessments of twenty-five clinical situations were found to be consistent, with the severity and anxiety and urgency and attention scales closely related. An objective risk score was calculated based on published data (2, 3), which failed to correlate with subjective estimates of worst case outcome.
Many authors have pointed to the inadequacies of auditory alarms used on patient monitoring systems in the Intensive Care Unit (ICU) and Operating Room (OR). Hass (1) summarising the work of Edworthy and Meredith (4), details five critical problems associated with auditory alarms in the ICU and OR:
Acoustic similarity of alarms.
Abundance of alarms.
Inappropriate mapping levels.
Incidence of false alarms.
All these lead to loss of confidence in the alarm systems, as witnessed by the disarming of alarms or refusal to use them (5). Based upon previous research (6), Edworthy introduced the concept of ‘urgency mapping’ as a means of addressing some of these problems. She defined ‘urgency mapping’ as ‘manipulating acoustic parameters in order to relate the perceived urgency of the alarm to the urgency of the situation’. Thus the sound chosen for the alarm should map appropriately the objective risk of damage to the patient to the subjective urgency induced in the hearer. This principle may be generalized to all features of instruments in the ICU or OR that seek to warn or advise clinical staff about the condition of the patient. However, in this more generalized form, the concept of ‘urgency mapping’ is problematic, as there is no truly objective measure of risk.
The best estimates of risk for patients in a clinical situation are given by morbidity or mortality figures obtained from clinical audit. Where such figures exist, they provide information as to the likelihood of a specific undesired or significantly negative outcome occurring. But such studies have limitations. For instance they may rely on retrospective analysis of incident records, which may under-report some classes of incident. A more reliable use of clinical audit evidence is to establish the effects of undesired situations that remain uncorrected. These can range from transient abnormality with no permanent effect, through potentially permanent disabling damage, to death. Such classifications of the outcomes of undesired situations are used widely in critical incident reporting systems such as the on-line ‘Critical Incident Reporting System (CIRS) used to support incident reporting within Swiss Departments of Anaesthesia (7). Thus, though we may have reservations about the use of clinical audit information as a basis for the objective assessment of risk, it potentially provides us with a measure of risk that meets Edworthy’s requirements for a total hazard score, i.e. ‘The total hazard score for specific situations is the objective measure to which warnings should be matched’ (8).
A similar problem arises as to how the ‘perceived’ urgency can be quantified? Edworthy (9, 10) allowed users to map different candidate sounds to their subjective urgency by describing them using a list of adjectives whose subjective urgency had already been assessed by other workers. She thus avoided the problem of having to explicitly quantify the subjective ‘urgency’ opinion of the users that, being based on experience and past events, may differ widely across clinical disciplines and levels of experience. In extending the ‘urgency mapping’ principle to features far more complex than simple alarm sounds, we cannot avoid having to measure subjective ‘urgency’ explicitly. Thus constructing a suitable ‘urgency mapping’ of adverse events to warnings can only be undertaken once a generally accepted and calibrated hierarchy of adverse events has been established, and once any possible differences between subjective and objective opinion have been assessed within this hierarchy.
There is one more problem that needs to be addressed in establishing this hierarchy: ‘urgency’ is not the only quality of a situation that has an effect on the actions taken by an anaesthetist. A review of critical incident reports, from sources such as closed claims studies, indicate that other subjective effectors may come into play, notably the perceived severity of the situation of the patient, the anxiety induced in the anaesthetist, and the level of attention required to treat the situation. Thus the hierarchy needs to address these effectors, which may each need their own equivalent of ‘urgency mapping’.
The research reported details the methodology and preliminary results in the development of a questionnaire to establish the required hierarchy. The questionnaire designed addressed three specific research aims. The first was to establish a subjective scale of clinical situations as judged by anaesthetists. The second, based on the arousal strength work of Wogalter and Silver (11), determined whether the perceived importance of these clinical situations differs with respect to the four effectors of severity, urgency, anxiety and attention. The final aim was to determine whether the subjective evaluation of severity and likelihood of worst-case outcomes for the situations as assessed by the anaesthetists corresponded to the objective assessment of severity shown by clinical audit studies of morbidity and mortality.
Questionnaire Design and Development
The final version of the questionnaire is shown in:
An iterative process of questionnaire development and refinement was used.
An initial pilot questionnaire was constructed using nine commonly cited clinical situations chosen from the CEPOD, CIRS and RCA audits (2, 7, 12), which could be displayed on monitors or simulated on computer. The initial list was kept short as it was considered important that the anaesthetists themselves generated the clinical situations in their own terms. The questionnaire included space for the participants to add any clinical situations that they felt were missing and to score them. The rubric of the questionnaire made it clear that the participants should consider they were
This preliminary version of the questionnaire was administered to ten trainee anaesthetists on the FRCA course at Manchester University. After examination of the completed questionnaires, and following discussion with a senior anaesthetist, the first proper version of the questionnaire was constructed and sent to 100 participants randomly selected from a database of responders to a previous study (13). The questionnaire was successively cycled to new samples from the same database, until it reached a ‘steady state’. This was achieved after three cycles, with the final version of the questionnaire consisting of 25 clinical situations. The principal changes made to the questionnaire involved the addition, deletion, and clarification of clinical situations based on comments made by the respondents and on the reliability of responses as judged by simple descriptive statistics. Minor changes were made to the layout and to instructions provided on how to complete the questionnaire. A summary table of the changes made after each successive cycle appears in:
Analysis and Results
Response rates to each cycle were 52%, 47% and 40% respectively. The declining response rate was attributed to the ageing address database, and the transient nature of trainees within the database at a given hospital.
Given the ordinal nature of the data, analysis proceeded using descriptive non-parametric statistics (mode, median and inter-quartile range) calculated for each situation on the four effectors. The results are summarised using box and whisker plots in figures 1-4 for the results from the final cycle. Outliers, cases with values between 1.5 and 3 times the inter-quartile range from the nearer quartile, are depicted by a ‘0’. Extremes, cases with values more than 3 times the inter-quartile range from the nearer quartile, are depicted by a ‘*’.
To produce a more usable and sensitive scale Friedman Ranks were calculated for the situations. The scales produced are shown in figures 5-8.
Scatterplots of the median scores (not shown) revealed a positive linear relationship for all six pairings. Correlation coefficients were calculated to determine the degree of association between effectors. These results are shown in table 1 broken down by situation.
Finally, two overall risk scores were derived. The Subjective Risk Score (SRS) was calculated by multiplying the scores for worst-case outcome with the scores of likelihood of the worst-case outcome occurring. The Objective Risk Score (ORS) was calculated in a similar way from figures obtained from the 1996/1997 CEPOD enquiry (2) and the 1993 Australian Incident Monitoring Study (14,15,16). Comparing the Friedman Rank for the severity effector with the calculated risk scores revealed a statistically significant correlation of r = 0.73 with the SRS, but no significant correlation (r = -0.15) with the ORS.
Generally the box and whisker plots of figures 1-4 show that for each of the individual effectors, responses to the majority of clinical situations are concentrated at the upper end of the scale. This reflects the nature of the clinical situations appearing in the list, which are all relatively urgent and likely to lead to death if not corrected. The effectors with the highest dynamic range are severity and anxiety. However, the Friedman Rankings, shown in figures 5-8 demonstrate that all four are capable of forming the basis of sensitive scales.
There is evidence of a positive linear relationship for all six pairings. This is strongest in the ‘severity and anxiety’, and the ‘urgency and attention’ pairings, which have R2 values of 0.904 and 0.896 respectively. This association is reinforced by the results shown in table 1, which shows the matrix of Kendall’s Tau-b correlation coefficients for each situation and each pairing. For the majority of situations, the strongest relationship as indicated by the magnitude of the coefficients is urgency and attention, followed by severity and anxiety. We would suggest that effectively the urgency and attention, and severity and anxiety scales are not truly independent, and that they measure the same effects on anaesthetist response.
Deriving the ORS proved very difficult. The Australian Incident Monitoring Study gave details of critical incident reports on 18 out of the 25 situations included in the final version of the questionnaire, although these were in different sub-groups of the study. Careful examination of the size of each sub-group used was employed to ensure that the rates of incident reporting for each situation were adjusted properly and that the incidents reported were for patients in the ASA I-II category specified by the questionnaire. The study also gave death rates within the situations for those situations where death occurred. Incident reporting studies do not normally report the total number of anaesthetics that given during the period of study. So to derive an estimate of the number of situations expected per million anaesthetics, the rates in the incident study were adjusted appropriately, to give an overall death rate for the Australian study equal to the often quoted CEPOD derived rate of death from critical incidents in anaesthesia of 1:100,000. This adjustment allowed the combination of the Australian data with the CEPOD data on the same basis, i.e. number of occurrences per million anaesthetics of ASA I-II patients. Where more than one estimate of occurrence rate for a situation was available, a mean was taken. Appearance of a situation in the CEPOD report was deemed to be evidence that the worst-case outcome of the situation, in the terms of the questionnaire, was death. This applied to 17 out of the 18 situations for which an estimate of likelihood of occurrence could be derived. The only exception was ‘profound bradycardia’, where the reported estimate of Wang and Hagerdal (17) most closely matched the questionnaire grading of ‘
It is clear from this part of the study that the subjective assessment of severity by anaesthetists is reasonably consistent. However, deriving what we hoped was going to be an objective risk score proved difficult. The lack of a statistically significant correlation between the subjective effector and the calculated objective risk suggests that either the subjective scales bear little or no relation to the real risk for a given incident, or that the estimates of the likelihood of the situations occurring cannot be reliably obtained from the data presented in the literature. Before a judgment between these two interpretations can be made, it is clear that a separate study needs to be performed to establish the incidence of these situations in real life.