Introduction

Many authors have pointed to the inadequacies of auditory alarms used on patient monitoring systems in the Intensive Care Unit (ICU) and Operating Room (OR). Hass (₁) summarising the work of Edworthy and Meredith (₄), details five critical problems associated with auditory alarms in the ICU and OR:

• Acoustic similarity of alarms.

• Abundance of alarms.

• Irritating alarms.

• Inappropriate mapping levels.

• Incidence of false alarms.

All these lead to loss of confidence in the alarm systems, as witnessed by the disarming of alarms or refusal to use them (₅). Based upon previous research (₆), Edworthy introduced the concept of ‘urgency mapping’ as a means of addressing some of these problems. She defined ‘urgency mapping’ as ‘manipulating acoustic parameters in order to relate the perceived urgency of the alarm to the urgency of the situation’. Thus the sound chosen for the alarm should map appropriately the objective risk of damage to the patient to the subjective urgency induced in the hearer. This principle may be generalized to all features of instruments in the ICU or OR that seek to warn or advise clinical staff about the condition of the patient. However, in this more generalized form, the concept of ‘urgency mapping’ is problematic, as there is no truly objective measure of risk.

The best estimates of risk for patients in a clinical situation are given by morbidity or mortality figures obtained from clinical audit. Where such figures exist, they provide information as to the likelihood of a specific undesired or significantly negative outcome occurring. But such studies have limitations. For instance they may rely on retrospective analysis of incident records, which may under-report some classes of incident. A more reliable use of clinical audit evidence is to establish the effects of undesired situations that remain uncorrected. These can range from transient abnormality with no permanent effect, through potentially permanent disabling damage, to death. Such classifications of the outcomes of undesired situations are used widely in critical incident reporting systems such as the on-line ‘Critical Incident Reporting System (CIRS) used to support incident reporting within Swiss Departments of Anaesthesia (₇). Thus, though we may have reservations about the use of clinical audit information as a basis for the objective assessment of risk, it potentially provides us with a measure of risk that meets Edworthy’s requirements for a total hazard score, i.e. ‘The total hazard score for specific situations is the objective measure to which warnings should be matched’ (₈).

A similar problem arises as to how the ‘perceived’ urgency can be quantified? Edworthy (₉, ₁₀) allowed users to map different candidate sounds to their subjective urgency by describing them using a list of adjectives whose subjective urgency had already been assessed by other workers. She thus avoided the problem of having to explicitly quantify the subjective ‘urgency’ opinion of the users that, being based on experience and past events, may differ widely across clinical disciplines and levels of experience. In extending the ‘urgency mapping’ principle to features far more complex than simple alarm sounds, we cannot avoid having to measure subjective ‘urgency’ explicitly. Thus constructing a suitable ‘urgency mapping’ of adverse events to warnings can only be undertaken once a generally accepted and calibrated hierarchy of adverse events has been established, and once any possible differences between subjective and objective opinion have been assessed within this hierarchy.

There is one more problem that needs to be addressed in establishing this hierarchy: ‘urgency’ is not the only quality of a situation that has an effect on the actions taken by an anaesthetist. A review of critical incident reports, from sources such as closed claims studies, indicate that other subjective effectors may come into play, notably the perceived severity of the situation of the patient, the anxiety induced in the anaesthetist, and the level of attention required to treat the situation. Thus the hierarchy needs to address these effectors, which may each need their own equivalent of ‘urgency mapping’.

The research reported details the methodology and preliminary results in the development of a questionnaire to establish the required hierarchy. The questionnaire designed addressed three specific research aims. The first was to establish a subjective scale of clinical situations as judged by anaesthetists. The second, based on the arousal strength work of Wogalter and Silver (₁₁), determined whether the perceived importance of these clinical situations differs with respect to the four effectors of severity, urgency, anxiety and attention. The final aim was to determine whether the subjective evaluation of severity and likelihood of worst-case outcomes for the situations as assessed by the anaesthetists corresponded to the objective assessment of severity shown by clinical audit studies of morbidity and mortality.

Method

Questionnaire Design and Development

Analysis and Results

Response rates to each cycle were 52%, 47% and 40% respectively. The declining response rate was attributed to the ageing address database, and the transient nature of trainees within the database at a given hospital.

Given the ordinal nature of the data, analysis proceeded using descriptive non-parametric statistics (mode, median and inter-quartile range) calculated for each situation on the four effectors. The results are summarised using box and whisker plots in figures 1-4 for the results from the final cycle. Outliers, cases with values between 1.5 and 3 times the inter-quartile range from the nearer quartile, are depicted by a ‘0’. Extremes, cases with values more than 3 times the inter-quartile range from the nearer quartile, are depicted by a ‘*’.

To produce a more usable and sensitive scale Friedman Ranks were calculated for the situations. The scales produced are shown in figures 5-8.

Scatterplots of the median scores (not shown) revealed a positive linear relationship for all six pairings. Correlation coefficients were calculated to determine the degree of association between effectors. These results are shown in table 1 broken down by situation.

Figure 1

Table 1 Kendall’s Tau-b correlation coefficients

Figure 2

Finally, two overall risk scores were derived. The Subjective Risk Score (SRS) was calculated by multiplying the scores for worst-case outcome with the scores of likelihood of the worst-case outcome occurring. The Objective Risk Score (ORS) was calculated in a similar way from figures obtained from the 1996/1997 CEPOD enquiry (2) and the 1993 Australian Incident Monitoring Study (₁₄,₁₅,₁₆). Comparing the Friedman Rank for the severity effector with the calculated risk scores revealed a statistically significant correlation of r = 0.73 with the SRS, but no significant correlation (r = -0.15) with the ORS.

Discussion

Generally the box and whisker plots of figures 1-4 show that for each of the individual effectors, responses to the majority of clinical situations are concentrated at the upper end of the scale. This reflects the nature of the clinical situations appearing in the list, which are all relatively urgent and likely to lead to death if not corrected. The effectors with the highest dynamic range are severity and anxiety. However, the Friedman Rankings, shown in figures 5-8 demonstrate that all four are capable of forming the basis of sensitive scales.

Figure 3

Figure 1 Severity of situation

Figure 4

Figure 2 Anxiety experienced

Figure 5

Figure 3 Urgency of response required

Figure 6

Figure 4 Attention required

Figure 7

Figure 5 Friedman rank severity of situation

Figure 8

Figure 6 Friedman rank anxiety experienced

Figure 9

Figure 7 Friedman rank urgency of response required

Figure 10

Figure 8 Friedman rank attention required

There is evidence of a positive linear relationship for all six pairings. This is strongest in the ‘severity and anxiety’, and the ‘urgency and attention’ pairings, which have R2 values of 0.904 and 0.896 respectively. This association is reinforced by the results shown in table 1, which shows the matrix of Kendall’s Tau-b correlation coefficients for each situation and each pairing. For the majority of situations, the strongest relationship as indicated by the magnitude of the coefficients is urgency and attention, followed by severity and anxiety. We would suggest that effectively the urgency and attention, and severity and anxiety scales are not truly independent, and that they measure the same effects on anaesthetist response.

Deriving the ORS proved very difficult. The Australian Incident Monitoring Study gave details of critical incident reports on 18 out of the 25 situations included in the final version of the questionnaire, although these were in different sub-groups of the study. Careful examination of the size of each sub-group used was employed to ensure that the rates of incident reporting for each situation were adjusted properly and that the incidents reported were for patients in the ASA I-II category specified by the questionnaire. The study also gave death rates within the situations for those situations where death occurred. Incident reporting studies do not normally report the total number of anaesthetics that given during the period of study. So to derive an estimate of the number of situations expected per million anaesthetics, the rates in the incident study were adjusted appropriately, to give an overall death rate for the Australian study equal to the often quoted CEPOD derived rate of death from critical incidents in anaesthesia of 1:100,000. This adjustment allowed the combination of the Australian data with the CEPOD data on the same basis, i.e. number of occurrences per million anaesthetics of ASA I-II patients. Where more than one estimate of occurrence rate for a situation was available, a mean was taken. Appearance of a situation in the CEPOD report was deemed to be evidence that the worst-case outcome of the situation, in the terms of the questionnaire, was death. This applied to 17 out of the 18 situations for which an estimate of likelihood of occurrence could be derived. The only exception was ‘profound bradycardia’, where the reported estimate of Wang and Hagerdal (₁₇) most closely matched the questionnaire grading of ‘potentially permanent but not disabling damage’. Multiplying the score from the grades from the questionnaire (i.e. 1-6) for severity of worst case outcome, with the estimate of likelihood of occurrence per million anaesthetics, gave a risk score.

It is clear from this part of the study that the subjective assessment of severity by anaesthetists is reasonably consistent. However, deriving what we hoped was going to be an objective risk score proved difficult. The lack of a statistically significant correlation between the subjective effector and the calculated objective risk suggests that either the subjective scales bear little or no relation to the real risk for a given incident, or that the estimates of the likelihood of the situations occurring cannot be reliably obtained from the data presented in the literature. Before a judgment between these two interpretations can be made, it is clear that a separate study needs to be performed to establish the incidence of these situations in real life.