User Attitudes To Computer-based Decision Support In Anesthesia And Critical Care: A Preliminary Survey
alarms, anesthesia, anesthesiology, computers, critical care medicine, education, electronic publication, human factors, intensive care medicine, internet, multimedia, online, peer-review, regional anesthesia, trauma
P Beatty. User Attitudes To Computer-based Decision Support In Anesthesia And Critical Care: A Preliminary Survey. The Internet Journal of Emergency and Intensive Care Medicine. 1998 Volume 3 Number 1.
This paper reports the results of a preliminary study to validate the use of a
questionnaire to measure the preferences, prejudices and expectations of clinic
al staff to the attributes of computer-based support systems for use at the bed
A Likert style survey was circulated to 207 anesthetists, critical care nurses
and other members of clinical staff involved in anesthesia and critical care. C
hoices concerning design philosophy of such systems and detailed choices of suc
h issues as alarms, data presentation, and interpretation of physiological meas
urements were presented. Statistical analysis identified overall preferences, d
ifferences between professional groupings and differences between those special
izing in general anesthesia or critical care.
The results indicated that staff would favor characteristics in systems that gu
ided rather than controlled their clinical practice. Direct intervention, locki
ng out of staff or other coercive methods that could be implemented in the comp
uter systems were given low rankings within questions. Non-medically trained st
aff were consistently more prepared to give control to computer systems than we
re their medical colleagues. Responders to the survey showed a desire to see co
mputer systems compensate for failures in vigilance or other human factors and
offer expert opinion provided this opinion was offered off line.
Since the late 1960s it has been regularly predicted that computers would have a revolutionary impact on the provision of health care and medical decision-making . In anesthesia and critical care, though microprocessors within instruments have had great impact on physiological measurement, more general applications have been slow to enter clinical service, in spite of a considerable amount of research and development. Research in other areas has found that one of the many reasons for the slow introduction of such systems is a failure to take account of user attitudes and expectations. The purpose of this study was to validate a simple questionnaire-based research tool that could be used to investigate user attitudes and produce preliminary data to guide further investigation in anesthesia and critical care.
A Likert style survey form was circulated to 81 anaesthetists (30 consultant, 51 trainees), 100 critical care nurses and 26 other members of staff including recovery nurses at the Manchester Royal Infirmary, Manchester, UK. Six questions with up to twelve choices in each question, concerning attitudes to the use of and characteristics of computers in anesthesia and critical care were asked. The questionnaire included notes that were intended to give staff unfamiliar with the concept of computer-based decision support systems sufficient background information to answer the questions.
A separate section on the questionnaire posed demographic questions about profession, grade, experience, age and gender. Subjects were also asked to score their access/use of computers and computer literacy on two scales of 0-4. Non-responders to the questionnaire were not followed up individually because initial prototyping of the questionnaire indicated that direct contact lead to discussions about the details of potential decision support systems that could bias the subject’s responses. As a result, the rate of return of questionnaires was low with a mean of 26.6% and a range from 39.2% (trainee anesthetists) to 19.0% (critical care nurses).
To ensure that responders were representative of the population of staff sampled a “demography only” questionnaire was sent to the same target groups after the close of acceptance of full questionnaire returns. This second questionnaire asked for replies only from those who had not responded initially.
All responses were pooled and the relative rank of choices within questions was examined using Friedman rank analysis. This analysis gave an overall measure of the approval/disapproval rating of the choices by the responders. To study differences in attitudes between different user groups, responses were divided into two in two different ways. Firstly, medically trained responders were separated from others to test for differences between professions. Secondly, responders were divided according to those who said they specialized in general anesthesia and those who included intensive care in their activities. Differences between the relative ranking of choices within questions between groups were tested using the Mann Whitney U statistic. The pooled data was examined using factor analysis by maximum likelihood to detect underlying “issues” identified by responders within questions. Clustering of responses was not considered significant unless the maximum likelihood weighting factor for cluster membership exceeded 0.5. Finally, responses were tested for correlation with the self-assessed computer access or use and computer literacy scores using Pearson rank correlation.
No statistical differences between sample or analysis groups in age, years post original professional qualification, years in grade, computer literacy or computer access were detected using T-testing or Mann Whitney U testing where appropriate (p<0.05). Statistically significant differences in gender ratio were found between the groups, with significantly higher proportions of male responders in both the medical analysis group and general anesthesia analysis group.
Question 1: What words should characterize the function of a decision support system?
This question consisted of a list of 12 words suggesting different general characteristics for any decision support system design. The results are shown in table 1 in descending order of rank. The key to all the tables is noted at the end of the paper. No statistically significant differences between the anesthesia (A) and critical care (CC) groups were seen. Statistically significant differences medical (M) and non-medical (NM) groups were found for 4 words “direct”, “manage”, “command” and “insist”. In all these cases the non-medical group gave higher rankings to these words than did their medical colleagues.
Three groupings in the words were identified by factor analysis. “Insist”, “command”, “order” and “manage” were separated from “suggest” and “advise”. The word “direct” was also separated as a group. There was a significant correlation of the word “insist” with computer access scores (r=0.34, p=0.016). Responders indicated a clear preference for systems that advised, suggested or guided, rather than controlled.
Question 2: What clinical tasks should a computer system perform?
In this question choices were given relevant to the types of functional tasks a computer based decision support system might make. Generally the choices would be considered to be general questions of design philosophy or specific areas of overall system design. Table 2 shows the rank order of the responses to questions. Two questions showed differences in responses between the medical and non-medical group. Again, the non-medical group ranked the responses to choices more highly than did their medical colleagues. One statistically significant difference was seen between the anesthesia and critical care groups, with the latter giving a higher ranking to choices concerned with offering advice when requested. Factor analysis found four groups. In group I responders identified offering advice to juniors and compensation for lack of experience as being logically linked, though they showed a certain ambivalence in the ranking of these two responses.
A second grouping consisted of the choice concerned with offering advice when asked combined with that compensating for failures of concentration. Both these responses were given reasonably high overall ranks. However, on the face of it their grouping by responders is somewhat illogical. How can a system compensate for failures of concentration or vigilance and yet still only offer advice when asked? As will be seen in the other results, responders tended to prefer better alarm systems that would compensate for concentration failure but would not offer detailed advice on line. This strategy would make it possible to compensate for failure of vigilance and offer advice off line. The question on lock out received a very low rank, although the very high statistical difference indicated between the medical and non-medical groups suggests that non-medical staff are prepared to restrict the actions of clinicians more than are the clinicians themselves. Taking direct action was given a low rank and seen as a different type of feature design. No correlation with any demographic data and responses was demonstrated.
Question 3: What type of information should a decision support system give?
This question was aimed at eliciting details of how information gathered by a decision support system should be used. Table 3 shows the responses in overall rank order. The highest rank was given to the combination of offering advice with alarms. This was significantly higher than choices limited to advice only. No significant differences with any choice and between groups was demonstrated and only one correlation between the choice “give detailed procedure specific advice” was found with computer literacy score (r = -0.38 and p = 0.004).
Factor analysis indicated that 7 choices could be segmented into four groups. The choices “Offer advice and alarms” and “Give general advice only” were only seen as distinctly different to other choices. As would be expected, responders grouped two of the choices concerned with recording performance details. However, again a certain ambivalence was shown in the ranking of the two choices grouped. It appeared that staff were almost evenly divided between those who would ask that no performance details of the individual operator were ever recorded and that performance should be recorded for junior staff for the perusal of senior staff. The fact that the responders did not group these two questions with the choice concerning a private performance record probably indicates that it is “publication” of the performance that is really the issue of concern. The last group consisted of three responses about how guidance information should be presented. Responders showed a marginal preference for step by step guidance offered off-line at the request of the operator.
Question 4: What sort of expert information should be used in a decision support system?
Present computer technologies offer novel ways of obtaining expert information. It would be possible to design a decision support system that would gather its own information in a structured fashion as it was used and make that the basis of advice. Such a system would be the antithesis of a system where the rules were laid down by a panel of experts or a single senior clinician inside a hospital. Question 4 was aimed at detecting differences in attitudes to different strategies for information gathering.
Table 4 shows that three responses reached statistically significant difference between the medical and non-medical group with the non-medical group again prepared to give higher aproval. The overall rankings indicated that people preferred opinions and information to be based on as broad a basis as was possible. They were very much less keen on information derived from an individual, even when they themselves were that individual. A relatively high ranking was given to inclusion of economic criteria in the expert information used by the system. Factor analysis indicated four groups. One was concerned with where data would be gathered from if it was the system that gathered the data. A second was concerned with the authority giving the information. A third connected questions of economic criteria with information about individual practice, presumably showing the underlying feeling about how economic judgment should be fed back to individuals. Lastly responders differentiated between international advice and advice from UK based sources.
Question 5: How much control should a Decision Support System have?
This question describes choices of design for difference alarms, e.g. general or condition specific with different methods of alarm (sounds, lights, etc). Different combinations of different types were presented as choices.
Table 5 shows the overall ranking to responses. Choices suggesting high levels of automatic control were ranked poorly. Response to the choice “Set off a general audible alarm and put up a condition specific visual alarm” showed a significant statistical difference with the general anesthesia and critical care groups. The critical care group were very much more against the strategy than were the anesthesia group. This response was to be expected, as general audible alarms are perceived as being an annoying problem in critical care. Factor analysis indicated that responders identified three distinct issues. Responses were grouped in accordance with combinations of alarms. Responders considered that the details of audio alarm systems were important. They also identified the ability to change the patient’s treatment as a distinctly different feature. The idea of a private audible alarm was seen as being a separate strategy as well scoring reasonably high on overall ranking. Two significant correlations with computer literacy score and choice were demonstrated. The choice “Set off a non-specific audible alarm heard by other staff and patients” gave r=-0.27, p=0.04, and the choice “Change the patients treatment if alarm is not cancelled and the vital signs show an unequivocally dangerous condition”, gave r=-0.29, p=0.04. A significant correlation with computer access scores was demonstrated for the choice “Under no circumstances change the patients treatment automatically” (r=-0.41, p=0.002). The overall responses to question 5 indicate that staff have very subtle preferences for alarm types and combinations which it would be as well for designers to take into consideration when designing physiological measurement equipment.
Question 6: How should a computer decision support system communicate with the clinician?
This question was aimed at obtaining more information about detailed choices between methods of communication that were not predominantly alarm based, though some alarm related choices were given for comparison purposes. The results are summarised in Table 6. The second highest rank given to the choice that corresponds to the Paterson sounds alarm strategy was negatively correlated with computer literacy (r=-0.33, p=0.019). In other words the more computer literate the responders the less they liked the idea of structured alarm sounds as recommended in international standards. Factor analysis found four groups: offering graphical animated displays, giving instructions using flow diagrams, having spoken alarms and providing extra physiological measurement displays. The overall rankings within these groups indicate that spoken alarms have a low acceptability. Flow diagrams appeared to be the most popular communication method.
* In this case the critical care specialist group ranked this choice significantly below that of the general anesthesia specialists
Offering of extra physiological instrument displays received overall high ranking. This is somewhat at odds with indications from human factors analysis which would suggest that there is more than enough physiological information and numbers inside either anesthesia or intensive care at the present time. Indeed, extra physiological instruments and numbers might be considered to be potentially detrimental to good human factors performance and safety.
Broadly the results of the study can be summarised as follows:
All responders favored characteristics in systems that advised or guided rather than controlled. Direct interventions, lock-out systems, or any other coercive method ranked lower than less coercive methods. This applied equally to medical or non-medical staff with the possible exception of one response choice in question 2.
The non-medical group were consistently more prepared to give control and trust to computer systems than were their medical colleagues. This general observation is true even where statistically significant differences were not reached. There were no consistent corresponding trends in responses between the critical care and anesthetic groups.
Responders wanted systems to compensate for failure of vigilance as one of their primary objectives.
Responses indicated that staff welcomed the idea of detailed advice on treatment off-line, preferably through flow diagrams and wanted on-line advice limited to alarms that should be more sophisticated.
The “expert” opinion that should embodied in the computer system as the basis for advice given, should be drawn from as wide a group of experts as possible.
Extra physiological measurements would be welcomed.
Generalizing from the preliminary study should be treated with caution. Whilst a considerable amount of statistical analysis is possible on the data collected it should be borne in mind that it is a relatively small sample of staff in a single hospital. However, the lack of statistically significant differences between non-responders and responders in the study suggests that the responders were not self-selected for those in favor of computer systems or more familiarity with them. The safest conclusion is that this preliminary survey must be followed up over a wider range of hospitals and with a greater number of individuals. At least the validity of the technique and questionnaire have been demonstrated.
The study also shows considerable subtlety in responses to certain features of any decision support system. This subtlety cannot be resolved simply by questionnaire-based work. The study further indicates therefore that a qualitative interview-based approach should be pursued to elicit some of the detail differences and attitudes that underlie the overall responses to the questions and choices.
Underlying attitudes towards decision support systems that might be designed are clearly important to manufacturers. More importantly, the features of a decision support system would interact with the complex cognitive environment of the critical care area or operating theatre to affect human factors safety. Authors such as Reason, Rasmussen and Vicente [2,3] emphasise the need to make systems in environments that involve complex decisions as cognitively ergonomic and ecological as is possible. That is they should fit in with the way in which people think, not force them into unfamiliar modes of thought. There is considerable evidence that such systems promote safety. The results of this preliminary study suggest that users would prefer such systems. Further, the results are consistent with other attitudinal surveys, usually made on a more limited methodological basis, [e.g. 4,5]. In laboratory-based testing one of the characteristics of such attitudes is that they indicate reasonably high degrees of confidence in computer systems until some event undermines the operator’s confidence in the computer system. In the event of a false alarm or incorrect advice confidence in such systems plummets and is very slow to recover. If the attitudes detected in this preliminary study are generally representative, manufacturers should heed the warning that nothing discredits equipment in the eyes of users more than false alarms or incorrect advice.
I would like to acknowledge the assistance in statistical analysis of Dr. Valerie Hillier and Mrs. Tripti Halder of the Department of Medical Computation, Manchester University.