Source of funding

This work was done by the authors at the Unit for Health Promotion, University of Southern Denmark, Esbjerg, Denmark using data gather while working for “SARSControl : Effective and acceptable strategies for the control of SARS and new emerging infections in China and Europe”, a European Commission project funded within the Sixth Framework Programme, Thematic Priority Scientific Support to Policies, Contract number: SP22-CT-2004-003824.”

Introduction

Development of international health policies has proven to be an urgent and crucial issue in containing emerging infectious diseases such as SARS and threats of pandemic influenza. International Health Regulations have been updated to meet the new challenges. Surprisingly, there is hardly any published documentation on how policies are developed other than the use of methods such as consultations, technical meetings, and workshops of experts and key stakeholders. Without formal and transparent documentation on exactly how these methods have been used, and how well they have functioned, it is impossible to use available evidence in systematically developing policies.

The Delphi technique, originally developed by the RAND Corporation [₁], helps in structuring a group communication process that is particularly useful when there is little knowledge or uncertainty surrounding a complex area being investigated [₂,₃,₄,₅]. Van Zoligen & Klaassen [₆] classify the Delphi technique into: the Classical Delphi - to establish facts; the Policy Delphi -to generate ideas; the Decision Delphi -to make decisions; and the Group Delphi -for group discussion. No studies are reported in the literature on the use of Delphi technique in developing infectious disease policies. It has been used in other areas of healthcare such as identifying priorities in clinical guideline development mental health [₇], and in advocating tobacco control [₈]. Barker's analysis of methods and techniques to analyse and develop health policies showed the Delphi technique to fit well for issue definition , objective/priority setting, both process and substance, prescriptive as well as explanatory use, and mostly for qualitative data [₉].

The Delphi technique needs to be applied systematically and rigorously to produce reliable and valid results [₇] and to avoid discrepancies [₁₀]. Based on literature, the five core criteria when using the Delphi technique (Table 1) are: a) panel composition (geographic and professional representativeness, size, heterogeneity [₁₁,₁₂]; b) participant motivation (response rate, written consent, clarity of questions, reminders; c) problem exploration [₁₃]; d) consensus definition e.g. as percentage of agreement /medians [₇,₁₄]; and e) format of feedback e.g. individual responses, measures of tendency and spread of responses [₅], different statistical description using median, mean or percentage [₁₅] which can decrease unnecessary disagreement [₁₆]. Other criteria include - number of rounds, anonymity to encouraging open expression of opinions [₁₇], and sufficient resources which include time and administrative services [₁₈]. Appropriately addressing the issues while carrying out a Delphi process, determines efficient application of the method to obtain the desired result (Delphi outcome). The methodological challenges while carrying out a Delphi to develop emerging infectious diseases sparked this study.

Setting and methods

The European Community funded research project SARSControl (more information available at www.sarscontrolproject.org) included 17 international organizations. Its policy evaluation work package was the setting of this study and the basis of our study was the SARSControl hence, data and results from it have been used for our analysis. Delphi results of the SARSControl study are available on the project website and have also been submitted as a separate manuscript[₁₉] describing in detail the outcome of the Delphi-process i.e. international policy recommendations developed for SARS and SARS-like diseases.

This paper uses the criteria defined in the literature to assess the process of using the Delphi technique in one of the largest up to-date projects aimed at developing international emerging infectious disease policies. The criteria listed in table 1 were used to evaluate the application of the Delphi technique. The evaluation was done using the qualitative description of the SARSControl Delphi and carrying out a critical analysis of different aspects of each criterion.

Figure 1

Table 1: The Delphi technique: key features of design, administration and analysis of the data

Results

Group composition and participant motivation (see table 2)

The Delphi process, which was carried out over a period of nine months consisted of a pilot round, two written rounds and a final face-to-face meeting. Sixty infectious disease experts who were national experts in the field of infectious diseases who working at senior levels nationally and internationally, and represented their country on the Advisory Forum of the European Centre for Disease Control, Stockholm (ECDC) were approached. Of the 60, 47 accepted the invitation-written consent was obtained from them. Thirty-eight experts (from 22 countries) participated in the 1 ^st written round and 28 experts (from 19 countries) in the 2 ^nd written round; and 11 newly recruited experts with similar expertise as the participants from the written rounds (as five panellists invited from the written rounds were not able to participate ) from 9 countries participated in the face-to-face meeting. A possible explanation for this is that as national experts, they were extremely busy, and also the panelists were not invited early enough and nor were they informed of it at the recruitment phase. Two reminders were sent after the written round invitations. The response rate for the 1 ^st round was 80% and the 2 ^nd round was 74%. Seven replies received after the face-to-face meeting were excluded from the analysis. Misinterpretations of questions in the written rounds were rectified by either deleting or clarifying them in the 2 ^nd round questionnaire.

Figure 2

Table 2: Accomplishment of the process criteria in the SARSControl Delphi study.

The Delphi panel participants consisted from various health-related disciplines, however their specific experience with SARS was not known. Other relevant specialists for e.g. behavioural scientists and communication specialists were not included in the Delphi panel. European countries were well represented- France, Germany, Singapore, Sweden and the U.K. were countries with experience of ‘probable' SARS cases. Learning from Chinese and Canadian experiences was one of the aims of the project. Two experts from China (with the most SARS experience) were invited to the written rounds but they were unable to participate. Canadian experts were not invited.

Problem exploration and consensus definition (see table 2)

Feedback, number of rounds and anonymity (see table 2)

Resources (see table 2)

Discussion

This is the first methodological study to assess the Delphi technique in developing international infectious disease policies. Although this is a case study within one project, it exemplifies some crucial issues in applying the Delphi technique. The major strength of the policy Delphi used was its ability to ascertain expert opinions and potential policy options, including new alternative ones, from countries representing different cultures and health policies on the timely issue of infectious disease control. This contributed to the policy recommendations to be put forward to policy makers to agree upon. The major weakness of the Delphi was that the panel represented mostly European experts, missing countries with most SARS experience, and also the discontinuity from the written rounds to the face-to-face meetings due to drop out and financial constraints.

A rather homogenous professional representation in the Delphi panel on the other hand was not necessarily such a drawback since the aim was to generate feasible policy recommendations and alternative and innovative strategies, and according to Adler and Ziglio [₁] this can be achieved also with more homogeneous panels. Use of pre-existing information increases the effectiveness of the Delphi process [₂₂]. The SARSControl Delphi used information from the literature, HACCP analysis and from other work packages within the SARSControl project in developing a close-ended questionnaire worked well. Addressing emerging infectious diseases generally was a drawback in the 1 ^st questionnaire as it led to misunderstanding of some questions. The lack of an infectious disease expert in the research team was possibly the reason for inappropriate question formulation. Anonymity in the written rounds provided an opportunity to express opinions openly without feeling pressure by others. This most probably ensured truthful replies making the results more reliable as stated by Goodman [₁₇] and Couper [₂₃].

The use of 75 % cut off, which is in line with McKenna et al. [₂₄] and Kilroy & Driscoll et al. [₂₅], proved to clearly differentiate the consensus and non-consensus results. The use of a wide 9-point Likert scale had only the extreme values labeled verbally which probably led into different interpretation of the categories. A more narrow scale with clear verbal labels might have been more informative. However, the comments written by the panelists clarified and brought useful additions. The feedback of the written Delphi rounds given to the participants ensured that all the views were explicitly presented and common interpretations of the variables and responses were established. These procedures prevent unnecessary disagreement among participants [₁₆].

The two written rounds of the Delphi and a face-to-face meeting were found sufficient to obtain the outcomes in terms of generating policy options and alternatives. The 2 ^nd Delphi round was crucial in clarifying issues from the 1 ^st round. Thus the SARSControl Delphi process succeeded in its aim to generate policy options and alternatives - this in spite of the discontinuity in the involvement of the experts throughout the Delphi process.

Consensus was reached on most of the issues put forward in the SARSControl Delphi questionnaires. Non-consensus questions on non-medical interventions were taken up in the face-to-face meeting and experts suggested solutions to them. Further, alternative strategies were suggested to prevent and control future spread of SARS and SARS like diseases. These measures would improve also transferability of the research-based knowledge from SARS to other potential infectious disease outbreaks.

The range of the policy areas covered by the SARSControl Delphi covered all the areas listed by the CDC document (CDC, 2004). The outcome of the Delphi gave statements with opinions of experts on the vast policy areas, which need to be considered in preparedness and response to an outbreak of SARS or SARS-like diseases. These opinions coincide with what was reportedly found to have worked and not worked during the 2002/2003 SARS outbreak and gaps identified in planning and response of pandemic influenza by Coker and Mounier-Jack [₂₆], hence it can be said that the Delphi technique validated the earlier findings.

In the absence of published methodological literature on the Delphi application on international infectious disease policy development, it is not possible to compare directly the results of this study with the previous ones in the field. Its use in other areas e.g. mental health to develop clinical guidelines, tobacco control to develop tobacco polices etc. has been shown to be very fruitful. The use of the Delphi technique in comparison to other methods such as consultations, meetings etc. highlights its strengths as systematic group communication process allowing it to deal with a complex problem by gathering expert opinions . It also brought about new and alternative policy options. In terms of resources, Delphi is quick and cheap. It enables participation of a vast number of geographically dispersed individuals.. The technique also gives more transparency to policy development as compared to other methods for e.g. IHR has been reported to have been developed using meetings and consultations, however, no published information is available on the development process [₂₇,₂₈]. As an overall impression from the Delphi process and the policy options developed, it can be said that it was a positive experience in terms of developing policy options and alternatives.

Use of explicit, published criteria meant that information considered relevant was gathered. The reliability of this study was strengthened as the assessment of the Delphi technique was done based on the criteria reported in the literature for the Delphi process. The validity of the results will only be found out when the policy recommendations will be tested by presenting them to politicians and decision makers for their judgment, and this remains to be tested.

Conclusions

The Delphi technique can aid the international policy development process and it can be a versatile tool, which creatively collects expert opinions and suggestions in a new topic. Central criteria to be met include representative panel composition, high panelist motivation, and effective but flexible administration of the Delphi process. Based on the assessment of SARSControl Delphi technique, it can be concluded that when rigorously administered, analyzed and reported, is a valuable method to develop international health policy recommendations for emerging infectious diseases, even though discrepancies in its application existed. The SARSControl Delphi technique was a positive experience, and can also be considered by others for similar purposes.

Competing interests

Author contributions

Acknowledgments