Participation Rates In Epidemiology Studies And Surveys: A Review 2007–2015
C Keeble, P D Baxter, S Barber, G R Law
Keywords
bias, non-response, participation, rates, selection
Citation
C Keeble, P D Baxter, S Barber, G R Law. Participation Rates In Epidemiology Studies And Surveys: A Review 2007–2015. The Internet Journal of Epidemiology. 2016 Volume 14 Number 1.
DOI: 10.5580/IJE.34897
Abstract
Understanding the factors associated with participation is key in addressing the problem of declining participation rates in epidemiological studies. This review aims to summarise factors affecting participation rates in articles published during the last nine years, to compare with previous findings to determine whether the research focus for non-participation has changed and whether the findings have been consistent over time.
Web of Science was used to search titles of English articles from 2007–2015 for a range of synonymous words concerning participation rates. A predefined inclusion criteria was used to determine whether the resulting articles referred to participation in the context of study enrolment. Factors associated with participation were extracted from included articles.
The search returned 626 articles, of which 162 satisfied the inclusion criteria. Compared with pre-2007, participant characteristics generally remained unchanged, but were topic-dependent. An increased focus on study design and a greater use of technology for enrolment and data collection was found, suggesting a transition towards technology-based methods.
In addition to increased participation rates, studies should consider any bias arising from non-participation. When reporting results, authors are encouraged to include a standardised participation rate, a calculation of potential bias, and to apply an appropriate statistical method where appropriate. Requirements from journals to include these would allow for easier comparison of results between studies.
Abbreviations: Missing at Random (MAR), Missing Not at Random (MNAR), Doctor of Philosophy (PhD), Short message Service (SMS).
Introduction
Identification of the factors associated with participation could help to understand why participation rates have been declining over the last 30 years or more [1, 2]. Reasons for the decline itself include an increasing number of refusals, more stringent participant criteria and changes in lifestyle [1]. Refusals can be explained by the increasing number of research requests, general decreases in volunteering and the increased expectations of the participants, while changes in lifestyle include more mobile telephones, fewer telephone directories and longer working hours [1]. Regardless of the underlying motivation, non-participation can lead to participation bias and cause the results to not be generalisable to the intended population [1].
To calculate participation rates, a range of definitions have been suggested [3], which can cause problems when comparing studies. Although standard calculation formulae developed by experts exist [4, 3] as far as we are aware, there is no formal consensus, even within journals, on which rate to adopt or which calculation to use [5]. This may result in researchers selecting the definition or formula which shows their work most favourably. For a given definition, the rate value can also differ merely by the assumptions made by the researcher, leading to different rates quoted from the same study or survey [6].
In 2007, a detailed review of participation rates in epidemiology studies was conducted, including what was known about who participates in epidemiologic studies [1]. However, we could find no such review since then. Briefly, the 2007 review found that participation was associated with individual characteristics such as age, sex, race/ethnicity, socioeconomic status, education level, employment status and marital status [1]. Regarding the study design, participation rates tended to be higher for face-to-face recruitment rather than less personal means and, understandably, for those requiring less commitment from the participant [1]. Studies offering monetary incentives were generally found to increase participation, as were those offering a choice between modes such as paper surveys or telephone communications [1]. Web-based surveys were beginning to be utilised and found to be particularly useful when recruiting young or college participants, but not universally successful, especially with concerns such as data security [1].
In an attempt to increase participation rates, researchers have utilised these findings and incorporated incentives into their studies [7] or oversampled groups of people known to participate less frequently [8]. While these approaches may aid participation [9], they may not be reducing the bias associated with nonparticipation. “(Non)participation bias refers to the systematic errors introduced in the study when reasons for study participation are associated with the epidemiological area of interest” [1]. Therefore non-participation can lead to participation bias, but bias does not always occur [6, 5, 10]. A study with a very low participation rate may contain little or no bias, while another study with high participation rates may have considerable problems associated with participation bias [5, 11]. Often participation rates are reported for a given study, however it is possible that participation bias may vary from one estimate within a study to another [12], causing participation rates to be a poor proxy for participation bias [6]. Participation bias is known to invalidate conclusions and generalisations which would otherwise be drawn, yet unfortunately its consideration is frequently omitted from articles [13].
Advances in technology, increased use of the Internet, more open data and increased data sharing have all occurred in recent years. These changes may have affected the way in which data are sought and recorded, and in turn may have affected participation rates. In addition, societal shifts may have led to differences in participant characteristics. This work is intended to build upon the findings in the previous review [1] which summarised studies prior to 2007, and so includes more recent articles from 2007–2015. We wish to know whether these developments have influenced the type of person participating and the way in which they do so. This will be answered using a literature review, with the findings available to inform future work requiring participants, or for analysis by behavioural psychologists with the view to a multidisciplinary approach to understanding participation.
Method
Inclusion Criteria
Web of Science [14] was used to search titles of English (language) articles from 2007–2015 for a range of synonymous words concerning participation rates. The title search used on 8th September 2015 was TI=("selection rate*" OR "participat* rate*" OR "nonresponse rate*" OR "response rate*" OR "nonparticipat* rate" OR "cooperat* rate*" OR "noncooperat* rate*"). This returned 626 articles for further consideration.
The abstract of each of the 626 articles was read to determine whether the article met the next phase of the inclusion criteria which ensured participation rates were in relation to a study or survey. Specifically, participation here refers only to the willing enrolment, or involvement, of an individual to a survey or study, where adequate data are provided to assist the research question. Synonyms of participation include ‘(self-)selection’, where an individual volunteers, ‘cooperation’, where an individual agrees to be involved, or ‘response’ relating to, say, the return of a completed questionnaire. Therefore these synonyms are provided in the context of participation in research rather than the general definition of the term. Linking these terms is the willingness of the individual to contribute data. Similarly, non-response, non-cooperation and non-participation were of interest, to understand those individuals who decline a survey or study. If the abstract was not sufficiently detailed to determine inclusion or not, the full text was sought and read. All study designs were included such as cohort studies, case control studies, trials and surveys, with the overarching requirement that the individual had to consent to involvement in the data collection, that is, willingly participate.
From the 626 article abstracts read, 162 articles satisfied the inclusion criteria. The results included a brief summary of each article, the year it was published and any participation findings. The results were later split into two sections; those concerning the person participating and those relating to the study design.
Exclusion Criteria
Unintended interpretations of the search terms such as ‘response’ to an intervention, ‘participation’ in a physical activity, or ‘cooperation’ with an event were not of interest, and hence these articles were excluded from the review.
During the final phase of the inclusion criteria, 464 articles were excluded, with the main reason being that the term ‘response’ related to a patient response to a drug or treatment (282). Other reasons were repeated articles (6), articles regarding best practice (67), where ‘participation’ described the uptake
or acceptance of an intervention (26), articles investigating the labour force participation rate (22), articles where ‘participation’ described involvement in a sport or activity (27) or where ‘response’ described a reaction using a stimulus or similar (34).
Results
Participant Characteristics
Characteristics of the people found to participate or not, are listed starting with the most reported theme, and their correspondence with previous findings noted.
Age was found to differ between participants and non-participants, as in the 2007 review [1], with studies reporting findings such as those who were 30+ [15], 40+ [16], 51+ [17], 75+ [18] or older [19, 20, 21, 22, 23] being more likely to participate. Although these studies used different age categories, they each concluded that older people were more likely to participate than younger people. One study simply stated that age was important [24], while another found those who were younger [25] were more likely to participate in a text messaging study, although this may be a finding unique to text messaging.
Higher education levels were associated with higher participation rates in studies [26, 27, 28], or the education level of participants was found to differ by sampling technique [20]. This was a known characteristic associated with increased participation in 2007 [1]. Being a homeowner was also found to be associated with increased participation probability [28]. There may be an association between education levels and homeownership, or between homeownership and age. Employment type was associated with participation [24]; full-time employment was associated with lower participation rates [20, 23], while unemployment was associated with increased participation rates for studies offering incentives [29]. This may be related to the amount of free time potential participants have to complete a survey or be involved in a study, but does contradict the findings in 2007 [1].
Race and ethnicity differed between those who chose to participate and those who did not [24]. Those more likely to participate were found to be non-Asian [30], white [31, 32, 33], or Western [34], generally agreeing with the previous review [1]. Participation was found to differ by country [35], which may incorporate factors such as ethnicity and race. Location generally was also found to differ between participants and non-participants [24, 36], with those in rural locations more likely to participate [32]. Location may be associated with other factors discussed earlier, such as employment status, education level and homeownership.
Sex was found to be associated with participation [24, 22], with females more likely to participate than males [19, 34, 32, 37, 16, 38], as commonly found in studies through time [1].
Smoking status was found to be associated with participation [39], with non-smokers (or those who are not lifelong smokers) usually more likely to participate [40, 41, 28, 23], as also found in the earlier review [1]. Smoking may be a factor specifically related to the study of interest, since it is unlikely to be recorded routinely for all studies.
Marital status was found to differ between participants and non-participants; with those classed as married [28] or not single [15] being more likely to participate, again agreeing with previous findings [1].
Socioeconomic class was associated with participation, with those categorised as not lower class [34] or not manual social class [28] being more likely to participate. Similarly, previous work has concluded that upper class people or those with a higher socioeconomic status are more likely to participate [1].
Physicians with less than 15 years’ experience were found to be more likely to participate than those with more experience [32], which may be specific to physicians or even this particular study. Mental health problems were associated with lower participation [34], although this is a variable which may only be recorded in studies where mental health is of interest. Obesity was found to be associated with lower response rates [28], but again obesity is a factor which is often only recorded in studies associated with weight. Multiparous women, or women with preterm deliveries were less likely to participate in a pregnancy study [15]; variables which are likely to only be recorded in pregnancy or pregnancy-related studies. Lower pain intensity was found to be associated with increased participation probability [23] when the study considered surgery; which may or may not be generalisable to other surgery studies. These factors are less commonly recorded and hence cannot easily be compared with the 2007 review findings.
Heavy drinkers were assumed to be less likely to participate in alcohol consumption studies [42]. Although specific to this study, or studies of alcohol consumption, it may be that people who indulge in habits with negative connotations are less likely to participate in a study regarding that aspect of their lifestyle. Alternatively, one’s function may be impaired by overindulgence in particular areas such as alcohol consumption or drug use and hence this may affect their participation in a study or their completion of a survey.
Cases were found to be more likely to participate than controls [25], as found in the previous review and frequently in case control studies [1]. This may be related to their motivation to participate, to find a cause or potentially a cure.
Study Design
Investigation into the procedures or details within a study or survey which may or may not be associated with participation are summarised here, with the most frequently reported themes listed first within each topic. Some are specific to particular studies, whereas others could be generalised to a range of data collection methods or study topics.
Study Design: Prior to the Study
Participation was found to increase with incentives or free gifts in some studies [43, 44, 45, 46, 47, 48, 49, 50, 26, 51, 52, 53, 54, 55, 20, 56, 57, 58, 59], but not in others [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 21, 74]. Some found that small incentives were not quite sufficient to encourage potential participants [75], while larger incentives were [76]. There were studies comparing sizes of incentives with participation rates, which could help to determine a threshold amongst certain populations of interest, but this may not generalise to all populations. Often those who found incentives to not help study enrolment, were those offering less valuable incentives. Incentives were also usually more successful in studies which sought to enrol those who are less wealthy, or those who are busy and expect compensation for their time. A small incentive such as a free pen may be sufficient for a short survey for non-personal data, but a larger incentive may be required for a survey requiring a blood sample, sensitive data or a significant time commitment. The immediacy of the incentive was also important [77, 78], that is whether the incentive was given at the time of enrolment, or promised at a later date. This mixed influence of incentives was also found in the previous review [1].
Prenotification was found to be helpful in some studies [76, 79, 80, 56, 22, 81], but not in others [82, 83, 84, 85], even when personalised [86]. In 2007 it was thought to be a positive measure [1]. The type of prenotification used was generally found to be unimportant [87]. However, advanced mailing of the questionnaire before a telephone survey, was found to be associated with reduced participation rates [88].
Study Design: Mode of Contact
Paper surveys have been found to be effective [52], to be required in addition to electronic surveys [89, 90], to be better than web surveys (completed online) [50, 91, 92, 51, 75], or electronic surveys (completed electronically but not necessarily using the Internet) [93, 94, 95, 73], or advantageous over telephone surveys [96]. Although conversely an investigation into organisation surveys found participation rates in electronic studies to be as good as or higher than mail [62]. Web surveys were found to be better than mail surveys in a study of PhD (Doctor of Philosophy) holders [97], although offering a web option was associated with decreased participation in another study [69]. Item non-response was similar in web and mail surveys [98], but online surveys were better for open-ended and text answers in a study of item non-response [99]. For web surveys, a welcome screen describing a survey with a short length and including less information regarding privacy, was found to be most effective [100]. Recruitment using a direct email was more successful than through a newsletter [49] and tablet device surveys [101] or facebook [102] were found to help recruit reluctant or hard-to-reach potential participants. Exclusively online surveys were found not to be suitable for a doctors survey [103], generally not effective in a medical practitioner survey [104], or less effective than other modes [105].
Telephone calls can be useful [43, 106, 81] and there exists a simple positive relationship between the number of calls made and the response rates [107]. Utilising multiple sources to obtain a telephone number, followed by multiple phone call attempts and postal approaches, was successful at increasing participation rates in one study [108], although this could be viewed as unethical and as a form of harassment or coercion.
Short message service (SMS) was successful in an arthritis study [109] and an SMS reminder was found to increase response rates [37]. Text messaging an invitation received faster responses than email invitations [110] and particular combinations were found to be highly effective, such as an SMS prenotification followed by an email invitation [111].
The previous review also found differences in participation between survey modes [1], but with less emphasis on modes utilising modern technology such as web surveys and SMS. Recent advances in technology may alter the effectiveness of each mode of recruitment now and in future research.
Study Design: Survey Delivery Mode & Design
Mailing was found to be an effective mode [43, 112, 113, 114, 56, 38], and better than emailing [63], although being handed a survey by an acquaintance was found to be more effective than mailing in studies involving older communities [115]. Priority [74] or registered mail [116, 56] were found to be associated with higher response rates [45], but tracked mailing was associated with lower rates [117].
Repeated mailing [74] and reminders [118, 119, 120, 121] successfully increased participation rates, as did rewording the reminder [55]. Follow-up generally was viewed as useful [52, 122], with follow-up more effective for mail than web surveys [123], but not helpful in all cases [113]. One study even found reminders to be associated with decreased participation rates [62]. Sending a newsletter initially was found to be more beneficial than sending a reminder later [124] and electronic reminders were not found to improve response rates in postal studies [125]. This generally supports the previous review finding of increased participation with follow-up [1].
Response rate does not differ with envelope type [126], envelope colour [127], whether the material was aesthetically pleasing [128], enhanced [17], or contained an envelope teaser regarding an incentive [70]. However, the invitation design was found to be significant [129] as were the size and colour of the paper [130]. The location of the respondent code (on the survey itself or on the return envelope) was not found to significantly affect participation rates [131], neither was numbering the questionnaires [132]. Inclusion of a return stamp aided participation rates [43], and stamped envelopes were found to be more effective than business ready envelopes [124]. Investigations into these factors were not so common in 2007 [1] and so show a recent shift in focus of how to improve participation rates.
Study Design: Choice and Personalised Surveys
The illusion of a choice between surveys (but in fact just a different ordering of questions) was found to increase participation [133], as was locating the demographic data at the start of the survey [134]. Presenting the survey in multiple languages also increased participation rates [135, 136], whereas single (opposed to double) sided questionnaires and the Internet, were not found to produce significantly improved response rates [137]. Survey length was significant in some studies [43, 120, 138, 121], but not in others [63, 87, 64, 139]. Participation differed with the time of day [37] and with the day of the week in some studies [56, 37], but not in others [117]. These are again areas not covered by the 2007 review [1], so show recent developments for investigations into participation rates.
A choice of survey mode (i.e. electronic, paper, etc.) possibly increases participation [43, 140, 141, 48, 142, 143], but does not necessarily reduce the error associated with non-participation [141]. These views were also found in 2007 [1]. However in another study, the addition of a fax option was found to increase response rates, but other electronic options were not [144]. Multiple contact methods can increase participation rates [145] and it was found that the preferred survey mode differed between participants of different professions [123]. Similar findings were reported in 2007 [1].
Personalisation of the survey, such as through tailored letters or interaction with the potential participants, was associated with increased response rates in some studies [146, 43, 147, 148, 120, 122, 55, 56], but not in others [149, 150, 151]. Personalisation is another more recent consideration in studies of participation [1]. A persuasive message can be helpful [152] and surveys at an institutional level are more successful at recruiting respondents than those conducted nationally [95].
Study Design: Specific Studies
Participation rates were associated with features exclusive to particular studies, such as the number of days prior to surgery in an arthroplasty study [153], or the type of cancer amongst cancer patients [64]. A child-focused protocol was also found to be more effective in children’s health research, than a parent/teacher or teacher-only protocol [154]. A survey into male escorts [155] found increased response rates when the researcher posed as a client rather than a researcher, but this approach using deception may be seen as unethical. Sending a female responder to recruit male participants increased participation rates [79], as did having a dedicated centre for data collection rather than using a generic centre [36]. Generally, the survey content was found to affect participation rates [156], including whether samples were required such as saliva or blood [157]. These findings specific to particular studies are not easily comparable with the 2007 review.
Expert help was useful in one study [158], as was endorsement [43], but the additional of a logo or senior faculty’s signature was not found to be helpful [159]. One view is that the potential participants need to be intrinsically motivated for participation to occur [65], although offering the results from the study was not found to increase participation rates [160].
Study Design: Opt-Out
Using the approach of allowing the potential participants to actively decline a postal questionnaire, rather than actively agree, may be one way in which to increase participation rates [161], since active consent was found to reduce participation [162]. Alternatively, using default settings in a web survey could be useful [163], but this approach has the potential to lead to biased results with an excess of default responses.
Discussion
Consistency and Changes Through Time
Changes over time have not generally affected the demographic of participants. Only employment status contradicted previous findings [1], with three studies concluding a negative association of employment with participation [23, 20, 29]. One of these studies could be explained through the inclusion of incentives [29] raising participation rates in unemployed people, but the other two studies concluded full-time employment to be associated with decreased participation, possibly showing a shift in participant demographics. However the small sample size of these studies is not sufficient to draw any definitive conclusions.
In recent years, greater attention has been paid to techniques which increase participation. Studies researching envelope size, colour, style and composition are examples, with the results seen to differ by target population. This valuable information can be used to inform future studies, to ensure resources are not wasted and that the most suitable sample group is obtained. However, increased participation does not necessarily lead to reduced participation bias, since those participating may still differ from those who do not [141, 57].
The greatest change over time relates to participant recruitment and interaction. Although paper surveys remain the predominant survey mode, increasingly web-based approaches are being employed for recruitment and electronic tools are being utilised during data extraction. Technology has advanced greatly in recent years and is expected to continue to do so, suggesting an even greater involvement of electronic devices in future research. The availability of tablets and smartphones has allowed users to participate ‘on-the-go’ and complete surveys at a time convenient to them. Facilities such as facebook enable studies to be advertised easily and encourage the involvement of previously hard-to-reach participants. The Internet grants researchers the ability to quickly contact and enrol participants from all over the world, rather than be restricted to those locally. Advances in technology and the wider availability of devices in conjunction with social media, could result in significantly higher participation rates, particularly for studies where physical contact is not required. Even for studies requiring contact for blood or urine samples, advertisements can be circulated more widely. There will of course be studies for which this information will not be helpful. Examples include recruitment in locations where modern technology is not common, or for populations which are not able or not willing to use technology. In some instances, this ‘digital divide’ could lead to increased participation bias.
Limitations and Assumptions
One criticism of this review, but which does not limit the findings, could be the search terms, since 282 of the results related to treatment response, not satisfying the inclusion criteria. Although common words such as ‘virologic’ or ‘pathologic’ were used in these studies, there was no exhaustive list of terms which would have excluded all treatment articles. This resulted in increased data collection time, but ensured no relevant studies were missed. The search was conducted using only the article titles assuming that research relating to participation would use this or a similar word in the title. The abstract and keywords were trialled for inclusion in the Web of Science search, but since words such as ‘cooperation’ and ‘participation’ are used so frequently in English language, many results unrelated to the research question were returned. There was one article which met the inclusion criteria, but which could not be included in the review as the article was unavailable using the means available [164]. It compares email and postal surveys methods, but the conclusion is unknown.
There were instances where articles reported the same data set, either because the data appeared in multiple studies or since meta-analyses, included to contribute studies otherwise not captured, included the same data. Although this may have altered the findings, the effect should be reduced by the large sample of articles reviewed. Repeated articles were excluded from the review.
Study-specific findings were included, perhaps questionably, to demonstrate successful tactics for participation. Since the future direction of studies requiring participation is unknown, it may be that topics rarely studied now increase in frequency in the future, rendering these specific findings generalisable, hence they were not excluded.
Some articles assumed a causal link between study design and response rates, but it is recognised these may only be associations. Some, such as reminders resulting in reduced participation rates, seem unlikely.
Association Between Participation Factors
Many of the variables found to be associated with participation may be linked, for example it may be that higher proportions of older people live in rural locations or that more employed people live in urban areas. These are merely speculations, but these apparent reasons for participation or non-participation may be due to another recorded or unrecorded factor for which the identified reason acts as a proxy. Also some variables may differ between participants and non-participants, but may not have been recorded. For example, sex and age are often recorded, but factors such as obesity or pain intensity may only be recorded if relevant to the study. There is always the possibility of unidentified or unrecordable factors being associated with participation.
The Future of Participation
Although factors affecting participation have been considered, some authors correctly highlight that increased participation does not necessarily result in reduced participation bias [141, 57]. Using techniques such as incentives to increase participation rates may in fact increase bias. A shift of focus from participation rates to bias may save time and resources by not chasing unwilling participants, which in turn could be used to increase the sample size with willing participants or to conduct a detailed participation bias analysis [6, 165]. To aid this shift, journals could insist all surveys or studies requiring participants detail a participation bias calculation, for judgment by the reader. Alternatively journals could adopt standardised formulae to calculate rates such as those proposed by The American Association for Public Opinion Research (AAPOR) [3], which would at least provide guidance to researchers and allow easier comparisons between studies.
Regardless of the requirements imposed by journals, authors should provide a participation statement so the reader can compare sample-population characteristics, to judge population representation and hence the generalisability and validity of the results. Providing details of the population of interest can also help to assess bias, for example a study may have more female than male participants but if the study is concerning breast cancer survivors, a higher proportion of females than males is expected. Unfortunately details regarding the expected population of interest were not available for all studies reviewed here.
Where participation bias may be a concern, methods developed to reduce this form of bias should be considered. If non-participation causes data to be missing at random (MAR), multiple imputation [166] could be considered to replace missing variables with estimates calculated using the recorded variables. When non-participation causes data to be missing not at random (MNAR), external resources such as population level data could be used to draw conclusions [167]. Alternatively sensitivity analyses can help to estimate the direction and magnitude of participation bias so the true estimate can be adjusted accordingly [168]. The choice of an appropriate method for reducing participation (or selection) bias can be eased using a guidance tool [169], to ensure the study conclusions are optimal. Researchers should consider participation bias and readers should not outrightly dismiss findings on the grounds of low participation rates.
Non-participation is still an issue in studies and surveys, with different study designs and topics of interest suffering from non-participation in different ways and for different reasons. It is unlikely that one strategy would increase participation rates or reduce participation bias for all studies, but insight and knowledge gained from articles such as those covered here, should be used to aid future work. Even negative findings highlight where researchers should not focus their efforts, and hopefully areas which should be targeted, have been identified.
Funding
Claire Keeble is funded by an MRC Capacity Building Studentship. Paul D Baxter, Stuart Barber and Graham Richard Law are funded by HEFCE. The funding sources had no involvement in the study design, in the collection, analysis and interpretation of data, in the writing of the report or the decision to submit the article for publication.
