Introduction

For the past decade the Internet has served as a conduit for the global dissemination of information, allowing for the rapid distribution of research data via the World Wide Web. More recently this venue has been exploited for the collection of research data as well. Advances in technology and the omnipresence of both computers and individual access to the Internet have increased the potential effectiveness of online data collection strategies. Benefits of this approach may include increased accuracy of data entry, timely collection of participant responses, and a reduction in costs specific to survey replication.

Indeed, the literature offers support for the use of Internet-assisted distribution of information as well as data collection. For example, Tate, Wing, and Winett (₁) used the Internet to deliver a behavioral weight loss program where participants received instructional material via electronic mail, submitted online dietary progress reports, and communicated with others through an online bulletin board. Similarly, Baehring, Schulze, Bornstein, & Scherbaum (₂) constructed an online assessment tool to assist in the identification of those with undiagnosed non-insulin dependent diabetes mellitus. A web-based approach was also used by Pereira, Bruera and Quan (₃) in collecting data from 417 palliative care health professionals regarding their use of the Internet. In terms of online survey efficacy, a number of authors have discussed their successes at obtaining data. In a study regarding the opinions of emergency physicians on the use of emergency ultrasound, Blaivas (₄) gathered data via an anonymous online survey from 1,600 physicians yielding a response rate of 63%. In a similar study of emergency medicine residency applicants, data were collected from 138 of a possible 250 subjects, a response rate of 55%. (₅) Evidence also exists that online modalities may be as effective at conducting psychological experiments compared to lab-based experiments, lending additional support for the utility of the electronic medium. (₆) Other authors have reported similar findings (₇).

Although the literature discusses the possibility of the anonymous online survey becoming a commonly used tool in research, few authors discuss the deficiencies, biases, and limitations of this methodology especially from a first hand perspective. Like other methods of data collection, bias is an issue to be considered. For example, bias is introduced when data are collected electronically with disregard for those who lack computer skills or technology sufficient to complete the instrument. Error also emerges when computers are used to collect information compared to traditional paper and pencil approaches. To examine this type of bias Yates, Wagner, and Suprenant (₈) reported a comparative study in which 63 undergraduate males and females were randomly assigned to one of two data collection methods: Computer assisted or printed questionnaires. Students were asked to recall his/her frequency of risky health behaviors such as alcohol consumption, sexual behaviors, and drug use. Though responses were not collected over the Internet, the researchers reported concerns regarding the comparability of the data collection methods. Students queried via the computer-assisted approach reported significantly fewer risky sexual behaviors. Binik, Mah, and Kiesler (₉) echoed this concern by discussing the potential shift in individual responses by virtue of being studied online particularly when the subject matter is personally sensitive.

Apparently, electronic data collection is widely used and accompanies both advantages and disadvantages. The purpose if this paper is to discuss some of the problems encountered by the authors when doing online research. Based upon our analysis we offer basic guidelines that we hope will improve the effectiveness of this research method.

The Original Study: Background Information

Anecdotal and limited empirical evidence suggest that physicians may be somewhat lacking in their knowledge regarding HIV/AIDS disclosure laws and policies. The purpose of the original study was to measure the level of physician-knowledge as related to HIV/AIDS legislation, policy, and reporting practices at a southeastern academic medical center. To answer the question, “How knowledgeable are physicians regarding HIV disclosure laws and policies?” the research team designed a 55 item primary-choice survey to be administered via the World Wide Web. The instrument was intended for all faculty, fellows and resident physicians (n= 779) at the institution. After pilot testing, the universal resource locator (URL or web address) to the online survey was electronically mailed to all members of the response pool via institutional-supplied email addresses which were assumed to be 1) accurate and 2) utilized by subjects. The email message accompanied information regarding: the purpose of the study, a request for their participation, informed consent, and issues of debriefing. As an incentive, participants who agreed to complete the survey were given the chance to win one of two $250 book vouchers, an approach to marketing a survey that has been largely ignored in health-specific research. (₁₀)

Limited Rates of Response

Of the original electronic messages to each of the 779 members of the subject pool, a total of five responses were received by the end of the first month of study. To increase response rates, a written reminder indicating the online location of the survey was sent via intercampus mail to each participant, yielding seven additional responses. In a final attempt to obtain responses to the electronic survey, a letter announcing and requesting support was sent via intercampus mail to each departmental chairperson. This announcement mentioned the addition of a third $250 book voucher for the department with highest response rate. At the conclusion of month three, a total of 17 responses had been collected, yielding a response rate of 2.2% (see Table 1). The research team reconvened to discuss the limited survey responses and a decision was made to gather data using traditional methods (i.e., a captive audience with paper and pencil instrumentation). These results will be reported elsewhere.

Figure 1

Table 1: Cumulative Responses by Attempt to Increase Participant Responses

Improvements For Web-Based Data Collection

Although traditionally successful methods of web-based data collection were employed, the overall return rate of the original study was poor (2.2%). To examine this limitation, two focus groups involving medical residents from the original study pool (n=12) were conducted. Residents first completed a brief ten question written survey involving their views of online survey approaches, acceptable incentives, and effective notification of survey availability. The survey was comprised of five objective questions and five subjective questions. Subjective questions served as a basis for an open dialogue regarding online research and survey appropriateness. Analysis of these data afforded the researchers much needed insight into the utility of collecting data via online survey modalities.

Descriptive And Qualitative Analysis

A summary of responses to the five objective questions is offered in Table 2. Of the 12 focus group participants, 11 reported home access to the Internet. Seven participants reported that they had previously completed an online survey in the past. With reference to the recent HIV/AIDS instrument, only half of focus group participants, (all of whom were members of the original survey pool) reported that they received notification (either by electronic mail, written or verbal correspondence) of the survey. As indicated by participants many do not use their institutional accounts, instead relying upon secondary email accounts. When asked about the appropriateness of the incentive offered (a chance to win one of two $250 book vouchers), 11 participants reported that this was an acceptable incentive and one that would increase the probability of survey completion.

Figure 2

Table 2: Objective Survey Responses

The importance of this information lies in the need for online researchers to identify sample characteristics necessary in order for participants to provide input. Issues such as reliable access, technological experience, and comfort level need to be ascertained prior to data collection in an effort to limit subject exclusion. By more clearly identifying the skill levels and communication behaviors of participants more targeted approaches can be developed to ensure equanimity among sample members.

During the focus group, the researchers took notes based on participant's comments. A thematic analysis was conducted to examine information collected via subjective questions and comments made during group discussion. What follows is a summation of the most common recommendations offered by potential study participants when asked, “If given the opportunity to complete an online survey in the future, please list five recommendations that would increase the likelihood of your completing the instrument.”

1. “The survey should be related to a specific academic or clinical position.”Many participants reported that they would be more inclined to complete a survey if the material related specifically to their given academic or clinical positions or duties. Not only would this approach motivate the participant by virtue of interest, it would also serve as a pragmatic method of self-assessment. It is interesting to note the reluctance of many medical residents to view HIV disclosure laws and policies as relevant to their specific positions, given their academic and professional training and setting.

2. “Methods of notification should be enhanced.”As reported above, a number of participants reported that they were never informed of the survey. However, attempts to notify participants via intercampus mail and, more indirectly, via academic chairs also proved unsuccessful leading the research team to speculate that etiology of limited response also lies outside any deficiencies regarding our selected routes of notification. Focus group participants suggested other routes of notification, including access through the Chief Resident, provision of notification through a pager service or beeper, and announcements made at weekly staff meetings such as grand rounds.

3. “Survey length and time required to complete the instrument must be reduced.”The original survey consisted of 55 multiple-choice questions and required between 20 to 30 minutes to complete. Group participants reported that, given individual schedules, as well as the novelty of Internet-based data collection, survey length should be reduced to no more than 25 multiple-choice questions.

4. “Incentives should be given to all participants / Non monetary incentives should be offered.”One concern that was discussed by group participants was the possibility of offering subjects a “guaranteed” incentive of $5 to $10 for taking part in the survey rather than a diminished chance of receiving one of two $250 book vouchers. The participants reported that this approach would increase response rates substantially. Additionally, participants reported their desire to take part in research studies where food (e.g., a free lunch) and learning opportunities specific to survey questions were utilized as incentives rather than money.

5. “Immediate feedback should be given to participants regarding their individual responses.”A common recommendation made by participants revolved around the instructional quality of the instrument. Not only is it important to offer “education” as a tangible incentive, participants identified its importance immediately after survey completion. Respondents reported an increase likelihood of completing an online survey if they knew they would receive some form of assessment and explanation upon survey submission such as a final “grade” or a list of supplemental resources.

6. “The survey and access to the survey itself should be user friendly.”Many participants viewed the online survey as “strange” and “new.” Others reported that their inability to participate originated from their inability to access the survey online due to difficulties with their Internet Service Provider (ISP) or specific operating systems, issues that fail to lend themselves to researcher control. Had the survey been perceived as more accessible and “user friendly,” participants may have been more apt to complete the questionnaire.

Conclusions

The purpose of this investigation was to examine the limited response rate of our original study, in an effort to construct guidelines specific to online data collection among physicians. To this end, the research team conducted a descriptive inquiry among members of the original subject pool to understand the lack of response on the part of medical residents. Given the increased use of web-based surveys among members of the scientific community, urgency abounds regarding our understanding of the dynamics of Internet-based data collection and the incorporation of electronic notification of such instruments. Though the utility of electronic mail as a means of collecting data have been discussed in the literature (₁₁) a more through investigations are warranted.

Based on discussions among focus group members the following recommendations are offered. It is hoped that these results will foster a dialogue on the applicability of online data collection methods to contemporary research.

1. Length should be reduced to no more than 25 multiple-choice items (requiring atmost 10-15 minutes to complete).

2. Potential participants should be notified of the availability of the survey in person or through a common source such as the Chief Resident or faculty representative.

3. Incentives in the form of education or food should be offered to all participants who agree to complete the survey.

4. Immediate feedback in the form of a grade or other assessment should be given to each participant immediately upon completion.

5. Finally, the survey must be globally accessible and utilize a “user friendly” interface, one that is easy to read and navigate.

The authors acknowledge the limitations of the current study in terms of focus group size; however, this analysis was our initial attempt to better understand the dynamics of online data collection. As researchers, it is imperative that we utilize the Internet to its fullest potential, as a point of both data distribution and data collection. If participant involvement is to be enhanced and bias better controlled, unique venue-specific criteria must be utilized during the process of data acquisition. By following guidelines offered herein, other researchers might avoid the errors committed in our original study. Regardless of these limitations, the utility of online data collection and survey development should be further explored and cultivated as these modalities offer a host of advantages over traditional research approaches.

Correspondence to

Dr. McKinley Thomas Augusta State University Department of Kinesiology and Health Science 2500 Walton Way, Augusta, Georgia 30904 706-731-7924 bthomas@aug.edu