A task force was convened in 1999 to launch an international initiative devoted to exploring public attitudes toward stuttering. Its long-term objectives were to develop a survey instrument that could effectively obtain baseline measures of public attitudes toward stuttering in comparison to various other stigmatizing conditions potentially in any part of the world. Once completed, the empirical data to be gained from the instrument could be used by various stakeholders to foster and to evaluate effectiveness of strategies for mitigating societal stigma to which people who stutter are subjected. St. Louis (2005) briefly reported the rationale and vision for this “International Project on Attitudes Toward Human Attributes” (IPATHA). This paper elaborates the rationale and vision followed by explanations of initial steps in the development of an instrument designed to obtain such baseline measures. Specifically it describes the rationale, design, and initial field-testing of the first experimental version of the Public Opinion Survey of Human Attributes (POSHA-E).

Scope of the Problem of Stigma

Measuring Stigma

Needs and Challenges in Measuring Attitudes Related to Stigma Cross-Culturally

Most measures of public attitudes or stigma ask people about various aspects of disorders or their own reactions to them. In studies of stuttering, a wide variety of data collection methods have been used: paper-and-pencil questionnaires (Blood, Blood, Tellis, & Gabel, 2003; Gabel, Blood, Tellis, & Althouse, 2004; Hulit & Wertz, 1994; Klein & Hood, 2004), semantic differential scales (Doody, Kalinowsky, Armson & Stuart, 1993), questions to store clerks who had just spoken with a severe stutterer (McDonald & Frick, 1954), face-to-face interviews with people on the street (Van Borsel, Verniers, & Bouvry, 1999), telephone interviews (Craig Hancock, Tran, & Craig, 2001; Ham, 1990), open-ended written statements (Ruscello, Lass, Schmitt, & Pannbacker, 1994), and extended tape-recorded interviews (Corcoran & Stewart, 1998; St. Louis, 2001), among others. A similar range of methods has been used with other conditions, and other innovative methods have been reported as well. For example, health professionals have directly rated the amount of stigma associated with health conditions (Bramlett, Bothe, & Franic, 2003; Ustun, et al., 1999).

Instrument Criteria and Considerations

Content and Format

The POSHA-E consisted of five components:

First component: Instructions. The survey began with a one-page instruction sheet. To minimize the likelihood that respondents might respond differently knowing that the primary intent was to determine public opinion about stuttering, nowhere in the questionnaires was it stated or implied that the questionnaire was designed to obtain information about stuttering per se. The instructions used the word “opinion” throughout rather than “attitude” since in American English, “opinion” has the more neutral connotation.

Figure 1

Figure 1: Sample of questionnaire components and key terms

Second component: General section about nine human attributes. (Beginning in this section, various terms are italicized in this somewhat unconventional and necessarily detailed paper to assist the reader in knowing when such terms refer to specific components, sections, and orders.) The one-page general section had questions and prompts eliciting ratings of nine human attributes, i.e., stuttering and eight other attributes or “anchors.” Task force members decided that the final POSHA would be most sensitive cross-culturally if stuttering could be placed in context with other stigmatizing and nonstigmatizing attributes. That is, stuttering is stigmatizing as compared to what? They further assumed that these anchors would range from negative to positive on the prototype survey: mental illness, wheelchair use, overweight, old, left-handed, good talker, multilingual, and intelligent. The general section prompts asked respondents to rate: (a) their overall impressions of stuttering and the other eight anchor attributes, (b) the extent to which they wanted to be or have each attribute, (c) their knowledge of each attribute, and (d) individuals they know who manifested each attribute. Based on previous research (St. Louis, 1999), respondents were likely to give lower ratings for negative attributes when asked if they “wanted to be or have” the attribute compared to their “first impression.” Figure 1 illustrates a sample of respondents' ratings for the general and other components of the POSHA-E. Additional information about this second component occurs in a later discussion of item order.

Third component: Detailed sections containing three of the nine human attributes or triads. The third component was lengthy, comprised of three detailed sections (or, more precisely, three detailed section triads [explained below]) of two or three pages each. See samples in Figure 1. The detailed stuttering section prompts elicited ratings on: (a) sources of knowledge, (b) causes, (c) sources for help, (d) reactions and feelings in the presence of people who stutter, (e) concern if selected people stutter, and (f) beliefs about stuttering and people who stutter. All of these areas are included in other surveys of attitudes toward stuttering (c.f. citations in section on Survey Methods: Stuttering). The detailed stuttering section served as a template for the other attributes (i.e., anchors), so that attribute-specific detailed sections were similar to the detailed stuttering section and to each other, but some prompts were deleted for specific anchors. For example, groups of items about where to get help were eliminated for the attributes of left-handed, multilingual, good talker, and intelligent since individuals with these attributes would not logically be assumed to need help.

Because a prohibitively large number of potential items would be required if each prototype questionnaire contained all nine attribute-specific detailed sections, a strategy was adopted wherein each respondent would rate only three detailed sections, i.e., a triad. On each questionnaire one of the three detailed sections was always a stuttering section so that we would have complete stuttering data from all respondents. The other two detailed sections of each questionnaire elicited ratings for two of the other eight anchors. The triads varied in order and content across groups of questionnaires (explained below). Other details on construction of this third component appear in the discussion of triad order.

Fourth component: Demographics. All questionnaires ended with a three-page demographic section comprised primarily of nominal variables. The demographic section contained the following prompts for each respondent: (a) date of birth, (b) current and birth residence, (c) citizenship, (d) living arrangement, (e) sex, (f) languages spoken, (g) education, (h) career or vocation, (i) religion, (j) ratings and comments about the respondent's health and other abilities, and (k) how and where the questionnaire was obtained.

Fifth component: Comments. The final page provided space for respondents' optional comments.

Quasi-Continuous Rating Scale

Length of the Prototype Instrument

The first purpose of this field study was to determine the extent to which the order of stuttering ratings was affected by ratings for other attributes. Stuttering ratings occurred both as items along with all the other anchors in the general section and as detailed sections in triads of detailed sections that followed. Since the detailed sections had 85 items, it was far beyond the scope of this investigation to systematize all possible item orders within the detailed stuttering sections. This field study analyzed various possible order effects for the general section and the order of detailed stuttering and other attribute sections in the triads. For example, would respondents be more or less likely to “want to have” stuttering if it were the first item rated, the middle item rated, or the last item rated? Similarly, would opinions about the causes of stuttering be affected if respondents had already speculated about the causes of mental illness versus left-handedness? To reduce item order influence, any potential respondent—or groups of surveys with respondents—should not have an appreciably greater chance of receiving one item order over another item order. That is, distribution of varying item order survey should be randomized.

Figure 2

Figure 2: Counterbalancing and collating schema to identify potential order effects

Accordingly, three levels of randomization were used. First, a systematic counterbalancing and item randomization scheme controlled for effects of item order of individual items among the four prompts in the general section and for the triad order of entire detailed sections devoted to stuttering or other anchors. The scheme is summarized in Figure 2. Four prompts appeared in the general section: (a) “My overall impression of a person who…”, (b) “I would want to be a person who…”, (c) “The amount I know about a person who…”, and (d) “People I have known who….” Each prompt was followed by nine line items eliciting ratings of the nine attributes, one of which was stuttering. Stuttering and all attributes held the same order at all prompts on a given survey. However, across survey versions, the place of stuttering in this item sequence was rotated so that stuttering might be the first item for all prompts, or the fifth, or the ninth item at all prompts in the general section.

Within each of these three item orders in the general section, a triad order of detailed sections was varied systematically so that the detailed stuttering section appeared as the first, second, or third section rated, and so that each of the remaining eight detailed anchor sections occurred with all the others in all possible combinations and occurred first or second. The latter pairs of eight anchors yields 56 possible combinations (8 attributes x 7 combinations of two attributes) as shown in Figure 2. One-third of the questionnaires represented each of the three item orders (first, middle, and last position), each of three triad orders of the subsequent detailed section on stuttering (first, second, or third), and the various combinations of the other attributes. Thus, 504 possible orders existed for a complete set of parallel POSHA-E questionnaires (3 general item orders x 3 detailed triad orders x 56 anchor combinations within the triads).

Second, content of packets with different survey versions was systematically varied before packets were given to partners/distributors for distribution. Research assistants prepared and collated packets in which successive questionnaires followed the various counterbalanced orders according to the samples shown in the bottom of Figure 2. In each packet of 14 or more questionnaires, adjacent POSHA-Es systematically varied in terms of general item order and at least two iterations of all anchors in the triad order. Third, the collated packets then were numbered and selected randomly to be given to distributors (explained below) who were “blind” to packet contents and questionnaire sequences (i.e., they had no knowledge of which respondent might receive which version of the questionnaire). Distributors were instructed to give out questionnaires sequentially, without mixing up the packets' order. All respondents received packets to which variously ordered surveys were randomly allocated.

Nonprobability Respondent Sampling Scheme

A central distributor gave questionnaire packets to other distributors who in turn gave individual questionnaires to potential respondents who either completed them or, in a few cases, recruited others to complete the survey. This is described by Dillman (1978) as the “person in charge” method. Specifically, the first author (KSL) gave packets of questionnaires to seven university “partners.” Four students were enrolled in an honors class, and two were engaged in independent research experiences. The seventh student was visiting from another institution. The students assumed the roles of distributing and collecting that international partners are expected subsequently to perform. The student partners distributed the questionnaires to family members, friends, acquaintances, and small groups. The only inclusion criteria were that survey recipients must be able to read English, be at least 18 years of age to comply with approved human subject protection policies, and be willing to consider filling out the POSHA-E. Four student partners received packets for distribution of 56 collated POSHA-Es each. They suggested this to be “…as many collated POSHA-Es as they believed they might be able to hand out to individuals likely to complete and return them.” Three student partners accepted 20, 22, and 37 questionnaires, respectively).

Return rate. Of 303 questionnaires distributed, 165 (55%) were returned. Distributor-specific return rates ranged from 35% to 89%. This response rate was not as uniformly high as some survey experts suggest (e.g., Dillman, 1978), but was high enough to obtain sufficient analyzable data when returns from all partners were combined into a single sample. This was encouraging especially since the survey was long. Subsequent inspection of demographic data suggested that the nonprobability sampling scheme resulted in respondents representing a wide variety of occupations and ages, the majority of whom were not members of distributors' extended families. That is, all partners were able to recruit respondents not only among family but also among friends, acquaintances, and friends of friends or relatives who were willing to complete the survey.

Demographic characteristics. Table 1, column 2, summarizes frequency distributions for demographic variables. The majority of respondents were women (71%) and had a mean age of 31.5 years. They were well educated, reporting a mean number of 15.3 years of education. One half were single and one-third, married. Thirty-six percent were students with the rest working for a salary. Virtually all (99%) reported English as their native language, and relatively few spoke other languages. Ninety-three percent reported their race as Caucasian, and 85% indicated their religious affiliation as Christian. From the fourth item of the general section where respondents were asked who they knew with each of the attributes, very few respondents in indicated that they were stutterers, mentally ill, wheelchair users, or old (2–5%). Somewhat more respondents indicated that they were overweight, left-handed, or multilingual (10-26%), and even more respondents marked that they were good talkers (30%) and intelligent (48%). Overall, as seen in such areas as language spoken, religious affiliation, and race, this was a relatively homogeneous group.

Figure 3

Table 1: Demographic data.

As noted, the intent of the nonprobability sampling procedure was not to achieve a representative (i.e., probability) sample but to explore the feasibility of methods that can be used in many cultures and in developing countries. Nevertheless, information about the extent to which the respondent sampling procedure produced a respondent sample deviating from census data could be used to inform analysis and utility of such sampling in interpretation and comparisons with public opinion data. Table 2 was prepared to compare the respondents in this field study with data from the US Census (US Census Bureau, 2000). It is clear that the collection procedure produced a sample of respondents, 67% (111/165) of whom were from West Virginia (WV), that was quite different from census characteristics of (a) the host county, (b) other counties representing at least 3% of the respondents, or (c) the state. US Census data indicate that only 15% of WV adults over 25 years report a bachelor's degree or higher. The table shows that the percent of ?25-year-old respondents whose demographic information reported at least 16 years of education (assumed to be generally equivalent to a bachelor's degree) in the sample groups is generally higher than would be expected in a probability sample of adults from counties and the state. The only notable exception occurred for the host county (Monongalia)—where 27% of the respondents resided—in which US Census data indicated that 32% of adults over 25 possessed bachelor's degrees or higher. More important, Table 2 reveals that county and state populations have an approximately equal sex distribution, different from the unbalanced distribution of the respondent sample, which contained about 2 ½ times more females than males.

Figure 4

Table 2: Respondent Distributions in the Survey Nonprobability Sample Compared with Census Distributions in the Host and Other WV Counties on Variables of Race, Sex, and Education

Distribution of orders in returned questionnaires. Of the 165 returned questionnaires, 46.7% were characterized by item orders in general sections that contained stuttering in the middle position, 20.0% had stuttering in the first position, and 33.3% in the last position. Obviously, the position distributions in returned questionnaires did not achieve even distributions across the three positions, even though respective percentages of distributed questionnaires were 32.3%, 37.0% and 30.7%. Nevertheless, the returned questionnaires had a more evenly proportionate distribution of positions in triad orders. The stuttering section occurred first among the triads in 30% of returned questionnaires, second in 36%, and third in 33%. Moreover, all 56 possible orders of the two detailed attribute (anchor) sections occurred in the 165 returned questionnaires. The frequency of occurrence for each possible order averaged 2.9 occurrences across all orders, which is the expected occurrence (165 / 56 = 2.9) for an equal distribution of orders. These results are not surprising given that individual distributors received packets for only one item order but up to all of the various triad orders.

General section order effects: Scaled ratings. In order to generate one comparable score, overall means of ratings for all nine attributes were combined from general section scale (i.e., 0 - 100) responses for comparisons among each of the three item orders, which are shown in row 1 of Table 3. The similarity among mean ratings of items appearing in each of the three orders, ranging from 54.1 to 56.6, indicate that the item order of attributes had little effect on the ratings.

Figure 5

Table 3: Overall Means for Three Different Item Orders of Stuttering in the General Section and for Three Different Triad Orders of the Detailed Sections for Stuttering.

These means reflect the combined ratings of all nine attributes across all 27 scale items from the first three prompts in the general section. To investigate potential order effects further, all pairwise t tests for independent observations compared the mean ratings of items eliciting opinions on overall impression, wanting to be, and knowledge about each attribute. For example, were there differences in ratings of overall impression for mental illness when stuttering occurred first versus last, or first versus middle, or middle versus last position? Of the 81 t tests, 74 (91%) were nonsignificant or had probability levels that were equal or greater than .05, with a mean probability level for the t tests of .48. The seven order pairs that were significantly different were not distributed in any recognizable pattern: stuttering, 1 of 9 pair-wise tests; overweight, 1 of 9 tests; multilingual, 3 of 9 tests; and mental illness, 2 of 9 tests. Nonsignificant comparisons in 91% of the tests support a conclusion that there were no systematic order effects for the order of stuttering in the general section.

It should be noted that a correction for multiple t tests, such as the Bonferroni correction (Maxwell & Satake, 1997) to minimize the probability for making a Type I error was not applied in these and in subsequent parallel comparisons. In these cases not correcting the alpha value is the more conservative alternative since our emphasis was on not making a Type II error (i.e., accepting—not rejecting—the null hypothesis that there is no difference between pairs of means) or concluding that a difference for first, middle, or last position for the stuttering attribute existed when in fact there was none.

Detail section order effects: Scaled ratings. In spite of the foregoing, it is evident that the order of stuttering items in the general section could potentially affect responses to stuttering items in subsequent detailed sections for stuttering. Again, to generate one comparable score, overall mean ratings of all judgments in the detailed sections of stuttering are shown in row 2 of Table 3 and were very similar, ranging from 42.9 to 43.7. Again, t tests compared all pairs of orders for the 85 scale items in the detailed stuttering section. Of the 255 pairs (85 items x 3 comparisons = 255), 247 (97%) were not statistically significant at the .05 level, with probability levels averaging .52.

Finally, and perhaps of greater interest, the overall mean ratings for detailed stuttering items were analyzed separately according to its occurrence in first, second, or third positions in the triads (row 3 of Table 2). Again, very few differences were found; 235 (92%) of the t test comparisons were nonsignificant. The mean probability level was .45.

General Section order effects: Categorical ratings. We also compared the percentages of respondents who knew someone with each attribute in the general section. (i.e., “nobody,” “acquaintance,” “close friend,” “distant relative,” “close relative,” “me,” or “other”). We ran Chi Square tests of the independence of expected versus observed frequencies for these experience categories. The categories were corrected for actual proportions of occurrence by adjusting the actual frequencies according to the number of returned questionnaires for each item order so that each set would represent one third of the respondents. The relevant data for item order are shown in Table 4.

Figure 6

Table 4: Overall Frequencies of Identifying Personal Experience with Nine Attributes According to Three Different Item Orders of Stuttering in the General Section.

These results were highly significant when stuttering appeared as the first or middle attribute in the general section item lists of nine attributes (p < .00001) but were not significant when stuttering appeared at the end of the list (p = .98). Table 4 indicates that respondents were most likely to categorize individuals (or themselves) with the various attributes (including “nobody”) when stuttering was the middle attribute. They were less likely to do so if stuttering occurred last but least likely when stuttering occurred first. Surprisingly, these differences in personal experience with attributes within the three orders occurred even though the attitudes regarding them did not (see Table 3). We cannot explain this difference but speculate that those distributors having POSHA-Es containing stuttering as the middle attribute of general items were more likely to distribute questionnaires to college students in speech-language pathology or other similar helping professions who seem to be more predisposed to carefully identify various attributes. Unpublished data from a preliminary report of a subset of the data reported here indicates that students identified many more people they knew with the various attributes than did adult nonstudents (St. Louis, Schiffbauer, Phillips, Sedlock, Hriblan, & Dayton, 2000). Order effects: Summary. Overall, the counterbalancing appears to have provided a reasonably consistent sampling of all possible orders. More important, it is quite clear that such counterbalancing had little if any systematic effect on the results, except perhaps for the unexpected findings of differences in categorization frequencies of various attributes in the previous section. We concluded that counterbalancing presentation orders was essentially unnecessary for future field studies and that respondents rated items similarly regardless of the order in which they were presented.

Several authorities on survey research have addressed the importance of readability, wording and question clarity in questionnaires (e.g., Babbie, 1990; Dillman, 1978, 2000; Fowler, 1995). A general principle among experts is “keep it simple.” However, Dillman (1978; 2000) also discussed the pitfalls of over-simplification in wording and item construction.

Components of the POSHA-E were analyzed according to the computerized Flesch Reading Ease Index that is based on the average number of syllables per word and the average number of words per sentence (Evaluating Readability Statistics, 1993-94). Index scores range from 0 to 100, wherein standard writing (e.g., the target level for most newspapers) averages 60 to 70. The higher the score, the greater the proportion of the population who can readily understand the document. A 60-70 Flesch readability index corresponds to writing at the seventh to eighth grade level.

The readability rating of the POSHA-E instructions component was 58.2 or at the 9.4 grade of school reading levels. This score means that populations who read at a mid-ninth grade level or higher would understand the instructions. Flesch readability estimates of the general and detailed sections ranged from 53.4 to 88.3 (mean = 70.6). In other words, text grade-level ranged from 2.8, or early second grade, to 7.8 or nearly eighth grade with a mean of 5.3 or early fifth grade level.

Thirty of the 165 respondents (18.2%) wrote comments about the POSHA-E at the end of their questionnaires. Seven (4.2%) commented on the limited response options, such as the lack of a “doesn't matter” or that marking a 1-5 scale would be better. Six respondents (3.6%) highlighted examples of unclear questions; another six stated that the questionnaire was too long. Five respondents (3.0%) described the questionnaire as boring, uninteresting, making no sense, or dumb. Another five criticized the requirement to generalize to all persons with a given attribute. Finally, eight (4.8%) suggested other topics for inclusion in future surveys.

This field study generated valuable data for proceeding with the IPATHA initiative. Whereas surveys are often generated and tested with limited attention to methodological issues presumed not to affect results, the task force believed that the POSHA-E must undergo rigorous evaluation of such concerns. In general, order of stuttering items did not affect the results in any large or systematic ways. With a carefully constructed counterbalancing and distribution scheme, convenience sampling could achieve unbiased distribution of different versions of the POSHA-E. A quasi-continuous scale can be used to measure opinions of stuttering and other attributes; however, respondent errors and omissions must be noted, even though they likely do not affect overall response results. Errors in measuring scaled responses with a special ruler and errors in tallying do not apparently affect results. Nevertheless, we concluded that the inconvenience, time, and difficulty of measuring and scoring a quasi-continuous scale is not consistent with the IPATHA goal of simplicity and user-friendliness. Furthermore, problems associated with computer formatting adaptations, spacing, varying sentence length, and pagination in various iterations of the survey in multiple languages become evident for ruler calibrations.

Not surprisingly, the model of convenience sampling employing partner distributors did not achieve a respondent sample that was representative of the geographic population of adults; however, it did achieve a sample that reflected a considerable range of ages, occupations, religious affiliations, marital status, and other demographic variables. Finally, although most respondents filled out the questionnaire without comment, those who did comment were likely to point out problems of length, confusion, and difficulty.

A field study was conducted to field-test the first experimental prototypes for the Public Opinion Survey of Human Attributes (POSHA-E), a paper-and-pencil instrument designed to identify public attitudes toward stuttering and other attributes among multi-national populations. This report does not focus on attitudes but on characteristics of the questionnaire and very importantly on identification of methodological problems that we expected in cross-cultural research. Field-testing the POSHA-E helped to identify its strengths and weaknesses with regard to procedural efficiency, sample representativeness, order effects, scoring ease and accuracy, data reduction, and response reliability and validity. A number of companion field studies informed by these efforts are already under way.

The nonprobability sampling scheme was carried out by seven different student partners, each distributing questionnaires to acquaintances, friends, family members, and others known by these individuals. The most important caution of these data collection procedures is that they produced a respondent sample biased by selection. The selection resulted from cultural, social, and educational characteristics of the selecting partner. A major consequence is that results from analyses of this respondent sample are not generalizable to public opinion and attitudes. This sample helps to inform the need for control strategies when data are analyzed. The return rates from 35% to 89% per partner suggests that this strategy could effectively be replicated with other national and international partners.

Length

Order Effects

Readability

Rating and Tallying Accuracy

Other field tests of POSHA-E are under way around the US and in other countries (St. Louis, 2005). Investigations are under way to estimate reliability and validity of the instrument. POSHA-E translations to other languages have also begun. We will compare survey results from translated versions to results from American English versions and identify problems inherent in translations, which despite our best efforts to eliminate them may be influenced by American English idiomatic expressions.

When additional field test data are available, item analyses will determine which items are most discriminating and which are redundant or not useful. The next activity is to develop a shortened, revised version of the POSHA-E and to field-test it with convenience and probability samples here and abroad. Inclusion of salary strata for the demographic section is under consideration to have direct SES assignment rather than having SES assigned from proxy variables.

With acquired sampling information, we are alerted to specific sampling questions for the larger international project. As noted above and shown in Table 2, respondents did not represent the area population. Educated persons and white females were over-represented. Future field studies will be designed to investigate these distributions in other non-representative samples and compare these distributions with results of probability sampling methods in some settings.

Considering that many international samples will not be generated by probability sampling methods, the possibilities for control at analysis rather in design, weighting procedures for sample representation, and other sampling manipulations can be explored for data generated in other cultures and communities.

This initial field study represents progress in moving toward the long-range goal stated in the introduction, that is to explore specific methodological issues in the ongoing process of developing an instrument to measure public attitudes toward stuttering. In this process, we are amassing a vast amount of data from on-going field studies that can be analyzed to identify country- and culture-specific attitudes and biases toward stuttering, other conditions, and differences. Inefficiency with the quasi-continuous scale and the use of nonprobability samples notwithstanding, this and other field studies under way allow us to carry out analyses of comparative data on attitudes toward stuttering within the context of other human attributes. Such contextual information can provide preliminary baseline data to inform strategies to modify negative attitudes. Accordingly, we intend to generate baselines indicating where attitudes are more or less positive and how attitudes are associated with one other.

The data from these ongoing and subsequent studies will allow analyses of demographic variables to determine differences in attitudes in controlled, stratified and appropriately weighted analyses of respondent populations' characteristics. Such knowledge about factors associated with people's attitudes is postulated as an essential foundation for those attempting to change attitudes.

This paper is dedicated to the late Charles C. Diggs, who was on the original task force. He would have been a coauthor of this paper were it not for his untimely death in 2006. We wish to acknowledge the valuable assistance of Jami Ebbert, Sharon Helman, James D. Schiffbauer, Carolyn I. Phillips, Andrea B. Sedlock, Lisa J. (Hriblan) Gibson, Rebecca M. Dayton, Amanda K. Cassidy, Rachel Fender, Amanda K. Efaw, Maren Greig, and Brenda Davis in collecting, scoring, tallying, and analyzing data.