Introduction
It is a good practice to pretest a questionnaire after the initial design and formatting are completed. There are several reasons to pretest a questionnaire. One is that no one is perfect in writing questions. Another is that a survey is designed for a specific population group and, without having input from that population, we cannot assure ourselves that questions and statements are well written and capture the information we intended to collect (Campanelli, 2008).
Pretesting is the process of evaluating the questionnaire and survey procedures in advance to assess whether they are going to cause any problems for respondents and interviewers and whether the survey is meeting its intended objectives (Presser et al., 2004). The amount and rigor of the pretesting should be appropriate to the intensity and resources of the associated Extension program (Israel, Diehl, and Galindo-Gonzalez, 2009). For example, a panel of peer Extension professionals and an evaluation specialist might be asked to review a simple client satisfaction survey or a retrospective pretest/posttest instrument. In contrast, a panel of experts might be combined with cognitive interviews and a pilot test of a questionnaire to be used for a statewide needs assessment or evaluation.
In pretesting, the survey designer evaluates how well the instrument facilitates the four cognitive steps of responding to a question: comprehension, recall, judgment, and reporting. According to these cognitive steps, the instrument produces erroneous information if respondents misunderstand the survey questions, are unable to recall the requested information, use an improper mental judgmental process to answer the questions, or conceal/distort some information and provide a socially desirable answer. Overall, comprehension is a major concern (Campanelli, 2008).
Based on studies conducted by researchers, pretesting can be implemented using various methods (Campanelli, 2008). A number of possible pretesting methods are described below.
Methods of Pretesting
Survey Development and Critique by Designer and Colleagues
The following methods are steps conducted by an Extension agent or specialist, often with help from a team of collaborators. These steps represent an "insider's" perspective of the survey.
Questionnaire development: This is the initial stage in the questionnaire development process, where the Extension agent or specialist does the background work of understanding subject matter by studying related literature and understanding cultural and personal characteristics of the target population. This background information can help an Extension team formulate appropriate questions or allow for adoption of previously developed, validated indexes or questions.
Question critique: The questionnaire designer assesses the wording of questions to avoid problems such as double-barreled questions (two questions in one question), negative questions (questions which contain negative words such as "not"), overly long questions, etc. It also includes assessing the flow of the questionnaire as a whole to ensure that the question order and transition statements make sense.
Informal testing: Informal methods include reading the questionnaire aloud to identify the differences between written and spoken language (which is important for face-to-face interviews) and difficulty in comprehension of the question. Another informal testing method is interviewing oneself, such that you play the role of respondent and try to answer each question. This can help identify difficulties in the question-answer process. A final informal method is the mock interview with a colleague.
Panel of experts: Experts are agents or specialists who have extensive experience or knowledge of a particular topic, field issues, questionnaire design and testing, and cultural perspectives of the survey, as discussed in the introduction. Experts can be consulted individually or in a group. Normally a panel of three to four experts and a survey designer are considered appropriate to review the questionnaire. Consultation with experts provides good initial feedback for the survey designer about problems with the questionnaire.
Systemic review of the questionnaire: While using a panel of experts is a less systematic and more informal method of survey pretesting, a systematic review compares a questionnaire to available standard checklists. This is often an initial step in pre-testing, where the designer of the questionnaire compares the newly developed questionnaire with a standard checklist to identify flaws in the questions. The systemic review involves no respondents to the questionnaire, while the questionnaire testing phase traditionally involves gathering responses to the questionnaire from a small number of respondents.
Both the panel of experts and systemic questionnaire review provide inexpensive, quick feedback and can uncover a wide range of potential problems, but some concerns with these methods are their dependency on the capabilities and availability of experts and lack of target population involvement (Campanelli, 2008).
Methods Involving Members of the Target Population
In contrast to the previous section, the methods described below are focused on obtaining input from members of the target population. These methods allow the survey developer to gain more insight into the cognitive processes of comprehension, recall, judgment, and reporting. Problems can be identified directly, for example, when a respondent says, "I don't know what this word means" or "I can't remember what I did about that last week." Problems also can be identified indirectly when, for example, unanticipated answers are given during the reporting process for self-administered questionnaires.
Pilot testing: Pilot testing is conducted to assess the whole questionnaire under actual survey conditions. The primary benefit of pilot testing is identifying problems before implementing the full survey. Pilot testing includes assessing the validity of each question (whether the question is capturing information it is intended to measure to meet the goals and objectives of the study) and analyzing the various aspects of questionnaire as a whole, such as:
- the flow of questions (whether the questions are arranged logically)
- the first-contact response rate (whether the response rate after the first contact meets expectations) as well as response rates for subsequent contacts
- time taken for responses to each question
- length of the questionnaire (whether it can be completed by respondents in the amount of time estimated)
- instances of confusion (whether a question or phrase in a question causes confusion for respondents)
- pattern of answers (how the different respondents answered the same question or sets of questions)
- possible answer categories for coding open-ended questions (receive input on possible open-ended responses to plan for coding)
A pilot test is also called a field test. There is no prescribed sample size for a field test; various researchers suggest sample sizes from 10–25 (Sheatsley, 1983) to 20–50 (Sudman, 1983), but the final sample size decision is made by the researcher based on available time and budget to carry out the field test. Larger sample sizes are recommended to achieve more robust testing results of the questionnaire.
The sample for a field test should be representative of the target population. If the questionnaire is intended to be administered through face-to-face or telephone interviews, the agent or specialist interviews members of the selected sample; for self-administered (e.g., mail or web) questionnaires, the surveys are sent to a selected sample. Once the data collection for the field test is complete, data analysis identifies problems with each question and questionnaire or interview as a whole (Campanelli, 2008). An example of a field test is discussed in Figure 1.
Credit: UF/IFAS
The Weeds in the Water survey was developed to collect data from freshwater recreational boaters and anglers to identify opportunities for targeted educational programming on aquatic invasive species (Wilson, 2012). The survey development process included several steps. First, the initial questionnaire was drafted and reviewed by a panel of experts. Second, cognitive interviews were conducted with six respondents from the target population. Finally, a field test was conducted using a sample size of 100 (with two-thirds from the boater list and one-third from the angler list).
The field test entailed sending a complete survey packet (cover letter, questionnaire, and postage-paid return envelope) to the sample. Forty persons responded to the pilot survey but this included 15 (38 percent) who were saltwater-only boaters and not in the target population. To better reach the target population, the survey team decided to exclude all boaters who resided in a coastal community. This procedure helped to eliminate many of the saltwater boaters from the full-scale survey (only 20 percent did not boat during the past year or were saltwater boaters). Without the field test, this problem would not have been detected until later. Moreover, it helped to save money by avoiding postage costs from sending questionnaires to "ineligible" boaters.
Respondent debriefing: In this method, special follow-up questions are asked to assess a respondent's understanding of survey questions after completion of the survey. With the help of respondent debriefing, the survey designer is able to identify how respondents understand the terms and phrases used in the questions and their overall understanding of the questionnaire. This also helps the survey designer determine how effective the questions are in capturing the intended information (see the example case study in Figure 2). Respondent debriefing also can be used to understand memory and judgment issues of respondents while answering the questions. Respondent debriefing also is called special probing or frame-of-reference probing. Some challenges with respondent debriefing include identification of potential problems in advance of the debriefing session and difficulty in writing good debriefing questions (Campanelli, 2008).
The Your Florida Yard and You survey was developed to collect baseline data on homeowners' fertilizing and irrigation practices, as well as to identify opportunities for targeted educational programming. Respondent debriefing was used by Extension faculty to identify problem questions in the survey. First, a group of Master Gardeners in Brevard County completed the draft survey. Then the survey team reviewed individual questions and the flow from section to section with the Master Gardeners. Based on the debriefing process, several questions were revised and others were moved to improve the flow.
For example, the initial version of a question asked, "Who makes the decisions about maintaining the lawn or landscape?" and included three response options: "I do," "Other member of the household," and "Lawn service provider." Debriefing participants suggested re-wording the second option to "Spouse or other member of the household" and adding a fourth option, "Homeowners association."
Because the survey was intended for residents living in subdivisions that may have a homeowners association, the new response option was an important improvement. Subsequently, the survey was used to collect nearly 500 responses in Brevard County (Israel, Kochert, Scalera, & Monaghan, 2011).
Focus group interviews: Focus group interviews are small group discussions under the direction of a moderator whose job is to keep everyone focused on the main topic. A possible scenario using focus groups for pretesting might include distributing the questionnaire before the start of the interview and asking participants to complete it. Once the respondents complete the questionnaire, they take a break. During the break the moderator and support staff analyze the questionnaire responses. After the break the moderator uses preplanned debriefing questions and spontaneous questions based on the review of the questionnaires as discussion topics to uncover issues with the questionnaire. The advantage of a focus group is that it encourages interaction among participants—i.e., it provides an opportunity for participants to respond to what other participants say and to hear other ideas—often yielding deeper insights to uncover problems with the questionnaire (Campanelli, 2008).
Behavior coding: This method is used for face-to-face or telephone surveys. It documents the way in which the interview was carried out, whether the interviewer read the questionnaire as worded to respondents, and whether the respondent interrupted the interviewer. In behavior coding, the interviewer can either use pre-designed codes or develop codes spontaneously. This is a quantitative method in which the behavior of respondents is counted; for example, 60 percent of respondents asked the interviewer to repeat a specific question. One benefit to this method is that it easily can be combined with field testing to evaluate the questionnaire in a more holistic way. Audio recording of behavior coding is required to count the frequencies and analyze problems. Less than 85 percent exact reading of the question by the interviewer and less than 85 percent adequate answers by the respondents is considered indicative of a problem with the question. Examples of behavior coding include how many times respondents interrupted the interviewer, how many times respondents asked the interviewer for question clarification, etc. The major challenges with behavior coding are that it requires a lot of time to implement, proper training of coders, and careful writing of codes. Sometimes even after investing much time and effort in behavior coding it is possible to miss finding issues with questions (Campanelli, 2008; Presser et al., 2004).
Cognitive interviewing: This is an in-depth interview process which is used mainly to understand the cognitive/mental/thought process used by respondents to answer survey questions. This process uncovers all four cognitive steps (comprehension, recall, judgment, and reporting) used by the respondents while answering questions. The motive of this type of interview is not to find the answer to questions but to find out how respondents arrive at their answers. With the help of this process the interviewer not only identifies problems with the questions but also gets an idea about possible solutions.
Cognitive interviews can be conducted at a lab or at any place which is free of noise and interruption. Whatever the location, the recording of cognitive interviews is recommended. The sample size used in this process is usually very small, and the interviewer uses a convenience sample. To uncover the thought process of respondents, techniques such as think aloud, probes, and observation can be used.
- Think aloud: Respondents are asked to verbalize all of their thoughts while answering the question. This can be done concurrently while answering the question or retrospectively, where the respondents are reminded of all the questions they answered and asked to explain why or how they came up with the answers. One example of a think-aloud question would be, "What is your job status—part time or full time?" The think-aloud response might be, "This is a tricky question. Sometimes I feel that I am full time and sometimes I feel that is not correct. As I am working at two places, one place three hours a day and other place four hours a day, that means seven hours a day in total, but I work six days a week, so a total 42 hours a week. By considering all these I think I am a full-time employee compared to part-time."
- Probes: These are special questions, similar to respondent debriefing, to explore the answers provided by respondents. For example, if a respondent is taking a long time to answer a question, the interviewer can interrupt by saying, "You look puzzled. Do you need any clarification for this question?"
- Observation: An interviewer observes the respondents' actions for clues about thought processes as they are responding to each question.
In practice, cognitive interviews are not conducted in isolation and either think-aloud, probes, or observations are used as part of the cognitive interview. A possible issue with cognitive interviewing is over-generalization of cognitive interview results, as a very small convenience sample typically is used for interviews. Other issues include the need for highly skilled interviewers and the time-consuming nature of analyzing these interviews (Campanelli, 2008; Presser et al., 2004).
Conclusions
According to survey experts, every method discussed in this document contributes in specific ways to finding problems with questionnaires. These are summarized in Table 1 below. In most situations, a combination of questionnaire pretesting methods is useful. The combination of methods depends on the time and budget available for pretesting of the questionnaire and the nature of the questionnaire itself. For best questionnaire pretesting, a four-step process is recommended. The first step includes the informal methods discussed above. This would be followed by the use of a panel of experts or systemic review of the questionnaire. Step three would be the use of cognitive interviews or focus groups, and the final step would be testing the questionnaire in the actual field situation as a field test.
The four steps mentioned above require considerable time and resources. A simpler method would be to conduct informal testing, followed by in-depth testing such as panel of experts, cognitive interviews, or focus groups and then the field test. In any case, some pretesting is recommended. No matter what techniques are used, it is important for Extension professionals to revise the questionnaire and survey procedures to incorporate lessons learned from the pretesting.
References
Campanelli, P. (2008). Testing survey questions. In D. de Leeuw, J.J. Hox, & D. D. Dillman (Eds.), International handbook of survey methodology (pp. 176–200). New York, NY: Lawrence Erlbaum Associates.
Israel, G. D., Diehl, D., & Galindo-Gonzalez, S. (2009). Evaluation situations, stakeholders, & strategies. (WC090). Gainesville: University of Florida Institute of Food and Agricultural Sciences. Retrieved from: https://edis.ifas.ufl.edu/publication/wc090
Israel, G. D., Kochert, J. L., Scalera, S. A., & Monaghan, P. F. (2011). Uncovering Barriers to Changing Homeowner Landscaping Practices. Paper presented at the 17th International Symposium on Society and Resource Management, Madison, WI.
Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1), 109–130.
Sheatsley, P. B. (1983). Questionnaire Construction and Item Writing. In Rossi, P.H., Wright, J.D., & Andersen, A.B. (Eds.) Handbook of Survey Research (pp. 195–230). New York: Academic Press.
Sudman, S. (1983). Applied Sampling. In Rossi, P.H.; Wright, J.D. & Andersen, A.B. (Eds.) Handbook of Survey Research (pp. 145–194). New York: Academic Press.
Wilson, K. L. (2012). Florida Freshwater Boater and Anglers' Awareness and Perceptions of Aquatic Invasive Species and Adoption of Preventive Behavior. Unpublished manuscript. University of Florida, Gainesville.
