University of FloridaSolutions for Your Life

Download PDF
Publication #AEC395

The Savvy Survey #5: The Process for Developing Survey Questions1

Glenn D. Israel and Jessica L. Gouldthorpe2

Introduction

Continuing on in the Savvy Survey Series, this fact sheet is one of four that provide an introduction to important aspects of developing a questionnaire. A survey that provides useful information requires careful design and attention to detail. Several sources provide in-depth information on developing surveys (Dillman, Smyth, and Christian 2009; de Leeuw, Hox, and Dillman 2008). This publication focuses on the process of developing survey questions to help ensure the “right” questions are included in the questionnaire.

Based on the experience of the authors in creating numerous survey instruments, one useful process for developing a high-quality questionnaire follows four steps.

1. Brainstorm questions on a topic.

2. Review the literature for concepts, questions, and indices.

3. Collect ideas for questions and wording choices through focus groups or key informant interviews.

4. Select questions linked to a program’s logic model.

Each of these steps is discussed in more detail below.

Step 1. Brainstorming Questions

Many questionnaires have their roots in a brainstorming session in which the project team works to develop a list of questions related to the topic at hand. One benefit of brainstorming is that a large number of questions can be generated very quickly. Brainstorming can work very well with small and medium-sized groups (6 to 10 people) to develop a set of questions to build into a questionnaire. For example, Figure 1 shows some of the questions agents developed during a brainstorming session for a landscape maintenance survey designed for homeowners living in neighborhoods with homeowners associations.

Figure 1. 

Agents' brainstormed questions.


[Click thumbnail to enlarge.]

On the other hand, if only the brainstorming process is used to create the question, then three problems can occur. First, important questions might be omitted because no one thought of them at the time. Second, questions that are irrelevant or tangential to the main topic might be included. And third, the organization of the questions can be haphazard because of the nature of the brainstorming process.

Step 2. Reviewing the existing literature for concepts and questions

To avoid omitting important questions or including irrelevant ones, a best practice is to consult studies published on the topic area. Journal articles are a good source of ideas about concepts that researchers are measuring and have found to be important. Frequently, the questions used to measure a concept are included in an appendix.

Studies published in peer-reviewed journals often include a set of questions that form an index or scale to measure people’s knowledge or attitudes related to a subject area (along with estimates of their reliability and validity). For example, one might want to measure attitudes about the environment for a needs assessment study (because these attitudes are seen as a precursor to behavioral intent and subsequent behaviors, see Hines, Hungerford, and Tomera 1987). Or, one might want to measure change in those attitudes before and after an environmental education program. In either case, existing, well-developed measures (see Dunlap and Van Liere 1978 and Dunlap et al. 2000) may be used as-is or modified slightly to fit the survey situation. Finally, collections of scales on many topics have been published in books (see, for example, Miller and Salkind 2002).

Step 3. Collect ideas for questions and wording choices through focus groups or key informant interviews

Collecting information though focus groups or key informant interviews can help to identify important questions for the survey. Interviews can also reveal how people think about the survey topic and the words that they use to express their ideas. Using this information can help ensure that questions are asked in a meaningful way. Moreover, talking with people who have different perspectives, such as residents, homeowner association officers, and landscapers in the example of the homeowner maintenance survey described above, builds a broader foundation for developing a respondent-friendly questionnaire.

Step 4. Selecting questions based on logic models

Developing a logic model for a program is a recommended best practice by program planners and evaluators (Israel 2001; 2010). However, logic models are also useful for ensuring that questions in the instrument are relevant and organized in a logical sequence. This is especially true when the logic model displays key elements of the program’s context, major activities, the sequence of outcomes, and the causal connections between each part (see Figure 2 for an example of a logic model that shows the causal pathways of the outcomes, and that was used to develop a successful homeowner survey). In this example, each component in the model was used to identify questions to be included in the instrument and to determine whether the questions identified during the brainstorming session were relevant. This led to the selection of a series of questions to collect information about homeowners’ fertilizer use (some of these questions are shown in Figure 3).

In Summary

Developing a high-quality questionnaire is a critical step for collecting useful data for assessing program needs and evaluating the outcomes of programs. By following a process of brainstorming questions, reviewing relevant research studies, collecting questions and wording choices through interviews, and linking questions to a logic model, one can create a set of appropriate questions. In the final selection process, it is also a good practice to use previously developed measures and to focus on measuring a few things well rather than trying to measure too many things and possibly sacrificing accuracy.

Figure 2. 

Logic model for landscape maintenance survey designed for homeowners living in a neighborhood with drainage into a surface water body.


[Click thumbnail to enlarge.]

Figure 3. 

Questions linked to the fertilizer input element in the logic model.


[Click thumbnail to enlarge.]

References

de Leeuw, E. D., J. J. Hox, and D. A. Dillman, Eds. 2008. International handbook of survey methodology. New York, NY: Lawrence Erlbaum Associates.

Dillman, D. A., J. D. Smyth, and L. M. Christian. 2009. Internet, mail, and mixed-mode surveys: The tailored design method. (3rd ed.). Hoboken, NJ: John Wiley and Sons.

Dunlap, R. E., and K. D. Van Liere. 1978. The “new environmental paradigm”: A proposed measuring instrument and preliminary results. Journal of Environmental Education, 9, 10–19.

Dunlap, R. E., K. D. Van Liere, A. G. Mertig, and R. E. Jones. 2000. Measuring endorsement of the new ecological paradigm: A revised NEP scale. Journal of Social Issues, 56(3), 425–442.

Hines, J. M., H. R. Hungerford, and A. N. Tomera. 1987. Analysis and synthesis of research on responsible environmental behavior: A meta-analysis. Journal of Environmental Education, 18(2), 1–8.

Israel, G. D. 2001. Using logic models for program development. Gainesville: University of Florida Institute of Food and Agricultural Sciences. Available at http://edis.ifas.ufl.edu/WC041.

Israel, G. D. 2010. Logic model basics. Gainesville: University of Florida Institute of Food and Agricultural Sciences. Available at: http://edis.ifas.ufl.edu/wc106.

Miller, D. C., and N. J. Salkind. 2002. Handbook of research design and social measurement. (6th ed.). Los Angeles

Footnotes

1.

This document is AEC395, one of a series of the Agricultural Education and Communication Department, UF/IFAS Extension. Original publication date August 2013. Visit the EDIS website at http://edis.ifas.ufl.edu.

2.

Glenn D. Israel, professor; and Jessica L. Gouldthorpe, doctoral candidate; Agricultural Education and Communication Department; UF/IFAS Extension, Gainesville, FL 32611. The authors wish to thank Paul Monaghan, Norma Samuel, Nick Fuhrman, and Marilyn Smith for their helpful suggestions on an earlier draft.


The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex, sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other UF/IFAS Extension publications, contact your county's UF/IFAS Extension office.

U.S. Department of Agriculture, UF/IFAS Extension Service, University of Florida, IFAS, Florida A & M University Cooperative Extension Program, and Boards of County Commissioners Cooperating. Nick T. Place, dean for UF/IFAS Extension.