The Savvy Survey #11: Mail-Based Surveys
Credit: Thinkstock Images/Stockbyte/Thinkstock
As part of the Savvy Survey Series, this publication provides Extension faculty with an overview of the process that uses mail for sending and receiving questionnaires. Mail surveys can be an effective way to collect data for needs assessments in program planning (see, for example, Gaul, Hochmuth, Israel, & Treadwell, 2009) or for follow-up surveys evaluating outcomes of Extension programs (see, for example, Israel, Easton, & Knox, 1999). Survey research consistently shows that mail surveys usually obtain as high or higher response rates than other methods (Messer & Dillman, 2011; Israel, 2013). This fact sheet provides guidance for constructing the questionnaire, addressing visual design and formatting considerations, and implementing the survey. Extension faculty who incorporate best practices of questionnaire design and mail survey procedures will be able to achieve a high response rate and collect more useful data than those who do not use these best practices.
Considerations Specific to Constructing a Mail Questionnaire
One of the major factors affecting the construction of a mail questionnaire is the available space on the printed page. Because of this limitation, mail questionnaires are best designed having two pages (a single sheet of 8½-x-11-inch paper printed front and back) or four pages (11-x-17-inch paper folded in half). The latter format can also be assembled into booklets of 4, 8, or 12 pages without adversely affecting response rates. Not only that, but legal sized paper can be used to create booklets with 7-x-8½-inch pages.
Once the format has been chosen, the survey designer can proceed to the questionnaire's content, constructing it in a way to fit the paper format. As discussed in The Savvy Survey #5: The Process of Developing Survey Questions, a logic model can be used to help identify and select the best questions to include in the questionnaire.
A second consideration that many fail to take into account is the survey packet's weight, which includes the questionnaire, cover letter, return envelope, and out-going envelope. When the weight of the survey packet is less than one ounce, postage costs can be held to the first class letter rate. If the survey packet weighs more than one ounce, however, it is still recommended that first class postage is used for the survey reminder letters to control their timing and sequence. Using bulk mail postage is not recommended because people associate it with junk mail and discount the survey's importance.
Visual Design and Formatting
With the survey's paper format chosen, it is time to consider the survey's visual design and format. The appearance of individual questions, as well as the entire questionnaire, can impact whether and how people answer the survey (Dillman, Smyth, & Christian, 2014). A good design provides cues to a respondent on how to navigate from one question to another and how much information the sender is seeking. For example, the size of the answer space for the question, "What could we do to improve our services to you and others in the county?" tells a respondent how much to write. A larger space encourages people to write more than a smaller space (see Israel, 2010).
Also, a "clean" design clearly and consistently indicates where the questions start and where the associated answer choices are located. This feature helps a respondent answer the questions accurately. Figure 1 shows a side-by-side comparison of a survey's original design with its revised version, using best practices in visual design. For example, the version with good visual design uses a bold font to distinguish the question from the response options, which are in a normal font. In this example, it is easy to see where the questions are located and how to move from one to the next.
For more detailed information about writing items or for formatting a questionnaire, see Publications #6a-e and #7 of The Savvy Survey series.
Implementing the Mail-based Survey
Preparation
Getting ready to send a mail survey involves several steps. One is preparing the correspondence that will be used in the survey process. A pre-letter is often sent to introduce the survey to the potential respondent, followed by a survey packet with a cover letter, questionnaire, and return envelope. Each letter or postcard should be carefully worded using the principles of social exchange theory to encourage people to complete the survey (Dillman, Smyth, & Christian, 2014).
Appendix A provides a set of figures illustrating a series of messages, with each one tailored to play a specific role in the survey process. While examining these examples, note how the dates chosen demonstrate the timing for the sequence of the contacts. In addition, the correspondence should provide all of the information needed for a person to make an informed decision to participate in the survey. (It is a best practice — and may be required — to have the questionnaire and letters reviewed and approved by the Institutional Review Board (IRB) to ensure compliance with regulations for conducting research on human subjects. See The Savvy Survey #9: Gaining Institutional Review Board Approval for Surveys for more information on this topic.)
When preparing the correspondence, consider how to personalize it. Research shows that personalization has a small but significant effect on increasing response rates (Dillman et al., 2014). Personalization helps connect the respondent to the survey. This can be done by using individual names of clients in contact messages (e.g., Dear Joe Client), using a group name that clients identify with (e.g., Dear Jackson County Cattlemen), or by signing individual letters in blue ink, showing the survey's importance to the respondent and showing it is not a simple photocopied paper.
A second step is preparing a mailing list with names, addresses, and other relevant information. Spreadsheet programs, such as Microsoft Excel, are useful for organizing information and keeping track of which people have responded and which have not. Table 1 illustrates a spreadsheet list, which includes a column listing the response status and one recording the date on which a completed questionnaire is received. In addition, data in a spreadsheet can be used in a mail merge to personalize cover letters by inserting names and other information.
A third step is estimating the number of copies needed for the survey. This estimate should include mailing envelopes, return postage-paid envelopes, letters, reminder post cards, and the questionnaire. Depending on the number of reminders, the quantity of items can be two or three times greater than the number of names on the mailing list. Once the estimate is made, supplies should be ordered.
Sending Survey Invitations
The survey invitation procedures should be tailored to the situation, as recommended by Dillman, Smyth, and Christian's "Tailored Design Survey Method" (2014). Table 2 shows examples of contact procedures. For mail surveys, a pre-letter is often used to alert potential respondents about the survey, followed by the survey packet, consisting of a cover letter, questionnaire, and postage-paid return envelope, mailed a few days later. Stamped or postage-paid envelopes should be provided, so respondents do not have to pay for mailing the completed questionnaire; this will also encourage many to respond. A best practice is to use multiple contacts to obtain a high response rate and minimize the risk of nonresponse bias.
Depending on the available resources for postage and targets for response rates, the number of contacts for the mail surveys can be more or less. In the case of the FCES customer satisfaction survey, a fifth contact was added in 2012 to reach the target of a 60% response rate.
In Summary
This publication in The Savvy Survey series has focused on procedures for constructing and implementing a mail survey. Specifically, this fact sheet discussed how the number of pages and weight of the survey packet are important considerations in the questionnaire construction. In addition, the questionnaire's visual design and format are critical in making a respondent-friendly questionnaire. Steps for preparing the correspondence and sending invitations for completing the survey were also reviewed, with an emphasis on using multiple contacts as a best practice for mail surveys.
References
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, Phone, Mail, and Mixed-mode Surveys: The Tailored Design Method. (4th ed.) Hoboken, NJ: John Wiley and Sons.
Gaul, S. A., Hochmuth, R. C., Israel, G. D., & Treadwell, D. (2009). Characteristics of Small Farm Operators in Florida: Economics, Demographics, and Preferred Information Channels and Sources. Gainesville: University of Florida Institute of Food and Agricultural Sciences. https://edis.ifas.ufl.edu/publication/wc088
Israel, G. D., Easton, J. O., and Knox, G. W. (1999). Adoption of Landscape Management Practices by Florida Citizens. Hort Technology, 9(2):262–266. https://doi.org/10.21273/HORTTECH.9.2.262
Israel, G. D. (2010). Effects of Answer Space Size on Responses to Open-ended Questions in Mail Surveys. Journal of Official Statistics, 26(2):271–285.
Israel, G. D. (2013). Combining Mail and E-mail Contacts to Facilitate Participation in Mixed-mode Surveys. Social Science Computer Review, 31(3):346–358. https://journals.sagepub.com/doi/10.1177/0894439312464942
Messer, B. L., and Dillman, D. A. (2011). Using Address-based Sampling to Survey the General Public by Mail vs. Web Plus Mail. Public Opinion Quarterly, 75(3):429–457. https://doi.org/10.1093/poq/nfr021
Appendix A
