University of FloridaSolutions for Your Life

Download PDF
Publication #PEOD9

Sampling Issues: Nonresponse1

Glenn D. Israel2

A number of issues related to sampling are reviewed by the author in Sampling The Evidence Of Extension Program Impact and Determining Sample Size. During the discussion of sampling, it was noted that sample size referred to the number of responses that need to be obtained. But no matter how well the sampling design is planned, a poor response rate to a mail or telephone survey or to interviews can render a study virtually useless. In an effort to obtain enough data for the analysis, many researchers commonly add 10% to the sample size to compensate for persons that the researcher is unable to contact. The sample size also is often increased by 30% to compensate for nonresponse. Thus, the number of mailed surveys or planned interviews can be substantially larger than the number required for a desired level of confidence and precision. However, inflating the sample size does not necessarily address potential bias from nonresponse. The following section discusses strategies for addressing nonresponse.

REASONS FOR NONRESPONSE

Nonresponse refers to the failure to obtain observations on some elements selected for the sample (Kish, 1965:532). Four common sources of nonresponse are:

  • Not-at-Homes

  • Refusals

  • Unable to Answer

  • Not Found

Not-at-homes are people who reside at the location but are away (e.g. for vacation, work) at the time when the interviewer attempted to make contact. Refusals are persons who are contacted but refuse to answer. Refusals may result from apathy, fear of invasion of privacy or any number of reasons. Some refusals are partial, where the respondent will answer some questions but not all. Others are temporary, where the respondent will answer on the second or third contact (Miaoulis and Michener, 1976). Unable to answer includes persons who are too infirm or temporarily incapacitated. Finally, Not Founds are persons who have moved and left no forwarding address, are deceased, or are not contacted due to error in the survey procedures.

NONRESPONSE BIAS

The consequences of nonresponse can vary. As nonresponse increases, the potential for a biased sample increase. This means that the obtained responses of a probability sample may no longer be representative of the larger population. In short, response bias can reduce a probability sample to what is essentially a convenience sample and consequently, the conclusions are much weaker.

STRATEGIES FOR ADDRESSING NONRESPONSE

There are several strategies to deal with nonresponse and estimate response bias. These include:

  1. Throw the data away. If the researcher feels that the data is invalid, this may be a reasonable solution. In practice, I have never heard of data being thrown away. Instead, the data is used to the best extent possible.

  2. Generalize to the respondents only. This strategy avoids making erroneous inferences about the larger population.

  3. Assume there is no response bias and generalize to the population. If you know the population well and examination of the data reveals no obvious abnormalities this may be reasonable. This strategy is analogous to judgement sampling.

  4. Call back nonrespondents. Finding out why people did not respond can help determine the extent of response bias. If none is apparent then generalization to the population can be justified. One disadvantage is that only nonrespondents who can be contacted are examined. Nonrespondents who cannot be contacted can continue to bias the sample estimates.

  5. Compare data in hand on respondents and nonrespondents. If data, e.g., sex, age, race, is available, the composition of respondents can be compared with that of nonrespondents to see if there are any differences. The presence of differences indicates response bias and that caution is necessary in making inferences.

  6. Compare characteristics of early respondents with late respondents (e.g., first and third waves in mail surveys). Differences indicate that non-respondents are likely to be different also. In Extension programs, early respondents are more likely to be adopters of practice change than late respondents or non-respondents. This is because early adopters/respondents tend to have higher levels of involvement in Extension programs.

  7. Increase mailings or contact efforts. Although this is probably the most costly and time-consuming, obtaining the highest response rate possible is the best way to reduce response bias.

Note that increasing the sample size does not remove potential response bias. Larger numbers of responses from large sample sizes can provide more information about the population but only if a high response rate is obtained. In my opinion, increasing the sample size to improve the precision of estimates is a poor trade-off with efforts to maximize the response rate and reduce response bias.

CONCLUDING REMARKS

A number of procedures and considerations in addressing nonresponse have been discussed above. If the data to be used for an evaluation or a needs assessment is to be both valid and appropriate, then two requirements must be met. First, the sampling process, including procedures for addressing nonresponse, must be clearly articulated. This will allow your colleagues to assess the potential of the sampling procedures to perform as intended. Second, the procedures must be implemented as specified. A poorly implemented sampling design can result in data that is not representative of the population at large. This can lead to inferences that are wrong. A well planned and implemented sampling design can yield results that are accurate and useful.

REFERENCES

Israel, Glenn D. 1992. Sampling The Evidence Of Extension Program Impact. Program Evaluation and Organizational Development, IFAS, University of Florida. PEOD-5. October.

Israel, Glenn D. 1992. Determining Sample Size. Program Evaluation and Organizational Development, IFAS, University of Florida. PEOD-6. November.

Kish, Leslie. 1965. Survey Sampling. New York: John Wiley and Sons, Inc.

Miaoulis, George, and R. D. Michener. 1976. An Introduction to Sampling. Dubuque, Iowa: Kendall/Hunt Publishing Company.

Footnotes

1.

This document is PEOD9, one of a series of the Agricultural Education and Communication Department, Florida Cooperative Extension Service, Institute of Food and Agricultural Sciences, University of Florida. Original publication date November 1992. Revised April 2009. Reviewed June 2012. Visit the EDIS website at http://edis.ifas.ufl.edu.

2.

Associate Professor, Department of Agricultural Education and Communication, and Extension Specialist, Program Evaluation and Organizational Development, Institute of Food and Agricultural Sciences (IFAS), University of Florida, Gainesville, FL 32611.


The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex, sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other UF/IFAS Extension publications, contact your county's UF/IFAS Extension office.

U.S. Department of Agriculture, UF/IFAS Extension Service, University of Florida, IFAS, Florida A & M University Cooperative Extension Program, and Boards of County Commissioners Cooperating. Nick T. Place, dean for UF/IFAS Extension.