Introduction
This publication provides an overview of group-administered surveys as part of the Savvy Survey Series. Group-administered surveys are commonly used at the end of an educational event to collect information from participants about the quality of the program and initial outcomes, such as perceived learning and aspirations for behavior change. These surveys are often called "end of the meeting" evaluations. Although these surveys are useful for formative evaluations and documenting short-term outcomes, follow-up surveys are more appropriate for assessing long-term outcomes and impacts (Israel et al., 2009).
This publication discusses the development and implementation of group-administered surveys, as well as how to prepare an introductory script, train survey administrators, and manage the survey process.
Developing the Group-Administered Questionnaire
The construction of every survey questionnaire requires careful attention to detail. However, for questionnaires that will be administered in a group, special attention must be given to how respondents will understand and answer the questions differently depending on whether they are asked questions orally, whether the questions are projected onto a screen, or whether a printed form is handed out to participants.
A well-developed questionnaire is the foundation for collecting useful data from group members. The group-administered questionnaire is most commonly created using a paper format, similar to that used in a mail survey. One advantage of a paper instrument is that a combination of close-ended, Likert-type questions and open-ended questions can be used (see Savvy Survey Series publications #6a–6c). The questionnaire can also be created using survey software, such as Qualtrics. The administrator might hand a tablet device (e.g., an iPad) to the participants to complete the survey while the interviewer facilitates the process or troubleshoots problems. Another option might be to provide a link or a scannable QR code for participants to use in order to access the survey on a cell phone.
For low-literacy adults and children, the questions can be asked orally and visual aids can be incorporated as appropriate. When relying on an interviewer to verbally ask the questions, whether by stating the options for answering or by projecting the response options on a screen, the survey administrator must carefully word his or her instructions and responses to questions so as not to knowingly (or unknowingly) interject bias through his or her behavior, word choices, or tone (Loosvoldt, 2008).
Use of Audience Response Systems
As an alternative to a paper questionnaire, various technologies might be used to record responses to questions that are projected onto a screen or presented orally. Audience response systems (e.g., clickers, "TurningPoint," e-poll sites such as PollEverywhere, or embedded polling tools in online meeting platforms such as “Zoom”) enable the presenter to rapidly collect and analyze the responses of participants (Bruff, n.d.). With an audience response system in which the group leader administers the questions, he or she can manage the pace of the survey to encourage thoughtful responses. This technology also combines the data collection and data entry steps. However, since these technologies can change the survey environment, their use needs to be thoughtfully considered before integrating them into the data collection process, as well as the entire educational event (Bruff, n.d.).
Survey Approaches
Audience response systems can be used for surveys that collect information for both formative and summative evaluations (Bruff, n.d.). In summative surveys, where there is a desire to collect individual responses, it is best to hide responses from the group and, instead, only project individual items for consideration, in order to limit influence of one respondent on any other respondent. However, if the results of the questionnaire/survey are to be used formatively, projecting responses and allowing time and space for participant discussion can also be a good practice. It is, however, a good idea to have a back-up plan for situations where the technology does not work.
Participant Considerations
Participants may not be familiar with using this type of technology. Time would need to be set aside before the start of the survey for instruction on proper use. Failure to do so may compromise the confidence in the responses from both the perspective of the respondent and the presenter (Bunz, 2005). Participants have the ability to maintain anonymity when using this type of response method, which may encourage increased cooperation within the group setting. However, when responses are displayed for participants to view, the potential for respondents to be focused on making comparisons between their response and others, rather than solely on their individual responses, increases.
Question Formats
Audience response systems are often limited to multiple choice or 5-point Likert-type scale questions (Bunz, 2005).
Preparation for Group Administration
There are several steps for getting ready to administer a print-based evaluation or questionnaire to a group.
- Prepare an introductory script for administering the survey to the group. A short script typically includes information about the purpose of the survey and what participants are being asked to do. Note: if the survey has been reviewed and approved by the Institutional Review Board (IRB, see Savvy Survey Series publication #9), an information sheet can be provided to participants, which may include additional information, such as what will be done with the collected data, rights of the participants, and contact information of the survey director. A sample information sheet is included in Appendix A.
- Make enough copies of the questionnaire to provide one to each attendee or secure a set of clickers for the Audience Response System.
- Obtain business envelopes for completed surveys and a large envelope to hold the set of completed surveys. As indicated below, these supplies are needed to collect answers and keep them confidential.
- In order to build trust in the survey process, recruit a staff person, another Extension agent, or a state specialist to assist with the data collection, entry, and analysis. This person must be capable of appropriately answering participant questions and willing to handle the data and ensure the confidentiality of the responses. Establishing the confidentiality of the survey will encourage participants to provide more accurate and complete information. If another person is not available to help, then the questionnaire should include instructions to exclude identifying information such as name, phone number, email address, or similar data.
For situations in which the same survey will be administered to multiple groups across several sites and with different group interviewers, a training session should be conducted to orient the interviewers in the same way that interviewers for face-to-face surveys are trained (see Savvy Survey Series #10). The session should include a discussion of the purpose of the survey and expectations for interviewers, as well as a practice group interview. The resulting data will be more directly comparable if the group interviewers all receive the same training on how to properly conduct the survey.
Implementing the Group-Administered Survey
Make sure to create an environment that encourages thoughtful responses and reduces "test-taking" behaviors (Dillman, 2007; Groves et al., 2009). This may include providing ample distance between the respondents to establish a sense of privacy, reducing the social presence of administrators, and minimizing environmental distractions (Groves et al., 2009).
The following procedures are designed to increase cooperation in completing an evaluation survey in a group setting, as well as to protect the confidentiality of attendees' responses. Use of these procedures is recommended regardless of whether paper-based or audience response systems are utilized.
- Use a scripted introduction. An example of a script for evaluating an in-service training program is shown in Table 1. The script is designed to provide a consistent stimulus to attendees at each educational event. The introductory text is designed to encourage thoughtful responses and reduce undesirable "test-taking" behaviors (Dillman, 2007).
The purpose of the statement, "Please wait for further instructions," is designed to prevent people from rushing through the survey and leaving, which can be disruptive for other attendees who are still completing the survey. Although some people may choose to leave (we cannot stop them from doing so), setting the explicit expectation of waiting creates a powerful normative pressure to conform. Consequently, most participants cooperate with the request.
Note: The statement, "Your answers will be kept confidential," can be replaced with "Your answers are anonymous" if the questionnaire design, survey procedures, and technology acts to protect anonymity.
2. Distribute survey materials. For paper-based questionnaires, make sure that each participant receives a questionnaire and an envelope. For ARS-based surveys, make sure that each participant receives a clicker and instructions for use.
3. Supervise the data collection. For paper-based questionnaires, wait until everyone completes the questionnaire before collecting the responses. Identify a staff person or participant to collect the envelopes containing the completed surveys and have that individual send the surveys to the person responsible for data entry and analysis.
For ARS-based surveys, wait until everyone responds to the item on the screen before moving on to the next item. Remember, this setting allows the group leader to manage the pace of the survey to meet the needs of the participants (e.g., low-literacy participants) while encouraging thoughtful responses.
4. Debrief participants. Thank respondents for completing the survey, answer any questions, and announce follow-up educational events that may be of interest to participants.
5. Obtain results. After the forms are sent for data entry and analysis, contact the responsible person to obtain tabulated results. The typical report should include frequency counts and percentages for each item, mean score for each index, and a compilation of participant comments. Likewise, the data for an ARS-based survey can be saved and compiled into a summary report.
These procedures have proven effective in conducting evaluations of in-service training for Extension agents and workshops for clients.
In Summary
This publication has provided a brief overview of how to develop and conduct group-administered surveys. Careful attention is needed for developing the questionnaire and supporting materials, as well as for training persons who will administer the survey in order to obtain credible data. In the right situation, such as in-service trainings or workshops, these surveys can be a valuable tool for Extension agents and specialists.
References
Bruff, D. (n.d.). Clickers and classroom dynamics. Washington, DC: National Education Association.
Bunz, U. (2005). "Using Scantron versus an audience response system for survey research: Does methodology matter when measuring computer-mediated communication competence?" Computers in Human Behavior, 21, 343–359. https://doi.org/10.1016/j.chb.2004.02.009
Dillman, D. A. (2007). Mail and Internet surveys: The tailored design method—2007 Update with new Internet, visual, and mixed-mode guide. Hoboken, NJ: Wiley.
Groves, R. M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, & R. Tourangeau. (2009). Survey methodology (2nd ed.). Hoboken, NJ: Wiley.
Israel, G. D., D. Diehl, & S. Galindo-Gonzalez. (2009). Evaluation situations, stakeholders, & strategies. WC090. Gainesville: University of Florida Institute of Food and Agricultural Sciences. https://edis.ifas.ufl.edu/publication/WC090.
Loosveldt, G. (2008). "Face-to-face interviews." In D. de Leeuw, J. J. Hox, and D. D. Dillman (Eds.), International Handbook of Survey Methodology (pp. 201–220). New York, NY: Lawrence Erlbaum Associates.
Appendix A: Sample Information Sheet
