Introduction
Extension professionals work with diverse clients and their ability to work with individuals and groups to solve a common goal is an important skill. The increasingly complex and interconnected social, political, and environmental environments in which Extension professionals work dictate that they must be facilitators of processes where community members work together to make a difference (Morse et al., 2006). This publication is intended for Extension professionals working in any discipline.
The ability to achieve a group consensus may be critical during specific activities throughout an Extension professional's career. For example, it may be important to identify an advisory committee's highest priorities or a group of stakeholders' most important programmatic needs. The Delphi method has been recognized as a suitable alternative to interviews and formal meetings in certain circumstances (Geist, 2002). This article provides an overview of the Delphi method and suggestions for using this technique to support Extension programming.
The Delphi technique is a research-based approach to achieving group consensus and has been described as a "method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem" (Linston & Turoff, 2002, p. 3). Delphi research has been used for numerous activities, including:
- Establishing programmatic objectives
- Planning for budget modifications
- Identifying essential professional competencies within a specific context
- Supporting various elements of curriculum development
- Collecting historical data
- Exploring perceptions related to potential policy changes
- Understanding personal motivations and values
- Investigating urban and regional planning possibilities
- Futuring, or exploring potential future options to determine what is likely to happen, what may change, and what effects can be anticipated (Conner, Roberts, & Harder, 2013; Harder, Place, & Scheer, 2010; Linstone & Turoff, 2002; Martin & Frick, 1998).
Suggested Applications
The following list includes just a few suggestions out of a large range of possibilities for applying the Delphi method to Extension programming. The Delphi method can be used to:
- Determine programmatic priorities among an Extension advisory committee
- Engage stakeholders in a local industry strengths and weaknesses identification activity
- Set priorities among cohort members of a long-term Extension class
- Establish Extension programmatic objectives
- Create a standardized curriculum
- Collect unpublished, historical data of a county Extension program
- Gauge the barriers and motivators of a target audience in relation to a specific behavior change
- Engage stakeholders in evaluating an Extension program.
Overview of the Delphi Method for Reaching Group Consensus
The Delphi technique is a means of reaching a group consensus through multiple rounds of anonymous feedback, or iterations (Geist, 2010; Martin & Frick, 1998).
This approach allows for data collection using a structured process. A questionnaire is used to gain feedback, yet there is no definitive format for the questionnaire itself (Martin & Frick, 1998). The Delphi process is useful when information about a specific problem is not available (Brodeur et al., 2011). The participants, the researchers, or some combination of the two may generate the questionnaire items. A significant characteristic of this technique is participant anonymity (Geist, 2010). Another benefit to this approach is that it reduces the possibility of certain individuals influencing group decision-making more than others (Geist, 2010; Linstone & Turoff, 2002). Additionally, the Delphi process is beneficial because it allows for participation without scheduling or geographic restrictions, meaning that all stakeholders may be able to participate (Geist, 2010). Figure 1 presents an example of the Delphi process applied to priority-setting for a water-conservation curriculum for landscape professionals.
The Delphi approach can be especially beneficial to an Extension program when one of the following circumstances exists:
- Disagreements between individuals are likely to be severe and unmanageable
- The question of interest cannot be answered with quantitative techniques, but may benefit from collective opinions
- The experts who need to be engaged in problem-solving activities either lack a functional communicative relationship or have a dysfunctional one
- There are too many people who need to be engaged that can be coordinated in live group meetings
- Resources prevent the group from engaging in person in one location
Adapted from Linstone and Turoff (2002).
Creating the Panel
The first major task in a Delphi study is to create an expert panel of individuals (Linstone & Turoff, 2002). The panel can be composed of any combination of stakeholders, subject experts, and facilitators (Linstone & Turoff, 2002).
There are several recommendations for the number of panelists that should be used, and it is also important to use an adequate number of panelists who are well-qualified to provide input on the topic.
One recommendation is to engage at least 13 expert panelists in order to achieve a reliable (reliability of 0.9) sample (Dalkey, 1969; Dalkey, 2002). A panel with 10–15 similar panelists has been recommended as the ideal number (Delbecq et al., 1975). While the number of panelists can vary greatly and is ultimately dependent on the research design, Delbecq et al. (1975) caution against using too many panelists, in that data analysis may be daunting.
Because the Delphi technique relies on engaging people who are knowledgeable about a specific topic, purposive sampling is used. Purposive sampling identifies the group members from whom the practitioner can learn the most and is based on a set of specific criteria (Dooley, 2007). Usually the lead researcher defines the qualifications of an expert in terms of the topic at hand and seeks out individuals who meet the qualifications. Sometimes, snowball sampling is used to increase the number of expert group members. Snowball sampling engages the initial group members in identifying additional members with similar qualifications (Dooley, 2007).
The Method
The Delphi process (Figure 2) takes place through a number of rounds of surveys that elicit panelists' opinions about the topic at hand (Geist, 2010). Delphi questionnaires are designed based on the problem at hand and emerge based on group input. Each round is based upon the results of the round before it. The surveys may be paper-based or electronic (Geist, 2010). The first round is referred to as the generative round, during which members of the group are encouraged to explore the topic through a prompt that describes the question or issue (Geist, 2010; Linstone & Turoff, 2002). The members provide qualitative input based on the prompt. In some cases, the first round may engage group members in responding to a list of existing information, such as previously published professional competencies or industry needs. The researcher analyzes the input and translates it into a survey for the following round. Subsequent iterations after the generative round often are quantitative. One common survey design used in these subsequent rounds employs a Likert scale placed next to the individual responses collected in the generative round. The Likert scale may consist of five to seven points and ask for group members to indicate their level of agreement, from strongly disagree to strongly agree. Many other survey designs can be used for this stage depending on the focus of the study.
The total number of rounds varies, with two rounds being the minimum (Linstone & Turoff, 2002; Martin & Frick, 1998) and four commonly considered the appropriate number (Geist, 2010; Linstone & Turoff, 2002). However, two to three rounds are generally considered adequate to achieve consensus (Delbecq et al., 1975). As with other survey methodologies, it is advantageous to provide group members with a survey prenotice, original questionnaires, replacement questionnaires, thank you notes, and possibly a final replacement questionnaire using a different mode of delivery (e.g., mail for an electronic survey or a different type of email for electronic studies) to ensure adequate participation (Dillman et al. 2009).
Data Analysis
Data analysis for a Delphi process depends on the type of data being collected and may consist of both qualitative and quantitative analysis. Generally, data collected during the generative round will be qualitative, open-ended responses. Usually, responses that are considered to have the same meaning are combined into a single response. In many cases, the researcher will want to create categories of data to make them easier to manage and easier for panelists to respond to in subsequent rounds. While several options exist, a common method for analyzing qualitative data is the constant comparative method (Glaser & Strauss, 1967). This analysis requires multiple steps. First, the data are assessed line by line and coded with temporary names, then recoded until categories become well-defined. Second the individual categories are examined to create meaningful relationships with other categories and subcategories. Finally, the researcher may use precise criteria to develop themes or core categories to describe the data (Glaser & Strauss, 1967).
The data collected in subsequent rounds are frequently quantitative and often consist of the level of agreement with items that emerged from the first round. Likert scales are often used and may be labeled from "strongly disagree" to "strongly agree." The data that emerge from these rounds are analyzed based on a definition of consensus that is defined by the researcher prior to initiating the Delphi process. While the definition may vary from practitioner to practitioner, a common definition of consensus is where two-thirds of group members indicate that they agree or strongly agree with an item (Boyd, 2003; Conner et al., 2013; Harder et al., 2010; Shinn et al., 2009).
Previous Uses of the Delphi Technique in Extension and Agricultural Education
A review of the literature reveals numerous applications of the Delphi technique in Extension and education. The following are some applications of note.
- The South Carolina Master Gardening program used the Delphi process to identify a uniform statewide curriculum (Callahan, Dobbins, King, Paige, & Wolak, 2010). This study used 14 statewide master gardener coordinators to identify the most appropriate topics and course requirements to be covered statewide.
- A national study was conducted to determine the skills that entry-level extension professionals would need in 2015 (Harder, Place, & Scheer, 2010). In this study, 12 experts identified 19 competencies that were necessary for Extension professionals.
- A panel of international agricultural development experts was used to identify the experiences and competencies that agricultural development professionals should have. This study identified 7 experiences and 26 competencies that new international development professionals should have when entering the field (Conner, Roberts, & Harder, 2013).
- Program participants from several Western Extension Leadership Development (WELD) and National Extension Leadership Development (NELD North Central) classes were engaged in a Delphi study to identify workplace issues that will be important in obtaining and retaining talented Extension educators (Kroth & Peutz, 2011).
- In 2009 a national Delphi study was conducted to identify the competencies that would be important in horticultural education (Basinger et al., 2009). The identified competencies were used to make recommendations for developing horticulture curricula.
- The Delphi method was used to assess the current situation, strengths, and weaknesses of horticultural cooperatives in Spain (Campos-Climent et al., 2012).
- Researchers at the University of Florida used a Delphi study to learn about new county Extension faculty's professional development needs and job satisfaction (Brodeur et al., 2011). The expert panelists used for the Delphi study included 92 Extension agents who were hired over a three-year period.
Guidelines
The following guidelines should be considered when employing the Delphi technique.
- Make sure that you have the right set of experts who are well qualified to provide input on the topic of interest.
- Use thoughtful consideration to determine if the Delphi technique is the appropriate method for answering the question of interest. The Delphi technique may be especially appropriate when little information exists about the topic, when experts in the area of interest are likely to readily engage in the process, and when anonymity is desired among individuals.
- Ensure that adequate time is allotted to complete the process. Depending on the design of the process, a Delphi study can take months to complete.
- Ensure that you have enough people who are willing to participate in the full process through completion.
- Take time to analyze data. Because each round is drafted upon the data collected in the previous round, a mistake in analyzing data can lead to inaccuracies and false understandings about the topic of interest.
Conclusion
An Extension professional's ability to achieve group consensus may be important to specific activities throughout his or her career. This article discusses the Delphi method as a powerful research-based approach to achieving group consensus. The benefits to using this approach include allowing individuals to have equal and anonymous input and accommodating people in different locations and on different schedules. Today's Extension environment is "one of activating and convening stakeholders and facilitating problem-solving processes that address public issues collaboratively" (Morse et al., 2006, Abstract, para. 1). While numerous potential applications of this tool exist, the Delphi process may be beneficial to operating in today's complex environment and identifying programmatic priorities with an advisory committee, setting learning priorities among Extension clients, and identifying strengths and weaknesses perceived by industry leaders.
References
Basinger, A. R., McKenney, C. B., & Auld, D. (2009). Competencies for a United States horticulture undergraduate major: A national Delphi study. HortTechnology, 19(2), 452-458.
Brodeur, C. W., Higgins, C., Galindo-Gonzalez, S., Craig, D. D., & Haile, T. (2011). Designing a competency-based new county extension personnel training program: A novel approach. Journal of Extension, 49(3). Retrieved from http://www.joe.org/joe/2011june/a2.php
Callahan, B.J., Dobbins, T.R., King, D.R., Paige, W.D., & Wolak, F.J. (2010). Establishing statewide uniformity within the South Carolina master gardener program. Journal of the NACAA, 3(1). Retrieved from https://www.nacaa.com/file.ashx?id=298ad574-8da5-4eb9-8b21-b563be6bc7bc
Campos-Climent, V., Apetrei, A., & Chaves-Avila, R. (2012). Delphi method applied to horticultural cooperatives. Management Decision, 50(7), 1266-1284.
Conner, N. W., Roberts, T. G., & Harder, A. (2013). Competencies and experiences needed by entry level international agricultural development practitioners. Journal of International Agricultural and Extension Education, 20(1), 19 – 32. Retrieved from https://www.aiaee.org/attachments/article/1503/Conner_Roberts_20102.pdf
Dalkey, N. C. (2002). Toward a theory of group estimation. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications [Electronic version] (pp. 231-256). Newark, NJ: New Jersey Institute of Technology.
Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinion. Santa Monica, CA: The Rand Corporation.
Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning : A guide to nominal group and Delphi processes. Glenview, Ill.: Scott, Foresman.
Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design Method. (2nd ed.), Hoboken, NJ: John Wiley Co.
Dooley, K. E. (2007). Viewing agricultural education research through a qualitative lens. Journal of Agricultural Education, 48(4), 32-42.
Geist, M. R. (2010). Using the Delphi method to engage stakeholders: A comparison of two studies. Evaluation and Program Planning, 33(2), 147-154.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine De Gruyter.
Harder, A., Place, N. T., & Scheer, S. D. (2010). Towards a competency-based extension education curriculum: A Delphi study. Journal of Agricultural Education, 51(3), 44–52. doi: 10.5032/jae.2010.03044
Hsu, C., & Sandford, B. A. (2007). The Delphi Technique: Making sense of consensus. Practical Assessment, Research & Evaluation. 12(10). Retrieved from https://doi.org/10.7275/pdz9-th90
Kroth, M., & Peutz, J. (2011). Workplace issues in extension—a Delphi study of extension educators. Journal of Extension, 49(1). Retrieved from https://archives.joe.org/joe/2011february/rb1.php
Linstone, H. A., & Turoff, M. (Eds.). (2002). The Delphi method: Techniques and applications [Electronic version]. Newark, NJ: New Jersey Institute of Technology.
Martin, A. G., & Frick, M. J. (1998). The Delphi technique: an informal history of its use in agricultural education research since 1984. Journal of Agricultural Education, 39(1), 73-79. doi: 10.5032/jae.1998.01073
Morse, R. S., Brown, P. W., & Warning, J. E. (2006). Catalytic leadership: Reconsidering the nature of Extension's leadership role. Journal of Extension, 44(2). Retrieved from https://archives.joe.org/joe/2006april/a9.php
Schindler, K. H. (2013). Delphi Technique. Retrieved from http://www.extension.org/pages/61541/delphi-technique#.VA9MgRazNKY
Shinn, G. C., Wingenbach, G. J., Briers, G. E., Lindner, J. R., & Baker, M. (2009). Forecasting doctoral–level content in international agricultural and extension education–2010: Viewpoint of fifteen engaged scholars. Journal of International Agricultural and Extension Education, 16(1), 57–71.
