IFAS ExtensionUniversity of Florida
Login

Conducting the Needs Assessment #10: The Delphi Technique

Laura Warner and Amy Harder

Abstract

Previous publications in the Conducting the Needs Assessment series outlined why, how, and when Extension educators and other service providers should use needs assessments in their programs. The four preceding publications in the series provide Extension educators and other service providers with specific techniques that can be used in needs assessments, and the current publication provides an overview of using the Delphi technique to conduct a needs assessment.

For a complete list of the publications in this series, refer to the overview of the Conducting a Needs Assessment series in the Appendix.

Introduction

Conducting a needs assessment is one of the most important components of the program planning process. Needs assessments help to identify gaps between the current and desired situation and should be used to make decisions about programmatic priorities and resource allocation (Benge et al., 2019). Extension professionals can use a variety of systematic qualitative and quantitative techniques to gather information about local needs (Benge et al., 2019). The Delphi technique is a group process characterized by structured communications through which experts on a given topic form a panel and provide multiple iterations of feedback until consensus is reached (Linstone & Turoff, 2002).

Extension Needs Assessment Situations Best Suited for Delphi

The Delphi technique can be used for any situation in which reaching consensus with a group is desirable, although it is often associated with forecasting for the future (Witkin & Altschuld, 1995). Extension professionals will find it works best for situations with the following characteristics:

  • Delphi panelists are recognized experts in the topic (Ludwig & Starr, 2005) and have credibility with the broader audience whose needs are being assessed (Goldstein as cited in Linstone & Turoff, 2002)
  • There are at least 13 people who can serve as panelists; the reliability of panel results is over .90 when 13 panelists complete the process (Dalkey, 2002).
  • There is enough time available to use the Delphi technique; a typical Delphi panel will be asked to provide at least three rounds of feedback, which makes this a time-intensive approach to generating consensus (Witkin & Altschuld, 1995).

Selecting Experts to Participate in a Delphi Panel

Delphi panels are more accurate when they consist of experts (Powell, 2003; Rowe & Wright, 1999). Experts should be individuals with "specialized knowledge about the topic" (Witkin & Altschuld, 1995, p. 195). However, the iterative nature of the Delphi technique means Extension professionals should also take into consideration the likelihood that a potential panelist will be able and willing to dedicate the necessary amount of time to make the process a success. This is so critical that Witkin and Altschuld (1995) state: "Respondent cooperation is absolutely essential for obtaining complete and interpretable data" (p. 195). It may be tempting to downplay the level of commitment needed when recruiting potential panelists, but Extension professionals should be transparent about the nature of the process and let potential panelists make educated decisions about engagement. Helpful tips on encouraging individuals to cooperate in any kind of survey can be found in detail in Dillman et al. (2014).

A few common types of Extension audiences lend themselves well to the Delphi technique. Dedicated volunteers who are deeply invested in a program may be good candidates for participation in a Delphi panel, such as Master Gardeners or long-time 4-H club leaders. Members of an Extension advisory council may also have suitable characteristics for engagement in the Delphi technique. Polush et al. (2016) successfully conducted a Delphi with members of the Practical Farmers of Iowa organization, an indication that the Delphi technique may work well with any organization with committed members. Finally, the Delphi technique has often been used internally with Extension professionals serving as panelists to find solutions to challenges facing programs or the organization (e.g., Boyd, 2004; Kroth & Peutz, 2011), such as identifying barriers to evaluating statewide program impact.

Identifying Needs through Delphi and Defining Consensus

First round. Once the needs to be assessed are specified and appropriate panelists have been recruited, the next step is typically an initial open-ended questionnaire designed to gather input on the topic of interest. Delphi technique processes are conducted either electronically or in writing, and many have only one or two questions (McCaslin & Tibezinda, n.d.). In the context of a needs assessment, this open-ended question may ask what an audience must achieve to reach the desired state, or what criteria would define the desired state. Some examples of this question might include:

  • "What is the most important issue facing teachers in your county?"
  • "What services do underserved residents in our county require to achieve food security?"
  • "What competencies do Extension professionals at your university need to be successful in urban counties?"
  • "What skills do students graduating from your high school need to be effective leaders in their community?"

Expert panel members are encouraged to provide as many responses as they can. An Extension professional or other service provider then compares and combines the responses into a list without duplications to build the subsequent questionnaire. To do this, the Extension professional often works with a list of responses, which will be reviewed many times. It can be helpful to color-code items with common themes (such as educational needs, resource needs, etc.) and then edit each theme individually. The Extension professional will examine individual responses looking for those that have the same meaning even though they may be worded differently. For example, "an understanding of local politics" and "familiarity with the city government" would most likely be combined as one idea. During this editing process, the Extension profession would also examine responses for those that represent more than one idea. For example, "knowledge of regulations influencing urban food producers and crops that thrive in our local soil" would likely be split into two ideas: "knowledge of regulations influencing urban food producers" and "knowledge of crops that thrive in our local soil." It is helpful and important to have panel members or others familiar with the context review the edited list of responses before moving to the second round. It is a good idea to keep a record of how responses were combined or separated so others can follow the logic used.

Second round. In the second questionnaire, the panel responds to the list they collectively generated in the first round. They are typically asked to indicate their level of agreement or the level of importance they associate with the items on the initial list (or, alternatively, they may be asked to indicate the level of importance they associate with each item; Mayfield et al., 2005). For example, "please indicate your level of agreement or disagreement with each of the following being among the most important issues teachers face in your county." The Extension professional or other service provider then analyzes the data so that only items that achieve consensus are retained. The definition of consensus should be set before engaging in the Delphi process (a priori) and is often a value of 2/3 of panelists indicating "agree" or "strongly agree" for an item.

Third and additional rounds. In the third round, the panel is presented with the list of items that achieved consensus during the second round and is asked to again indicate their level of agreement with the items. There is no absolute number of rounds, although most Delphi processes will use a minimum of three (Witkin & Altschuld, 1995). The number of iterations may depend on the Extension professional or service provider, the outcomes of the process, and the resources available.

The list of items that achieve consensus following the final round are then subject to further consideration. Needs can be prioritized using the expert panel responses from the final round of the Delphi process. For example, needs may be ranked by the percentage of the panel indicating they highly agreed with the specific item. In some cases, the final list of items could be prioritized by a voting process among the expert panel. In other cases, it may be advantageous to use the items to measure the gap between the desired and current states. In some situations, the Extension professional may want to understand the target audience's current state (for example, their perceived skill or knowledge level) and compare it with how important the item is to their work or life. A Borich model can be used to conduct this type of comparison using items generated by a Delphi study (see Conducting the Needs Assessment #8: The Borich Model in this series for more information).

Consider exploring examples of the variety of applications using Delphi techniques such as the following:

Benefits and Limitations to Using the Delphi Technique in Needs Assessments

Benefits of using the Delphi technique to identify needs include:

  • the ability to identify needs without influence from outside groups,
  • avoidance of travel and meeting costs,
  • not being limited by geographic area or time zone,
  • technique driven by participants,
  • participant anonymity,
  • flexible format, and
  • avoidance of conflict among experts who may disagree (Donaldson & Franck, n.d.; Mayfield et al., 2005; McCaslin & Tibezinda, n.d.; Warner, 2014).

Limitations of using Delphi to identify needs include:

  • requiring participants to have strong reading and writing skills, which could prevent some individuals from participating,
  • requiring a high level of motivation from the experts to complete several rounds, and
  • needing substantial time to complete the process (Mayfield et al., 2005; McCaslin & Tibezinda, n.d.; Warner, 2014).

For more information about the process of using the Delphi technique, see AEC521, Using the Delphi Technique to Achieve Consensus: A Tool for Guiding Extension Programs.

Conclusion

The Delphi technique is a flexible way to identify needs using input from experts. This process has many benefits that may make Delphi desirable to use in a broad range of needs assessment situations when compared to other group techniques.

References

Benge, M., Harder, A., & Warner, L. (2019). Conducting the needs assessment #1: Introduction. AEC677. Gainesville: University of Florida Institute of Food and Agricultural Sciences. https://edis.ifas.ufl.edu/publication/wc340

Boyd, B. L. (2004). Extension agents as administrators of volunteers: Competencies needed for the future. Journal of Extension, 42(2), Article 2FEA4. https://archives.joe.org/joe/2004april/a4.php

Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinion. Santa Monica, CA: The Rand Corporation.

Dalkey, N. C. (2002). Toward a theory of group estimation. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications [Electronic version] (pp. 231–256). Newark, NJ: New Jersey Institute of Technology.

Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Glenview, IL: Scott, Foresman.

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The Tailored Design Method (4th ed.). Hoboken, NJ: John Wiley & Sons, Inc.

Donaldson, J. L., & Franck, K. L. (n.d.). Needs Assessment Guidebook for Extension Professionals. PB1839. Knoxville: UT Extension. https://extension.tennessee.edu/publications/Documents/PB1839.pdf

Harder, A., Place, N. T., & Scheer, S. D. (2010). Towards a competency-based Extension education curriculum: A Delphi study. Journal of Agricultural Education, 51(3), 44–52. https://www.jae-online.org/attachments/article/84/Vol%2051%20No%203%20pg%2044%20-%20Harder.pdf

Kroth, M., & Peutz, J. (2011). Workplace issues in Extension – A Delphi study of extension educators. Journal of Extension, 49(1), Article v49-1rb1. https://archives.joe.org/joe/2011february/rb1.php

Linstone, H. A., & Turoff, M. (Eds.). (2002). The Delphi method: Techniques and applications [Electronic version]. Newark, NJ: New Jersey Institute of Technology.

Ludwig, L., & Starr, S. (2005). Library as place: Results of a Delphi study. Journal of the Medical Library Association [Electronic version], 93(3), 315–326.

Mayfield, C. A., Wingenbach, G. J., & Chalmers, D. R. (2005). Assessing stakeholder needs: Delphi meets the internet. Journal of Extension, 43(3). https://archives.joe.org/joe/2005june/iw1.php

McCaslin, N. L., & Tibezinda, J. P. (n.d.). Chapter 5 – Assessing target group needs. In B. E. Swanson, R. P. Bentz, & A. J. Sofranko (eds.), Improving agricultural extension. A reference manual. Rome, Italy: Food and Agriculture Organization of the United Nations. https://www.fao.org/3/w5830e/w5830e07.htm

Perry, B., Shaw, A., Johnsen, E., Enderton, A., Strohbehn, C. H., & Naeve, L. (2019). Assessment of midwest growers' needs for compliance with the food safety modernization act produce safety rule. Food Protection Trends, 39(3), 212–217. https://www.ncrfsma.org/files/news/files/fpt_may_june_article.pdf

Polush, E. Y., Grudens-Shuck, N., Exner, D. N., & Karp, R. (2016). Delphi survey of needs for on-farm research: Forecasting changes in a farm organization. Journal of Extension, 54(3), Article v54-3a3. https://archives.joe.org/joe/2016june/a3.php

Powell, C. (2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376–382. https://doi.org/10.1046/j.1365-2648.2003.02537.x

Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15, 353–375. https://doi.org/10.1016/S0169-2070(99)00018-7

Warner, L. A. (2014). Using the Delphi technique to achieve consensus: a tool for guiding Extension programs. AEC521. Gainesville: University of Florida Institute of Food and Agricultural Sciences. https://edis.ifas.ufl.edu/publication/wc183

Witkin, B. R., & Altschuld, J. W. (1995). Planning and conducting needs assessments: A practical guide. Thousand Oaks, CA: SAGE Publications, Inc.

Appendix: Conducting the Needs Assessment Series Overview

Conducting the Needs Assessment #1: Introduction

General summary of needs assessments, including what a needs assessment is, the different phases, and tools to conduct a needs assessment.

Conducting the Needs Assessment #2: Using Needs Assessments in Extension Programming

Overview of using needs assessments as part of the Extension program planning process.

Conducting the Needs Assessment #3: Motivations, Barriers and Objections

Information about the motivations, barriers, and objections to conducting needs assessments for Extension professionals and service providers.

Conducting the Needs Assessment #4: Audience Motivations, Barriers, and Objections

Information about the motivations, barriers, and objections that clientele and communities may have for participating or buying-in to a needs assessment.

Conducting the Needs Assessment #5: Phase 1—Pre-assessment

Introduction to the Pre-assessment phase of conducting a needs assessment, including defining the purpose, management, identifying existing information, and determining the appropriate methods.

Conducting the Needs Assessment #6: Phase 2—Assessment

Introduction to the Assessment phase of conducting a needs assessment, including gathering and analyzing all data.

Conducting the Needs Assessment #7: Phase 3—Post-assessment

Introduction to the Post-assessment phase of conducting a needs assessment, including setting priorities, considering solutions, communicating results, and evaluating the needs assessment.

Conducting the Needs Assessment #8: The Borich Model

Overview of using the Borich Model to conduct a needs assessment.

Conducting the Needs Assessment #9: The Nominal Group Technique

Overview of using the Nominal Group Technique to conduct a needs assessment.

Conducting the Needs Assessment #10: The Delphi Technique

Overview of using the Delphi Technique to conduct a needs assessment.

Conducting the Needs Assessment #11: The Causal Analysis Technique

Overview of using the Causal Analysis Technique to conduct a needs assessment.