Whole Document Navigator (Click Here)    ---------------------------------- Top of Document Introduction Advantages of Focus Groups Focus on CNR Implementing Focus Groups -Participants --Whom to Invite --Number of People --Ensuring Attendance -Logistics --Time of Day --Length of Session --Number of Sessions --Location and Equipment --Trouble Shooting -Conducting the Focus Group Session --Facilitation --Introductions --Getting Discussion Started --Monitoring the Flow of Discussion --Questions --Getting Everyone to Participate --Concluding the Session --Demographics -Recording -Analysis of Data and Making Changes -References Footnotes Disclaimer Copyright Infomation

Evaluating Web Sites through the use of Focus Group Interviews¹

Bartels, W., Breeze, M. Peterson, N.²

Introduction

The Internet is widely used by professors, Extension agents and other professionals as a data-gathering tool because of the unlimited possibilities for academic research. Results from a survey done by Leung (1998) show that the Internet greatly facilitates communication and encourages research collaboration.

This electronic transformation is not without its drawbacks. Despite its convenience, browsing on the Internet has a tendency to consume chunks of ones day without accomplishing any goals. Harris (1996) found that people wasted time when surfing on badly designed web pages.

An effective web site should provide people with useful, easily accessible information that offers depth but does not burden the reader with unnecessary details. Determining whether the Internet medium helps public organizations realize their missions is important because setting up and maintaining these sites is supported by taxpayer dollars. Evaluating the effectiveness of web-based communication can assess customer satisfaction and return on investment. It makes good business sense.

This fact sheet will give you suggestions for conducting focus group interviews to determine whether the design of your web site helps realize the mission of your organization.

Advantages of Focus Groups

Why not just post an online user survey on your web site? An online survey can be used to gauge audience satisfaction of a particular web site. Online surveys are convenient, don't cost much money and can easily reach the target audience. However, the type of information gained from online surveys is limited. Since Internet users are usually pressed for time, most on-line surveys are set up using answer boxes that the readers click on. Therefore, responses are defined by the questions. According to Merton, "There would be little point in using the interview at all, if it simply resolved itself into a fixed list of stock questions put by the interviewer." (Merton et al., 1990). In other words, you may not touch on a variety of elements that might need to be addressed simply because you hadn't included them in the survey.

A major advantage of conducting focus groups is the depth and richness of information that can be learned. Members of the group remind each other of details they might otherwise not have mentioned if interviewed privately (Merton et al., 1990). In addition, the moderator can ask probing questions and explore unanticipated issues. "The focus group discussion is particularly effective in providing information about why people think or feel the way they do." (Krueger, 1994)

Focus on CNR

To determine the effectiveness of web-based communication at the UF/IFAS Center for Natural Resources, two focus groups were conducted. Results from these sessions provided CNR with valuable insights from members of its target audience. Drawing upon this information, the web design team pinpointed changes to enhance the web site.

Initially, literature was gathered on conducting focus groups to evaluate web sites. Included here is general information gleaned from useful resources. In addition, the synthesis contains lessons learned from the two CNR focus groups. Other departments and educational organizations that use the World Wide Web as a communication tool may find these lessons valuable.

Implementing Focus Groups

This section provides an idea of the steps involved in preparing for, conducting and following up on focus groups that are designed to evaluate web sites.

Participants

Whom to Invite

• Participants for a focus group are selected because they have something in common within a defined area that relates to the research question (Krueger, 1994).

• Who better to decide on the effectiveness of a web site than members of the target audience? Know your audience! Having a clear understanding of who the members of your target audience are will make participant selection easier.

• Think about whether you want participants who have had previous experience with your particular web site or if you'd prefer people who are unfamiliar with it.

• You can either hand pick participants or develop a random sampling method. Despite using random sampling, focus group results cannot be used to make inferences about an entire population (Krueger, 1994). If you want to be sure to assess certain groups within your target population, you may want to specifically invite people from those particular groups (e.g., students, mothers, elderly people).

• Avoid intact organizations or situations where personal/professional relationships might hamper discussion.

• In the CNR focus groups, some participants were chosen because they were the most likely to use the natural resource information on the web site. Of these people, some had in fact previously used the site. Other participants were chosen specifically because they had web design or communication expertise.

Number of People

• The typical size for a focus group is between 6 and 10 people (Krueger, 1994). Due to site specifications that relate to computer availability, it is especially important to keep the groups manageable.

Ensuring Attendance

• Send out a personalized invitation to each participant well in advance of the proposed session. Tell them that they will be contributing to a research project for which their ideas will be greatly valued.

• Depending on whom you invite, you may need to consider an incentive for attendance. Food, money or gifts are options to compensate for attendance (Krueger, 1994).

• Participants might cancel at the last minute or simply not show up, so we recommend that you recruit a few more people than you absolutely need for the final group size.

• Be organized. Make a spreadsheet with participants' contact information so that you can get in touch with them a few days before to confirm attendance. Leaving telephone messages is not a good idea.

Logistics

Time of Day

• Accommodate your audience. Try to choose a time of day that will be convenient for your participants. Consider the traffic and driving time to the facility before scheduling the focus group first thing in the morning.

• If the session will run over lunch you might want to think about providing refreshments. However, eating during a focus group in which you'll be using computers is generally not recommended. It is both distracting and messy.

Length of Session

• A two-hour session will allow you to cover most elements of web design evaluation. Be flexible because depending on the group dynamic and the skills of the facilitator, you may need to adjust the time. The discussion might wind down before the time is up. On the other hand, try not to go over-time.

Number of Sessions

• This will depend on resources, time and ease of participant recruitment and audience breakdown. Plan to do about three separate focus group sessions (Krueger, 1994). You can modify this as you go along.

Location and Equipment

• Choose a comfortable, quiet room with adequate airflow and lighting.

• Arranging the chairs so members can see each other will aid discussion. Spontaneity is affected by the spatial distribution of the group (Merton et al., 1990).

• Ideally you'll want to make sure that each participant has a computer so that participants can navigate around the web site individually.

• Finding a facility with enough available computers for the entire group might be difficult. Computer rooms have to be reserved so be sure to do this well in advance.

• Usually the room will have a master station for the instructor so that the site can be projected up on a screen. This helps during the discussion to make sure everyone is on the same page. Find out if the room has the master station when searching for an appropriate venue.

• A flip chart or overhead projector is sometimes useful to keep the main ideas in front of the group.

• Provide a name tag for each participant.

Trouble Shooting

• Determine whether the facility has staff members who can be on stand by in case things go wrong. For instance, the web server might crash so that you cannot access the web site.

• Different browsers display the web site differently. Familiarity with Internet Explorer and Netscape will be most useful. Find out what browser the facility uses so that you can anticipate the "look" the web site will have during the focus group session.

Conducting the Focus Group Session

Facilitation

• Have a skilled group manager who can guide discussion and who has no affiliation with the organization. The neutrality of this person is important. "If the staff member or volunteer is readily identified with the organization or, for that matter, identified with any controversial issue in the community, the quality of the results could be jeopardized (Krueger, 1994)."

• Using the web designer or an information technology person can also bias the results because they may find it difficult not to interrupt the discussion. They might be tempted to try to justify certain elements of the web site.

• At the same time, it might be useful for at least one of the recorders to be familiar with the web site. There are instances when the audience can get stuck on one question and feel unable to move forward. Being able to give an answer on the spot is sometimes useful. For example, the CNR web site has an interactive calendar, which caused much concern among participants. The calendar is technically supported through a third-party vendor (outside the university). One of the recorders was able to tell the audience quickly about the degree of control the CNR web master had on the formatting and design of this calendar. This helped to move discussion along.

Introductions

• Briefly explain the purpose of the focus group. Tell participants that discussion will cover what they like and don't like about the web site. Emphasize that the session is nether a competition nor a consensus and that there will be full confidentiality. The facilitator can mention that he/she is serving as a non-judgmental disinterested party, free of concern for what participants have to say.

• Allow the participants to introduce themselves.

Getting Discussion Started

• As mentioned above, when you ask a direct question, the answer is defined by the way you ask the question. Instead of asking direct questions, tasks can be used to give participants a common experience "surfing" the web site. Six different tasks were given to the participants in the CNR focus group. The following are examples of the CNR tasks:

1. What animal was featured on the front page of the summer 2000 Linkages newsletter?

2. What 3-day-event is being held towards the end of January 2002?

3. Eco tourism vs. Eco indicators - what is the difference in how the content on these program pages is presented?

• The way these tasks were crafted allowed the participants to explore at least the first page from the ten main links given on the home page. Participants were asked to pay special attention to the route that they used to locate the page they were asked to find. A discussion followed each task.

Monitoring the Flow of Discussion

• Initially general comments or observations about the site can be noted and the participant's experiences with the site documented. Monitor the flow of discussion around the question categories given below.

• After that, further comments regarding, specifically, navigation, content and design can be gleaned.

• Finally, any questions of interest not spontaneously addressed can be covered.

Questions

The three major areas of inquiry are navigation, content and aesthetics (see Table 1). Discussion can be directed by going through some of the questions listed below. During the CNR focus group the moderator did not ask pointed questions but tried to steer the conversation so that each area was covered during the tasks.

Table 1. Major Areas of Inquiry

Navigation Is the web site easy to navigate? Do you have difficulty finding information? Do you find the navigational tools confusing? Do off-site links that open in new windows confuse you?

Content Should the scope of information provided be more specific? Is the information evenly distributed across various departments? Is the information accurate? Is the information useful? How might you use the information provided on the site?

Aesthetics Color: Is the color combination appropriate? What does the color combination remind you of? Would you want to change it? If you did, what would you change it to? Are the colors too bright/dull/distracting? Graphics: Are there enough pictures? If you'd prefer more, what sort of pictures would you like? If you'd prefer less, which would you remove? Are the graphics downloading quickly enough? Are the graphics appropriate/inappropriate? Do they complement the text? Font: Is the text difficult to read? Should the text be in a different color/font/size?

Getting Everyone to Participate

• Sometimes one or two people will dominate the meeting. The moderator should ensure even participation by giving every person a chance to express ideas says Krueger (1994, p.109): "the moderator and assistant are observing participant interaction and noting individuals who tend to dominate the group, are excessively shy, or consider themselves experts." This researcher suggests putting the talkative people near the moderator if possible and then using body language, like turning away from the domineering participant, as a signal to others to talk.

Concluding the Session

• When the session is coming to an end, the moderator can wrap things up and ask for any final overall comments about the web site.

• Thank the participants for their time and valuable observations and promise to send them a report of the session's findings.

Demographics

• A one-page survey of personal information and web use can be handed out at the beginning of the session and collected at the end (Table 2 is an example of the CNR questionnaire.) This information can influence the way the information is viewed during the final analysis of the focus group results. For example, a user's previous web experience can sway his/her response.

Table 2. CNR Demographics Questionnaire

Year you were born _______
Male _____	Female _____
How frequently do you access the web?
	Once a month _____
	Once a week _____
	Once a day _____
	More than once a day _____
If you access the web daily, how many hours do you spend on the web?
	Less than 2 hours a day _____
	More than 2 hours a day _____
If not daily, how many hours do you spend online when you DO access the web _____?
What is your level of familiarity with the web? (circle one)
	Not at all familiar	1 2 3 4 5	Extremely familiar
Do you access the web more frequently at work or at home?
	Work _____	Home _____	Equally _____
What % of the information you get from the web is work related and what % is leisure related?
	Leisure _____ %	Work _____ %
To access information on natural resources in Florida, which web sites would you most likely go to? (list the top 3 that you use most commonly)
	a)
	b)
	c)
How important do you consider the content of a web site?
	Not at all important	1 2 3 4 5	Extremely important
How important do you consider the design of a web site?
	Not at all important	1 2 3 4 5	Extremely important
How important is the web in your life?
	Not at all important	1 2 3 4 5	Extremely important

Recording

• Record the session with either an audio or audio-video recorder. Check that you have batteries/an extension cord and enough tapes.

• It is not practical to count on remembering the details of what is said during the focus group discussion. Have a note taker to write everything down. These notes are essential (Krueger, 1994).

• Pay attention to gestures and facial expressions. These should be noted, as well as any surprises.

• It is useful to weight the responses. In other words, note how many people feel the same way about a particular element on the site. Write down if everyone agreed on a comment that was made or if it was simply a matter of one person's opinion. This information is useful when it comes to analyzing the results so that those who did not attend the focus group can get a sense of how strongly the group felt about certain issues.

Analysis of Data and Making Changes

• Using the tape recorder and your notes, transcribe the session to include as much detail as possible. Sometimes transcription of the entire discussion might not be a viable option because it is very time consuming.

• Write up a report for each session. These can then be compared and a summary document compiled with observations and proposed changes to the web site. Remember to include the demographic information.

• At CNR we found it useful to summarize changes as they applied to each web page that we had covered in the focus group during the tasks. This allowed us to focus on the changes for particular web pages. However, you might want to cover the three broad areas of navigation, content and aesthetics when finally compiling the information.

• Schedule a meeting to present the results to the web team, supervisor and director of your organization. Discuss what changes will be made, when and by whom.

• Send a summarized copy of the final report (optional) and a thank you letter to each participant, inviting them to visit the site and see the changes.

References

The following documents offer useful background information:

• Harris, C. 1996. An Internet education: A guide to doing research on the Internet. Wadsworth, Belmont CA

• Israel, G. D. Using focus group interviews for evaluating Extension Programs. UF/IFAS Edis Publication Fact Sheet PEOD-14. Although this Extension paper covers general suggestions for conducting focus groups, it does not offer specific advice on web sites. This CNR fact sheet, along with a review of Israel's tips will help in understanding the task of conducting focus groups for website evaluation.

• Krueger, R.A. 1994. Focus groups: A practical guide for applied research. Sage Publications.

• Leung, Y. 1998. Using the Internet for natural resource research: Results from an online user survey. Journal of Natural Resources Life Sciences Education, Vol. 27.

• Merton, R K., Fiske, M., Kendall, P.A., (1990) The Focused Interview: A manual of problems and procedures. The Free press, NY.

---

Footnotes

1. This document is CNR14, one of a series of the Agricultural Education and Communication Department, Florida Cooperative Extension Service, Institute of Food and Agricultural Sciences, University of Florida. Original publication date November 2002. Reviewed June 2003. Visit the EDIS Web Site at http://edis.ifas.ufl.edu.

2. This study was conducted by The Center for Natural Resources (CNR). CNR, was established in 1973, at the University of Florida. The Center plays a major role in the conservation, preservation and restoration of our nation's natural resources by facilitating interdisciplinary collaborations between UF faculty and external stakeholders.

Wendy-Lin Bartels is a graduate student with a research assistantship as a communication specialist with CNR. Nancy Peterson is the Program Coordinator at CNR. Marshall Breeze is a Professor, Department of Agricultural Education and Communication, Coopertative Extension Service, Institute of Food and Agricultural Sciences, University of Florida.

---

The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex, sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other extension publications, contact your county Cooperative Extension service.

U.S. Department of Agriculture, Cooperative Extension Service, University of Florida, IFAS, Florida A. & M. University Cooperative Extension Program, and Boards of County Commissioners Cooperating. Larry Arrington, Dean.

---

Copyright Information

This document is copyrighted by the University of Florida, Institute of Food and Agricultural Sciences (UF/IFAS) for the people of the State of Florida. UF/IFAS retains all rights under all conventions, but permits free reproduction by all agents and offices of the Cooperative Extension Service and the people of the State of Florida. Permission is granted to others to use these materials in part or in full for educational purposes, provided that full credit is given to the UF/IFAS, citing the publication, its source, and date of publication.