Preview Issue - Research Articles
Evaluation’s Friendly Voice: The Structured Open-Ended Interview
Martha C. Monroe
An interview can be an extremely helpful tool in a
program evaluation. Guidelines for choosing to use
a structured open-ended interview are offered, as
well as three cases to demonstrate how interviews
can be used to gather important evaluative information.
Many program managers know the importance
of conducting program evaluation (C. Bennett,
1976; D. Bennett, 1988–89; Steelquist, 1993;
Stevens, 1993). Few organizations have the
luxury of operating without knowing if their
efforts are successful or how to suggest others
replicate their programs. Furthermore, evaluation
can provide ideas and directions for improvements
that may make the program more
effective. Patton (1986) defined program evaluation
as “the systematic collection of information
about the activities, characteristics, and
outcomes of programs for use by specific people
to reduce uncertainties, improve effectiveness,
and make decisions with regard to what those
programs are doing and affecting.” Since this
definition broadens a traditional ‘did-it-work’
evaluation to include a host of questions over
the life of the program, the challenge program
coordinators face is knowing which questions
to ask with which tool. Models, manuals, videos,
and consultants abound to help administrators
make these determinations, sometimes
overwhelming those who need some simple
answers to improve these programs. Though
questionnaires provide concrete data, there are
good reasons to consider using interviews in a
program evaluation.
A good evaluation would ask many different
questions, at appropriate times, with appropriate
tools, accommodating the program development
process in three distinct phases
(Herman, Morris, and Fitz-Gibbon, 1987;
Jacobson, 1991; Jacobson, 1999; Parsons, 1995).
First, evaluation can begin with the development
of the program itself, using a needs assessment
to better understand the audiences’ needs and
building the program’s objectives to match. Second,
evaluation may include formative strategies
that provide checks and balances as the
program is developed and tested. Finally, at the
program’s conclusion, a summative evaluation
measures student learning, teacher reaction,
environmental change, numbers attended,
funds raised, or trash barrels filled. This could
also include an analysis of observations and staff
impressions.
A variety of tools, such as surveys, case studies,
observations, records, and tests can be used
to gather information for an evaluation. Each
has an appropriate place and purpose. If it is
helpful to understand what participants already
know about the topic, how they perceive the
program, what they like, why they come, what
they would change, or how they think, evaluators
should consider one particular tool: the
structured open-ended interview. This tool can
be helpful in all three phases of a program evaluation
(needs assessment, formative, and
summative). It can yield important insights from
participants for program improvements, sometimes
at a comparatively low cost.
The recent use of structured open-ended
interviews to elicit useful data in several different
capacities (e.g., needs assessment, formative
evaluation) provides concrete examples of
the use of this tool to environmental education
program coordinators. This article will first define
the structured interview and describe how
it can be realistically used by practitioners
through these examples.
THE STRUCTURED OPEN-ENDED INTERVIEW
As the name suggests, a structured open-ended
interview is a systematic procedure to gather
spoken ideas, information, and opinions from
participants through a phone or personal interview.
Although some close-ended questions
(where a choice of responses is provided) may
be used to record demographics, the focus of
this type of interview is obtaining rich and informative
explanations offered to an openended
query. Such responses may not lend
themselves to quantification, but they are no less
valid when used to explain outcomes and offer
improvements than other sources of data. In
contrast, a traditional, close-ended marketing
interview would report percentages for each predetermined
response (i.e., nine out of ten recommend).
This tool is helpful in extrapolating
selected preferences and behaviors of the
sample to a large population.
Interviews with program participants or
leaders are considered an important source of
qualitative data in an evaluation (Patton, 1987).
Like every other tool, they have an appropriate
place and purpose, and are rarely seen as the
sole provider of information for an evaluation.
Rather, they are one source of helpful information,
and when combined with other tools, can
help complete the picture of the program’s
worth.
Michael Patton (1987) describes three
types of open-ended interviews that vary by the
degree of structure:
- the informal conversation, where questions are
spontaneously generated in context; each interview
is different and evolves in response
to the person being interviewed;
- the interview guide approach, where a list of questions
or issues to be covered is used during
the interview, in any order; a variety of questions
could elicit the general information
sought; and
- the structured open-ended interview, where a set
of carefully worded questions is created and
used for each interview.
The tradeoff along this continuum is between
flexibility in responding to unique perspectives
and minimizing variation with systematic
questions. It is possible, however, to combine
these techniques with carefully worded
questions followed by probes that address additional
subjects with less structure. Close-ended
questions can be added to collect specific and
comparable data. Although these combinations
may violate important philosophical distinctions
between inducing analysis based on observations
and quantifying differences in an experimental
setting, the combinations represent reasonable
techniques for practitioners interested in
understanding their program and how it might
be improved.
Open-ended interviews are not practical if
responses from large numbers of participants
are needed (as when generalizations are made
for a population) or if a practitioner wishes to
report percentages or categories (i.e., “fourth
graders enjoy this activity more than eighth graders”
or “teachers who have attended a workshop
use a greater variety of EE strategies than those
who have not”). The process of interviewing
enough people would take too long and openended
responses would be too difficult to summarize.
These data can be obtained with questionnaires
and closed questions, unless there are
other considerations, such as language challenges.
So when does one choose an interview
format?
The following list of questions may help
determine if the structured interview is an appropriate
data collection tool for your program.
Should we use a structured openended
interview?
To under understand stand the program
- How helpful is it to understand a process, a
history, or an opinion? Do you need to ask
questions like, “Why do you think this program
is successful?” or “How did you become
active in this effort?” or “Why do you like
coming here?” Is it helpful to understand
potentially unique perspectives from participants.
“How did you use these materials in
your classroom?” Are you interested in how
people feel or what they experience? This
information is best gathered through openended
questions. Respondents need the opportunity
to describe their ideas in their own
words.
- Is it possible to improve the program quality
with these data? Are the staff willing and able
to make adaptations and consider changes?
If the results won’t be used, perhaps the effort
should not be expended.
To interpret outcomes
- How important is it to enable others to replicate
this program? Is this a model or pilot
effort? Sometimes the stories, histories, and
explanations from a few key people are invaluable
in helping others build a similar
program.
- How important are the potential unexpected
outcomes of the program? Would it be helpful
to report gains and successes that were
not part of your original program objectives?
If you are only looking for the occurrence of
a few known characteristics, you can create a
closed question. If you are open to learning
things you didn’t know before, a series of
open questions may enable these ideas to surface.
Some important insights and explanations
may result that were not anticipated.
- Who receives the evaluation report? If readers
are unfamiliar with the program, they may
need to learn about the program from reading
accounts of participants’ experiences.
Regarding logistics
- Would quotes be acceptable in the evaluation
report? Some funders report that short
stories and examples may be the most memorable
and valuable aspect of an evaluation.
Other funders may prefer numbers.
- Will more than one person be conducting
the interviews? If so, some degree of structure
will be helpful (an informal conversation
may not yield systematic responses).
- Is it feasible to contact participants by phone?
This can greatly simplify the procedure and
shorten the time needed.
- Do you have less than 10 key questions? Unless
respondents are committed to the program
and prepared to devote a considerable
amount of time to the interview, participants
may balk at an interview that exceeds 20 minutes.
- Can you identify about 20 (or fewer) key individuals
who will have valuable insights? The
interview process does not rely on a large
sample, but rather a purposeful sample.
More than 20 interviews may become difficult
to report effectively and still capture the
depth of responses.
Affirmative answers to these questions
should suggest that a qualitative method of collecting
information could help in understanding
your audience, improving your program,
explaining your program and its successes, and
helping others conduct similar programs. The
final decisions about conducting observations
or interviews, by phone or in person, with open
and/or close-ended questions, would depend
upon the nature of your questions, the number
of people involved (both as interviewers and
interviewees), the resources available, etc. Below
are some cases in which structured openended
interviews were used to assist environmental
education programs.
Using the structured open-ended interview
Each of the following examples is extracted from
a larger program evaluation process. In these
cases, the program evaluation also used surveys,
observations, knowledge tests, and participant
reports to assess the impact and success of the
program. The examples presented here are but
one piece of those broader efforts.
As a needs assessment
Prior to the final meeting of a five-day teacher
in-service program that occurred over five
months, two types of short needs assessment
were conducted to develop an agenda that
matched the participants’ interests. The written
survey confirmed the date and asked participants
to rank 10 potential agenda topics.
Phone interviews were conducted with about
half the participants asking only three questions:
which activities teachers had conducted with
their students, how that experience went, and
whether teachers had begun working on the final
assignment. (Interviewees were randomly selected
based on who answered the phone and
was able to talk for 5–10 minutes on a weekday
evening.) Organizers believed these questions
would help them know what the participants
needed in the final day of the program.
The interviews were an easy way for teachers
to speak comfortably about a favorite topic—
their students—and some went into great detail
describing their projects. The calls revealed
that the final assignment was a source of confusion
since teachers took the opportunity to ask
questions about it. Few calls lasted more than
five minutes and all were completed in one
evening (24 calls resulted in 11 interviews, 2
wrong numbers, 9 not home, and 2 who preferred
not to talk at that time). These responses
helped organizers develop an icebreaker that
used participants’ experiences and suggested
that participants needed another handout
that more clearly defined the assignment (see
Note 1).
As a formative evaluation tool
A major three-year partnership involving five
federal sites and six school districts in developing
curriculum-based lessons for teacher-led
field trips used structured open-ended interviews
with each of 40 participants to record and
respond to their concerns and satisfactions. The
questions were drafted, revised, and approved
by the program coordinator; an outside consultant
conducted the interviews so potentially
negative responses could be more easily voiced.
A newsletter to participants explained the upcoming
phone interview, and an enclosed postcard
enabled participants to send the interviewer
their preferred interview time. Even so,
catching teachers and administrators at a phone
during the day sometimes required several tries!
Interviews ranged from 15 to 45 minutes depending
on the degree of detail the participants
provided and were completed over a six-week
period.
The interviews asked a series of open- and
close-ended questions of the teachers, rangers,
and administrators, depending upon the role
they played in the program. Responses were
reported by category, with the largest group
having 25 respondents. Teachers and rangers
were asked how successful their field trips were
and what accounted for this success, in their
opinion. All participants were asked to describe
the benefits they saw in the program, what concerns
they had at that point, and what they
thought their strongest memories would be of
their experience five years into the future.
The open-ended and qualitative nature of
this evaluation tool allowed several unexpected
outcomes of the program to surface, such as the
improvement of rangers’ and teachers’ teaching
abilities as a result of the curriculum development
process and the importance of powerful
friendships and networks. The interviews
enabled the program coordinator to understand
the variations in perspective toward the program
(some expectations held by teachers were not
shared by rangers, for example) and to begin
to focus on the concerns of some participants
in the program’s final year. Other benefits included giving participants who were typically
quiet in the workshops and meetings an equal
voice for the record and gathering interesting
details and rich examples of the teachers’ experiences.
It was not necessary to interview all
40 participants to obtain the diversity of views,
but the program coordinator did not want to
randomly select some participants and risk hurt
feelings. Because the interviews were conducted
at the program’s midpoint, they became part of
the program development process, providing
another exercise to help participants learn
about their own professional growth and development
(see Note 2).
To pre-test summative survey instruments
A small number of interviews can enhance the
usefulness of a large-scale survey. During the
development of a national survey instrument for
middle school students on biodiversity, small
groups of students from three different urban
schools were interviewed to make sure they were
reading, understanding, and answering the
questions as intended. While a pre-testing step
is often critical to the success of a survey, it is
rarely simple to administer. One school decided
to engage the class in a special activity while a
few students were pulled aside and interviewed;
another teacher wanted to involve all students
in the process over several days of interviews.
In both cases, students were interviewed in small
groups of five to eight because it was unclear if
they would speak openly with a stranger.
In each school, students completed a short
version of the survey then met with the interviewer
who used the survey and asked students
several additional questions: “What were you
thinking about when you answered this question?”
(e.g., which animals came to mind) and
“What examples can you think of?” Prompts
such as “What do you mean by that?” and “Can
you explain more?” were frequently used to elicit
comments, and sometimes direct questions were
used to challenge their tangential ideas (e.g.,
“Why don’t they shear tigers like sheep to get
their fur?” and “What would kill the animals if
someone built a mall on their home?”). With
these questions the interviews took on the characteristics
of an “informal conversation” (Patton,
1987). The results confirmed which survey questions
were appropriate for this age group and
identified a few confusing words and questions
(see Note 3).
SUMMARY
The structured open-ended interview can be a
simple, non-threatening tool to gather useful
information in a program evaluation. It can
complement other evaluation tools by pre-testing
forms or helping to create close-ended questions.
It can allow evaluators to describe a program
experience in specific detail and offer a small
selection of personal insights. It is cost-effective
with small groups, and indeed, may be the best
way to obtain certain types of information.
Beyond all the right reasons to use interviews
for an evaluation, they can be a friendly
tool for those who are unfamiliar with evaluation.
Both interviewees and interviewers may
find that a few open-ended questions provide
all the information they need to confirm or
improve on an educational program.
Note 1
World Wildlife Fund, in conjunction with the
Smithsonian Institute, Howard University, and
the National Zoo, conducted a two-part
Biodiversity Leadership Institute in Washington,
DC for educators. The Institute was supported
by the National Environmental Education and
Training Foundation and Eastman Kodak Company,
and the interviews were part of a broad
evaluation strategy.
Note 2
The Potomac Area Rural Teachers Using National
Education Resources for Students (PARTNERS)
program was funded by the National
Park Foundation with generous support from the Pew Charitable Trust, the National Park
Service, the National Biological Service, and the
Corning Foundation.
Note 3
World Wildlife Fund, in conjunction with the
Wisconsin Center for Environmental Education
at the University of Wisconsin-Stevens Point,
developed a Biodiversity Literacy Survey for
middle school students as part of their environmental
education program Windows on the Wild.
The open-ended interviews were used to pretest
the survey questions and gain information
about what students know about biodiversity.
REFERENCES
Bennett, C. (1976) Analyzing impacts of extension programs.
USDA: Agricultural Cooperative Extension, Report 511.
Bennett, D. B. (1988–89) Four steps to evaluating environmental
education learning experiences. Journal of Environmental
Education 20(2), 14–21.
Herman, J. L, Morris, L. L., & Fitz-Gibbon, C. T. (1987)
Evaluator’s handbook. Newbury Park, CA: Sage.
Jacobson, S. K. (1991) Evaluation model for developing,
implementing, and assessing conservation education
programs: Examples from Belize and Costa Rica. Environmental
Management 15(2), 143–150.
Jacobson, S. K. (1999) Communication skills for conservation
professionals. Washington, DC: Island Press.
Patton, M. Q. (1986) Utilization-focused evaluation. Newbury Park, CA: Sage.
Patton, M. Q. (1987) How to use qualitative methods in evaluation.
Newbury Park, CA: Sage.
Parsons, C. (1995) Project planning and evaluation model
workbook. Prepared for the Office of Training and Education
of the U.S. Fish and Wildlife Service.
Steelquist, R. (1993) Evaluation—Right from the start.
Developed for the Office of Training and Education of
the U.S. Fish and Wildlife Service.
Stevens, F., Lawrenz, F., & Sharp, L. (1993) User-friendly
handbook for project evaluation: Science, mathematics, engineering,
and technology education. Arlington, VA: National
Science Foundation.
Martha C. Monroe
School of Forest Resources and Conservation
University of Florida
PO Box 110410, Gainesville, FL 32611-0410
E-mail: mcmo@gnv.ifas.ufl.edu
<< Back to Research Articles
