What’s evaluation good for?

January 5th, 2011 by Wendy Pollock

Exhibits people sometimes wonder whether evaluation is really worth the time, effort, and expenseor whether it’s just something funders require that they’d rather do without. Charlie Carlson sparked a round of discussion on the ISEN-ASTC listserv this week with a series of questions that arose from his many years as exhibit developer, and Alan Friedman was among those who responded. As our science museum colleagues know, Alan was not only the editor of NSF’s Framework for Evaluating Impacts of Informal Science Education Projects, published in 2008, but he also wrote a section in a 1991 book called Try It! Improving Exhibits Through Formative Evaluation that offered a rationale for why museum directors should  support evaluation. Here’s their exchange (reproduced here with their permission):

Charlie: For some years I’ve wondered about the efficacy of exhibit evaluation, wondered whether or not it is useful, or more directly a bureaucratic hurdle that provides useless and specious validation that satisfies an inner need and social, political need to feel affective.

Alan: Sounds like you are talking about summative evaluation.  Would you put formative evaluation in that same bucket?  I have dozens of personal experiences in which a few hours of formative evaluation told me what visitors totally misunderstood, or what they got right away, or some other revelation.  The resulting exhibitions were dramatically better than they would have been without the formative evaluation.  I’ve written up a summary of four examples from my earlier years in the field [PDF], but I’m sure every exhibition developer can add many more. I think front-end evaluation has equally dramatic impacts in many cases.

Charlie: To put it bluntly: Are museums and taxpayers spending a significant amount of money on something of questionable value?

Alan: Some evaluation is poorly done, and/or doesn’t show anything interesting.  The same can be said of money spent on legislation, classroom lessons, movies, books, TV shows, and even—scientific research!  The challenge is to improve the practice, and I think we are getting better, judging by the dozens of reports of careful, useful, and enlightening evaluations I hear at every Visitor Studies Association meeting.

Charlie: Museum visits are, indeed, events, fraught with every personal and social dimension.

Alan: Yes!  And many excellent evaluation studies have shown just how the personal and social dimensions can hinder or help a visitor’s experience.  Again many examples, but the experiments with creating “family size” spaces around individual exhibit units (Explora, Exploratorium), are examples.

Charlie: As such they are part of noise and chatter of day to day existence. Importantly, museum visits are also brief—ever so brief, hours out of a year (some fraction of 6570 waking hours annually).

Alan: See Falk and Dierking’s passionate article, “The 95% Solution,” published a couple of months ago in American Scientist.

Charlie: On the face of it, anyone funding an exhibit needs to know that they’re getting value for their commitment of resources, and more broadly whether or not it is having an intended effect.

Alan: That’s one reason, but not the major one.  At the moment I am on the boards of three foundations which fund science education, in and out of school, among other things.  We ask for evaluation because we want to become smarter funders.  And we learn from the evaluations of just about every project we support.  We learn a lot even if the evaluation shows no impact.  Because then we look at the arguments we bought when we decided to make the grant, and now we know what questions to ask the next time to help applicants prepare better proposals.

Charlie: What are the key concepts that characterize an excellent exhibit or museum?

Alan: Great question, and now you have taken us out of evaluation and into the area of research.  There has been very little research on these questions (but I see others are citing what does exist.)  I’ve written up my summary of research and meta-studies (PDF).

Charlie: Is there evidence that evaluation has improved or positively modified an exhibit or exhibition?  I think the evidence is scant.

Alan: Please see my first response, above.

Charlie: How much do people generally remember of a museum visit?

Alan: Again, there is such research, some of it going back many decades.  See work by David Anderson and Harris Shettel.

Charlie: Do the specifics of an exhibition make a difference in human behavior?  Probably not for most people.

Alan: Having to say “probably not” is just the sort of weak argument that research and evaluation can help avoid.  We can measure, and yes, some specifics do make a big difference.  See response above on formative evaluation.

Charlie: Has a museum exhibit changed the course of human history?  Probably not!

Alan: There’s “probably not” again.  But there is evidence to help answer that question.  See Steven Dubin’s book Displays of Power, and studies of early influences on the lives of people who become scientists (cited in my summary of impact studies, above).  Some exhibitions have spurred consideration of issues that might have been ignored, and changed lives and careers along the way.  I am confident in saying that museum exhibits have changed individual human lives, including my own (see Lessons from an English Summer).

Comments are closed.