Exhibits people sometimes wonder whether evaluation is really worth the time, effort, and expense—or whether it’s just something funders require that they’d rather do without. Charlie Carlson sparked a round of discussion on the ISEN-ASTC listserv this week with a series of questions that arose from his many years as exhibit developer, and Alan Friedman was among those who responded. As our science museum colleagues know, Alan was not only the editor of NSF’s Framework for Evaluating Impacts of Informal Science Education Projects, published in 2008, but he also wrote a section in a 1991 book called Try It! Improving Exhibits Through Formative Evaluation that offered a rationale for why museum directors should  support evaluation. Here’s their exchange (reproduced here with their permission):

Charlie: For some years I’ve wondered about the efficacy of exhibit evaluation, wondered whether or not it is useful, or more directly a bureaucratic hurdle that provides useless and specious validation that satisfies an inner need and social, political need to feel affective.

Alan: Sounds like you are talking about summative evaluation.  Would you put formative evaluation in that same bucket?  I have dozens of personal experiences in which a few hours of formative evaluation told me what visitors totally misunderstood, or what they got right away, or some other revelation.  The resulting exhibitions were dramatically better than they would have been without the formative evaluation.  I’ve written up a summary of four examples from my earlier years in the field [PDF], but I’m sure every exhibition developer can add many more. I think front-end evaluation has equally dramatic impacts in many cases.

Charlie: To put it bluntly: Are museums and taxpayers spending a significant amount of money on something of questionable value?

Alan: Some evaluation is poorly done, and/or doesn’t show anything interesting.  The same can be said of money spent on legislation, classroom lessons, movies, books, TV shows, and even—scientific research!  The challenge is to improve the practice, and I think we are getting better, judging by the dozens of reports of careful, useful, and enlightening evaluations I hear at every Visitor Studies Association meeting.

Charlie: Museum visits are, indeed, events, fraught with every personal and social dimension.

Alan: Yes!  And many excellent evaluation studies have shown just how the personal and social dimensions can hinder or help a visitor’s experience.  Again many examples, but the experiments with creating “family size” spaces around individual exhibit units (Explora, Exploratorium), are examples.

Charlie: As such they are part of noise and chatter of day to day existence. Importantly, museum visits are also brief—ever so brief, hours out of a year (some fraction of 6570 waking hours annually).

Alan: See Falk and Dierking’s passionate article, “The 95% Solution,” published a couple of months ago in American Scientist.

Charlie: On the face of it, anyone funding an exhibit needs to know that they’re getting value for their commitment of resources, and more broadly whether or not it is having an intended effect.

Alan: That’s one reason, but not the major one.  At the moment I am on the boards of three foundations which fund science education, in and out of school, among other things.  We ask for evaluation because we want to become smarter funders.  And we learn from the evaluations of just about every project we support.  We learn a lot even if the evaluation shows no impact.  Because then we look at the arguments we bought when we decided to make the grant, and now we know what questions to ask the next time to help applicants prepare better proposals.

Charlie: What are the key concepts that characterize an excellent exhibit or museum?

Alan: Great question, and now you have taken us out of evaluation and into the area of research.  There has been very little research on these questions (but I see others are citing what does exist.)  I’ve written up my summary of research and meta-studies (PDF).

Charlie: Is there evidence that evaluation has improved or positively modified an exhibit or exhibition?  I think the evidence is scant.

Alan: Please see my first response, above.

Charlie: How much do people generally remember of a museum visit?

Alan: Again, there is such research, some of it going back many decades.  See work by David Anderson and Harris Shettel.

Charlie: Do the specifics of an exhibition make a difference in human behavior?  Probably not for most people.

Alan: Having to say “probably not” is just the sort of weak argument that research and evaluation can help avoid.  We can measure, and yes, some specifics do make a big difference.  See response above on formative evaluation.

Charlie: Has a museum exhibit changed the course of human history?  Probably not!

Alan: There’s “probably not” again.  But there is evidence to help answer that question.  See Steven Dubin’s book Displays of Power, and studies of early influences on the lives of people who become scientists (cited in my summary of impact studies, above).  Some exhibitions have spurred consideration of issues that might have been ignored, and changed lives and careers along the way.  I am confident in saying that museum exhibits have changed individual human lives, including my own (see Lessons from an English Summer).

For exhibition colleagues resolved to pay more attention to evaluation during the coming year, here’s a resource from the American Evaluation Association (AEA).

AEA365: A Tip-a-Day By and For Evaluators “is dedicated to highlighting Hot Tips, Cool Tricks, Rad Resources, and Lessons Learned for evaluators” with the goal of featuring “a post a day from and for evaluators around the globe.” You can read it, and you can submit contributions yourself. One of the recent posts was by our own evaluator, Carey Tisdal, on “practical applications of theory.” You can read it here.

Here’s to even better exhibitions in 2011 – and best wishes to ExhibitFiles members and supporters!

Paying attention: It’s at the heart of learning, an aspect of aesthetic experience, how we make meaning, a topic of recurring interest among those who design museum exhibitions. So how do we get people to pay more attention? In a major review of the topic commissioned by the Visitor Studies Association, Steve Bitgood offers background and practical guidance. Attention, he writes, “is a three-level continuum (capture, focus, engage) with a different combination of variables influencing attention at each stage.” People are making judgments all the time about where they direct their attention, he says. So “attention is perceived value (a ratio of utility/satisfaction divided by costs such as time and effort) of the exhibit element.”

The article offers a framework for thinking about things most of us already know, if only from being museum visitors ourselves. Fatigue, distraction, too many things to see and do all at once all work against attention.

But what about the value proposition–how do we make an exhibit that compels people to turn attention that way? It comes down to two simple things, Steve writes: “(1) by selecting high interest exhibit content; and/or (2) by designing exhibit elements that stimulate curiosity.” The heart of exhibition design, its mystery and challenge–and the reason we get to know the people we’re designing for.

“An Attention-Value Model of Museum Visitors” was published by the Center for Advancement of Informal Science Education (CAISE) and is available as a downloadable PDF on the CAISE website.

Peter Anderson, known to many in the science museum field, died October 15 near his home in Victoria, BC. A member of ExhibitFiles, Peter was author of the 1991 book Before the Blueprint, which offered guidance to science center planners during a time of peak growth. The book drew on his own extensive experience with museum start-ups and expansions, from Amsterdam to San Jose, Pittsburgh to Glasgow.

Peter was also instrumental in the Museum Impact and Evaluation Study, which resulted in a three-volume report published in 1993 on “Roles of Affect in the Museum Visit and Ways of Assessing Them.” The study group was interested in relationships repeat visitors form with what they called “icon exhibits,” like the Buhl Planetarium’s railroad layout and the Museum of Science and Industry’s Coal Mine.

Many of us also will remember Peter for his contributions in recent years to discussions on ISEN-ASTC-L, the Informal Science Education listserv, and his warm presence at ASTC and Ecsite conferences. He will be missed.

The latest issue of Exhibitionist, the journal of the National Association for Museum Exhibition (NAME), appeared in mailboxes last week.  In The International Project: A Cultural Journey, authors (mostly from North America) share frustrations and insights gleaned from work in other countries, ranging from France to Rwanda, Ukraine to Nepal.

As usual, there are also exhibition sightings (in Exhibits Newsline, edited by Beth Redmond-Jones) and Nuts and Bolts articles, including one on writing labels for translation by Penny Jennings.

NAME generously posts back issues after a year, but even before then, some articles can be downloaded free here, including an article from the Spring 2010 issue by Donna Braden called “Your Personal Toolkit: Easing through Friction, Fracas, and Free-for-All.”

NAME (the National Association for Museum Exhibition) has posted entry requirements for the 23rd Annual Excellence in Exhibition Competition. The competition recognizes outstanding achievement in exhibition from all types of museums, zoos, aquariums, botanical gardens, and non-commercial institutions offering exhibitions to the public. The exhibition must have opened to the public between November 30, 2008 and November 30, 2010, and at least one team member must be a member of the American Association of Museums (AAM). The deadline for entries is January 14, 2011. To find out more and download entry requirements and a submission form, visit the NAME website.

A guest blog post by Joanna Fisher, ExhibitFiles member since July 2007

Since the beginning of the year, I have been meeting with fellow museum professionals from a variety of local museums to experiment with and learn from the use of Judging Exhibitions: A Framework for Assessing Excellence, by Beverly Serrell. Our group includes people from a variety of backgrounds and a spectrum of museum types. Each person in the group uses the Framework to review an exhibition on their own. Then we meet over sack lunches once every two months to follow up and discuss our findings. I try to capture the views of the entire group in a review, informed by their reviews and the discussion, which I post on Exhibitfiles. (Two have been posted—Destination Argentina and galleries at the John Hutchings Museum of Natural History—and two more will be posted in the next few weeks.)

We held our first meeting in February and reviewed how we would use the Framework. (See below for link to PDF.) It is exciting to talk about what makes a good exhibition, we agreed, and there would be differences of opinion. The Framework would help get us all on the same page, speaking the same language, and thinking more carefully and thoroughly about what we see as evidence of excellence. The process of thinking and discussing has increased our abilities to review, critique and evaluate for excellence—and to strive for exhibitions that have the most excellence possible.

When we visit an exhibition on our own, we complete the steps to come up with a rating. Beverly defines them this way:

• Call-outs: your experiences in the exhibition as a visitor
• Aspects: the evidence you found that supported each criterion (Comfortable? Engaging? Reinforcing? Meaningful?)

I really enjoy coming up with call-outs. I can wonder on paper about the things I see and hear. I can like things just because I like them. And I can note when I think something is stupid. There aren’t really any “rules” that I need to consider, I just need to experience the exhibition. When I “finish” my visit, it is time to assess the “aspects.” I am mostly considering questions that are laid out pretty explicitly in the Framework. Basically, I just ask myself if the exhibition “is” or “isn’t” comfortable, engaging, reinforcing, and meaningful, based on what I wrote in my call-outs.

After finishing these two steps, it is time to use my brain, to “rate the criteria”—then to identify the evidence to support those ratings. Rating the criteria requires that I 1) start using a different part of my brain, and keep using it until my brain adjusts to the shift and 2) make commitments. I love to analyze and think, but I still have to talk myself into assigning ratings, to be a judge. Once I get into it, it does get easier and more enjoyable.

I am reminded of the dreaded annual reviews with the boss: I love it when the review tells me I have exceeded expectations and done more than required, and a bit disappointed when told that I am doing the job I was hired for at the level it needs to be done. I think we find most exhibitions also to be a mixture of good, bad, and “acceptable.” But I am very confident that most, if not all, exhibitions have something in them that is excellent. One purpose for the Judging Exhibitions experiment is to find the ways that we can increase excellence and not always have to settle for “acceptable.”

The group discussions have lived up to all my expectations. One discussion stands out in my mind: during one of our reviews, each of us was able to identify ways that the exhibition could be improved. Some were pretty simple things like removing some of the artifacts to allow room for the artifacts that were on display. Some ideas were getting pretty elaborate, including knocking out walls and redesigning entire galleries to focus the storytelling and interpretation to provide increased context and relevance for multiple styles of visitors. Frankly, I think we batted around some pretty good ideas that would address missed opportunities, improve the visitors’ experiences, and increase the excellence of the exhibition, but we had to stop to notice a very important point: The purpose of reviewing an exhibition (formally or informally) is not to completely redesign the exhibition or to tear down the work of those who planned it, but to critically observe where the exhibit planners did well and not so well so that we can learn from their experience. These grand schemes of redesign were not for the existing exhibition—to be honest, we all rather enjoyed ourselves and found much to be impressed with. We were using the existing exhibition as a shared reference. The ideas and recommendations were for future exhibitions we would work on. This is how we build on the past and learn from each other.

So far, this has been an exhilarating experiment. Everyone involved has found opportunity to learn from the experiences and insight of our colleagues. But even more exciting has been the opportunity to be guided by the Framework to pull lessons about what makes a truly great exhibition from each and every exhibition we visit, review, and discuss.

I am enjoying the involvement of different perspectives, learning from colleagues, and stretching my own thinking. I recommend that others try it – develop your own group in your own city, and use the Framework to focus and direct your discussions. I would love to hear how it goes.

Framework: Assessing Excellence in Exhibitions from a Visitor-Centered Perspective, by Beverly Serrell (PDF)

NAME (the National Association for Museum Exhibition) is seeking article proposals for the Spring 2011 issue of its journal, Exhibitionist, on the theme: is “Is It a Museum?” Editor Gretchen Jennings writes: “We are soliciting articles about museum-like institutions that call themselves museums but may or may not fit the definitions that AAM or ICOM provides. For example, the various creationist museums around the United States. What about museums at corporate headquarters? Disney? Besides the definitions provided by AAM and ICOM, are there other criteria that should apply? Does it matter if a place calls itself a museum but doesn’t fit the criteria?”

Articles are generally 2,500-3,000 words in length. To propose an article, send one or two paragraphs about how you would approach the topic. Editorial advisors review proposals and notify prospective authors by late September or early October. First drafts are due November 19 and final versions January 15. Contact: Gretchen Jennings, Editor, NAME .

Most of us in the exhibition field can’t stay away from museums, and quite a few reviews on this site were posted after holiday travels—like Eric Siegel’s review of Mind, which he contributed after his family visited the Exploratorium three years ago.

If you’re about to go on holiday, consider bringing back a review to share with your colleagues. When we surveyed members of the site earlier this year, we asked what kinds of reviews you’d like to see. There were nearly a hundred requests, from topics (like wetlands) to types of exhibitions (like those using green materials) to specific places. Anyone going to Te Papa? For those who’ve shared your experiences with family and friends (and your photos), many thanks.

ExhibitFiles members  Mike Levad and Chris Lee have worked for both museums and commercial exhibit firms and are now colleagues at Split Rock Studies in Minnesota. In this guest post, they reflect on the pros and cons of working on “both sides of the fence.”  (That’s Mike on the left.)

Hey Chris,

Having worked for museums for most of my career and now having worked for an exhibit design/build firm for the last two years I am noticing that there are some distinct things that I both miss about working for a museum and love about working for a firm. I was wondering if you had any thoughts about the matter now that you too are working for the other team?

Chris:

I have. My career has been pretty evenly split between the two worlds and I’ve had a chance to see the best in both. I love working on the consultant side of the equation, I thrive in the variety of the work. But at the same time I miss being able to walk an exhibit floor and see how the things I’ve worked on are being used every day. I miss the instinctive sense of what will work in a specific space —and what won’t— that develops after you’ve changed it out a few times. And I miss having a constant supply of visitors just a few steps away that makes prototype testing much easier.

Maybe it all boils down to one thing—missing visitors. Every exhibitionist craves an audience, after all. Consultants don’t often have the opportunity to even see their finished work, let alone see how visitors use the exhibit over time. They compensate for that in several ways; taking pictures, reading summative evaluation reports, and seeking feedback from clients on how everything is working. But I do miss the shortened route to better exhibits that day-to-day involvement with visitors provided.

What do you say we fix that by throwing open the doors of the studio to the public? We could charge admission!

Mike:
I miss visitors too. Working at a museum, your clients are your visitors. I liked helping the nervous father of a three year old find the bathrooms before something bad happened. I always enjoyed blowing something up for 300 screaming camp-in kids.

As an outside developer, your clients are the museum or park staff and the have very different needs than museum visitors. First off, they tend to know where their bathrooms are and generally explosions in the conference room are frowned upon. Secondly, as professionals they come to the table with a large amount of passion and knowledge about their topic and are hoping you can make the most of those two things within their often limited resources.

Chris:
Limited resources are the dark, shadowy figure always lurking in the corner. I like to problem solve, mind you, maybe more than anything else—it’s how my brain is wired (in fact, I’ll go out of my way to make problems just to have something to solve!), and value engineering is problem solving at its most applied. But it’s always frustrating to see all the good ideas that wind up on the cutting room floor for the want of money. On the other hand, fortunately, some bad ideas wind up there too.

Mike:

One big lesson that I have learned “the hard way” is that our clients are customers and not colleagues. When I was an in-house developer passionate—and sometimes rather loud—“discussions” about exhibit content and methodology were seen as an integral part of the process. Working for a firm has forced me to have a calmer, more reasoned, and frankly, a more professional collaborative approach when tough decisions need to be made.

Chris:

Perhaps you were just ahead of your time and the competition-of-ideas model seems set to become a trend. All kidding aside, one of the interesting things about consulting is that you get to experience the trends that sweep through the museum world in something other than an academic way. In part because as a consultant you get to play a small role in how those trends play out in individual institutions. On the museum side those trends (and sometimes buzzwords) can occasionally turn into a course of action that may not be a great fit with that particular venue. When we’re lucky any mismatches between message, delivery, and money become clear and we get to re-direct those resources somewhere better.

Mike:

Being an exhibit developer is the perfect job for me because I love learning new things. I am great at cocktail parties because I know a lot of useless information. The level of content diversity is a major difference in working at a firm. Not that I did not have the opportunity to work on a variety of subjects as an in-house developer, but it was nothing like working as a consultant. In my two years at Split Rock Studios I have developed exhibits about shipwrecks in the Great Lakes, a California slough, the loess hills of Iowa, a Michigan fen, and the environmental history of water in central Minnesota. As a former science museum staffer and self-described science nerd, I may read blogs about organic light emitting diode displays, but I tend not to seek out articles about schooners going down in Lake Huron in my spare time. Developing exhibits for these topics was a rewarding experience because I not only had to learn about these very specific stories, but then I had to create ways to make an array of engaging visitor-focused experiences to bring the stories to life.

Besides working on new topics, I am learning different exhibition techniques than the ones I used working in a science center. Smaller museums and nature centers require us to create more durable exhibit components because the sites generally don’t have any exhibit maintenance staff. At first I saw the lack of electro-mechanical interactivity as a serious limitation. Now I view it as a creative constraint that can lead to a more innovative solution.

Chris:

The last point you made is interesting because it points to one of the biggest benefits of there even being a consultosphere—cross pollination. Collection/location-based museums and idea-based science centers are fairly different worlds. Consultants often move back and forth between them and can bring over methods, materials, and even concepts that improve both. So it’s not just project variety that makes consulting rewarding (though I am a huge novelty junkie), but also being able to foster those kinds of exchanges.

Mike:

What about the benefit of working on a topic more than once? In a museum you probably will only do an exhibit on a given topic one time. Working for a firm one develops a repertoire of successful delivery devices that can be used in different ways depending on the content. When you are working on your third or fourth nature center and they want to do a food-web device you have a variety of methods to choose from. The animals and their connections might vary from region to region but the way visitors interact with the content is similar. This does not mean that we are using off–the-shelf solutions but rather adapting known mechanisms to best match the content.

Chris:

What’s interesting to me about that is how we can still create a unique experience even though some elements in an exhibit may be repeated. I think that in some cases a recognizable delivery system, like a flip label, can help people begin to navigate and feel comfortable with an otherwise unfamiliar environment. In fact, if we don’t provide some familiar delivery devices I think we actually run the risk of frustrating visitors. In part this is because visitors are bringing expectations that are formed over time from their previous interactions with similar devices. For example, how many times have you seen someone tapping the glass of a computer monitor that wasn’t actually a touch screen?

Mike:

Ah, yes. I believe I’ve actually done that once or twice.

Another benefit I’ve noticed working at a smaller organization such as a design firm is that I am able to go directly to the CEO with any concerns or opportunities. When I worked for a large science center there were two layers of management between the CEO and me.

Chris:

I agree that a flatter organization is more nimble and more open to opportunities, especially ones that come from unexpected places. Those layers serve some purposes, of course, but just as often they get in the way of you and the boss knowing what each other thinks. As contractors though, we have the luxury of being free to focus on just the exhibits we’re developing. Museum departments and staff have to wade through many competing internal demands on their resources. That makes it challenging for them to develop new exhibits at the pace they would prefer.

Mike:

Finally, along with a much flatter structure, one thing I appreciate about this firm in particular is the level of transparency in the way things are done. I can look up the job tracking information on any job we have ever done. This helps us learn from our experiences and eventually serve our customers better. The most important thing is to remember, no matter if it is designed by in-house staff or a firm, every exhibit should feed the “muse” and inspire the visitors that we are so lucky to serve.

This exchange first appeared in the March 2009 issue of Legacy magazine and is shared here with their permission.