Archive for the 'evaluation' Category

Bilingual Exhibits Research Initiative

Friday, August 17th, 2012 by ExhibitFiles

Nan Renner, Cecilia Garibay, Carlos Plaza, and Steven Yalowitz describe the Bilingual Exhibits Research Initiative, a National Science Foundation Pathways Exploratory Research Project

Does your institution create multilingual exhibits? Do you wonder about how multilingual exhibits may influence engagement and learning?

The Bilingual Exhibits Research Initiative (BERI), NSF DRL#1010666, strives to address professionals’ questions and build our collective knowledge related to:

• How informal science institutions create bilingual exhibits;
• How Spanish-speaking and bilingual visitors use bilingual exhibits; and
• How bilingual exhibits can support engagement with science, technology, engineering, and mathematics (STEM) among Latinos in the US.

Bilingual Exhibits—Professional Practices

The BERI team conducted interviews with professionals at 22 U.S. informal science education (ISE) institutions to document current bilingual exhibit practices.
This effort builds on a survey conducted by ASTC and the Exploratorium in 2011.Bilingual  study participants look at and point to a map within a bilingual  exhibit at the San Diego Natural History Museum. Photo shot with a  SenseCam worn by a study participant.

Research questions focus on staff knowledge, beliefs, and practices:
• Who is the audience for bilingual exhibits?
• What is the form and content of bilingual exhibits?
• What is the process for creating bilingual exhibits?
• How do visitors interact with bilingual exhibits?  (as observed and hypothesized)

Publication of these research results is in progress.

Many institutions believe that providing Spanish text increases engagement among Spanish-speakers, although we have very little information about how visitors actually use bilingual exhibits and the resulting benefits.

ISE professionals’ questions about audiences and bilingual exhibits have helped to shape the research agenda with visitors, e.g., Who uses bilingual exhibits, and how? Do bilingual exhibits create visual or mental overload? How much Spanish text is enough? Will bilingual exhibits encourage attendance?

Bilingual Exhibits—Visitor Uses and Benefits

Observations and interviews with Spanish-speaking visitors in social groups will document how visitors use exhibit resources and how they perceive the benefits of bilingual exhibits.

A  bilingual visitor group interacts with geology exhibits while  researchers record observations. Low resolution photo shot with a  SenseCam worn by a researcher.

Research questions focus on social activity involving language (speaking and reading):

• How do individuals and groups engage with text in bilingual exhibits?
• What indicators of learning can be observed?
• How do patterns of engagement and learning correspond with exhibit features?
• How do visitors perceive the benefits of bilingual exhibits?

You can participate! Help create an online archive of bilingual exhibits by adding your reviews, case studies, and “bits”  to ExhibitFiles. Don’t forget to tag your post with “bilingual.”

Post a comment! What are your questions, concerns, and conundrums related to bilingual exhibits? What have you learned about creating bilingual exhibits? Are you engaged in cross cultural visitor studies in museums?

Bilingual visitor groups use both languages when reading and speaking. Photographic data from a SenseCam worm by a study participant shows a rock with an embedded fossil, multi-party touching, and pointing to English text (left) and Spanish text (right). In analysis, the research team will integrate data fro audio recordings, live observations, SenseCam images, and activity-in-context photographs.

Research sites include:

San Diego Natural History Museum, the Miami Science Museum, and two other ISE institutions chosen to represent diversity of content, geographic region, and Latino cultural groups. We began data collection in June 2012 and will complete in fall 2012. This Pathways research project will be completed in June 2013.

This exploratory research, funded by NSF’s Division of Research on Learning, will build knowledge about how informal science institutions create bilingual exhibits, how visitors use bilingual exhibits, and how bilingual exhibits may expand access to science learning for Latinos in U.S. science centers and museums.

Image information

Top: Bilingual  study participants look at and point to a map within a bilingual  exhibit at the San Diego Natural History Museum. Photo shot with a  SenseCam worn by a study participant.

Center: A bilingual visitor group interacts with geology exhibits while researchers record observations. Low resolution photo shot with a SenseCam worn by a researcher.

Bottom: Bilingual visitor groups use both languages when reading and speaking. Photographic data from a SenseCam worm by a study participant shows a rock with an embedded fossil, multi-party touching, and pointing to English text (left) and Spanish text (right). In analysis, the research team will integrate data fro audio recordings, live observations, SenseCam images, and activity-in-context photographs.

ExhibitFiles at the Visitor Studies Association conference

Tuesday, July 19th, 2011 by Wendy Pollock

Coming to Chicago next week for the Visitor Studies Association conference? Get an update about evaluation of ExhibitFiles during a session on Tuesday, July 26, 9:00-10:15 a.m., when Carey Tisdal and I will be talking about Knowledge, Identity, and Networks in the Informal Learning Community. There are many other sessions on the program which should also be useful to exhibit practitioners. Find out more here.

If you contributed to the ExhibitFiles evaluation by responding to a survey or agreeing to be interviewed about your perspective, we truly appreciate your help. You can find the report of the remedial evaluation on the InformalScience website. Carey will also post results of the summative evaluation that is now underway.

How to make exhibitions catalysts for group interaction

Tuesday, July 12th, 2011 by ExhibitFiles

Minda BorunIn this guest post, Minda Borun, long-time director of research and evaluation at Philadelphia’s Franklin Institute, draws on earlier reports about the NSF-funded Philadelphia/Camden Informal Science Education Collaborative (PISEC). While dating back more than 10 years, this work is as relevant as ever for those designing exhibitions that stimulate the conversations we know are an important part of learning.

Exhibits are catalysts for encouraging group interaction. People bring their personal histories to their encounters with exhibits, they talk with one another – and that is how learning happens. We know this intuitively – and over the years, we’ve also gathered considerable evidence about characteristics of exhibition design that support and encourage this outcome.

In 1992, the Philadelphia/Camden Informal Science Education Collaborative (PISEC) started working on the Family Science Learning Project (NSF/ESI #9355504) to systematically test and refine our understanding of the learning behavior of the visiting unit characteristic of most museums: the small group. We identified seven characteristics of exhibits that our observations and collective experience suggested were associated with what we called “family-friendly” exhibits.

  • Multi-Sided  - Family can cluster around the exhibit
  • Multi-User - Interaction allows for several sets of hands or bodies
  • Accessible - Comfortably used by children and adults
  • Multi-Outcome - Observation and interactions are sufficiently complex to foster group discussion
  • Multi-Modal - Appeals to different learning styles and levels of knowledge
  • Readable - Text is arranged in easily understood segments
  • Relevant - Provides cognitive links to visitors’ existing knowledge and experience

Mechanics Maze, one of the "family friendly" exhibits in Kid ScienceThese features apply to any small group, not just families. The model has also been found to apply to programs by substituting “comprehensible” for “readable.”

Each of the PISEC partner museums – The Franklin Institute, Academy of Natural Sciences, New Jersey State Aquarium (now called the New Jersey Academy for Aquatic Sciences), and Philadelphia Zoo – added a new component that embodied these characteristics to an existing exhibit. Our results were impressive: In each museum, we saw a measurable increase in active family learning. After the publication of the results of this study (Borun et al., 1998), the PISEC museums and others went on to use these “seven characteristics of family-friendly exhibits) in the creation of new exhibitions. At the Franklin Institute, for example, the seven characteristics guided design of Kid Science (above), an exhibition for families with children ages five to eight, which opened in 2001. Prototypes of the interactives were extensively tested to be certain that they appealed to this age group and communicated their messages – and that design reflected those characteristics we knew were likely to be catalysts for group interaction. In our summative evaluation, we found that Kid Science engaged families longer and more actively than any other exhibit in the museum.

Sea Cave exhibit at Lookout Cove, Bay Area Discovery MuseumOther museums – like the Bay Area Discovery Museum in Sausalito – have also applied these principles.  Justine Roberts wrote about one of these exhibitions in her ExhibitFiles case study of Lookout Cove (left).

The PISEC group has since gone on to extend our experimentation to programs, including museum/community partnerships for underserved families (reported in our latest publication, In Their Own Voices: Museums and Communities Changing Lives).

It is important for museums to be facilitators of family exchange and group learning, not obstacles. PISEC’s findings offer important insights and guidance for those designing new exhibits.

Find out more

  • Borun, M., Cleghorn, A., and Garfield, C. (1995). Family learning in museums: A bibliographic review. Curator, 38(4), 262–270.
  • Borun, M., Chambers, M., and Cleghorn, A. (1996). Families are learning in science museums. Curator, 39(2), 124–138.
  • Borun, M., Chambers, M., Dritsas, J., and Johnson, J. (1997). Enhancing family learning through exhibits. Curator, 40(4), 279–295.
  • Borun, M., & Dritsas, J. (1997). Developing family-friendly exhibits. Curator, 40(3), 178–196.
  • Borun, M., Dritsas, J., Johnson, J., Peter, N. E., Wagner, K., Fadigan, K., Jangaard, A., Stroup, E., and Wenger, A. (1998). Family learning in museums: The PISEC perspective. Washington D.C.: The Association of Science Technology Centers.
  • Borun, Minda. (2002). Object-Based Learning and Family Groups in Scott Paris (ed.) Perspectives on Object-Centered Learning in Museums, Lawrence Erlbaum, Associates, Mahwah, New Jersey
  • Borun, Minda, Barbara Martin Kelly, Lisa Jo Rudy, (2011). In Their Own Voices: Museums and Communities Changing Lives, Philadelphia, PA: The Franklin Institute.

What’s evaluation good for?

Wednesday, January 5th, 2011 by Wendy Pollock

Exhibits people sometimes wonder whether evaluation is really worth the time, effort, and expenseor whether it’s just something funders require that they’d rather do without. Charlie Carlson sparked a round of discussion on the ISEN-ASTC listserv this week with a series of questions that arose from his many years as exhibit developer, and Alan Friedman was among those who responded. As our science museum colleagues know, Alan was not only the editor of NSF’s Framework for Evaluating Impacts of Informal Science Education Projects, published in 2008, but he also wrote a section in a 1991 book called Try It! Improving Exhibits Through Formative Evaluation that offered a rationale for why museum directors should  support evaluation. Here’s their exchange (reproduced here with their permission):

Charlie: For some years I’ve wondered about the efficacy of exhibit evaluation, wondered whether or not it is useful, or more directly a bureaucratic hurdle that provides useless and specious validation that satisfies an inner need and social, political need to feel affective.

Alan: Sounds like you are talking about summative evaluation.  Would you put formative evaluation in that same bucket?  I have dozens of personal experiences in which a few hours of formative evaluation told me what visitors totally misunderstood, or what they got right away, or some other revelation.  The resulting exhibitions were dramatically better than they would have been without the formative evaluation.  I’ve written up a summary of four examples from my earlier years in the field [PDF], but I’m sure every exhibition developer can add many more. I think front-end evaluation has equally dramatic impacts in many cases.

Charlie: To put it bluntly: Are museums and taxpayers spending a significant amount of money on something of questionable value?

Alan: Some evaluation is poorly done, and/or doesn’t show anything interesting.  The same can be said of money spent on legislation, classroom lessons, movies, books, TV shows, and even—scientific research!  The challenge is to improve the practice, and I think we are getting better, judging by the dozens of reports of careful, useful, and enlightening evaluations I hear at every Visitor Studies Association meeting.

Charlie: Museum visits are, indeed, events, fraught with every personal and social dimension.

Alan: Yes!  And many excellent evaluation studies have shown just how the personal and social dimensions can hinder or help a visitor’s experience.  Again many examples, but the experiments with creating “family size” spaces around individual exhibit units (Explora, Exploratorium), are examples.

Charlie: As such they are part of noise and chatter of day to day existence. Importantly, museum visits are also brief—ever so brief, hours out of a year (some fraction of 6570 waking hours annually).

Alan: See Falk and Dierking’s passionate article, “The 95% Solution,” published a couple of months ago in American Scientist.

Charlie: On the face of it, anyone funding an exhibit needs to know that they’re getting value for their commitment of resources, and more broadly whether or not it is having an intended effect.

Alan: That’s one reason, but not the major one.  At the moment I am on the boards of three foundations which fund science education, in and out of school, among other things.  We ask for evaluation because we want to become smarter funders.  And we learn from the evaluations of just about every project we support.  We learn a lot even if the evaluation shows no impact.  Because then we look at the arguments we bought when we decided to make the grant, and now we know what questions to ask the next time to help applicants prepare better proposals.

Charlie: What are the key concepts that characterize an excellent exhibit or museum?

Alan: Great question, and now you have taken us out of evaluation and into the area of research.  There has been very little research on these questions (but I see others are citing what does exist.)  I’ve written up my summary of research and meta-studies (PDF).

Charlie: Is there evidence that evaluation has improved or positively modified an exhibit or exhibition?  I think the evidence is scant.

Alan: Please see my first response, above.

Charlie: How much do people generally remember of a museum visit?

Alan: Again, there is such research, some of it going back many decades.  See work by David Anderson and Harris Shettel.

Charlie: Do the specifics of an exhibition make a difference in human behavior?  Probably not for most people.

Alan: Having to say “probably not” is just the sort of weak argument that research and evaluation can help avoid.  We can measure, and yes, some specifics do make a big difference.  See response above on formative evaluation.

Charlie: Has a museum exhibit changed the course of human history?  Probably not!

Alan: There’s “probably not” again.  But there is evidence to help answer that question.  See Steven Dubin’s book Displays of Power, and studies of early influences on the lives of people who become scientists (cited in my summary of impact studies, above).  Some exhibitions have spurred consideration of issues that might have been ignored, and changed lives and careers along the way.  I am confident in saying that museum exhibits have changed individual human lives, including my own (see Lessons from an English Summer).

Resolved to improve exhibit evaluation in 2011?

Thursday, December 30th, 2010 by Wendy Pollock

For exhibition colleagues resolved to pay more attention to evaluation during the coming year, here’s a resource from the American Evaluation Association (AEA).

AEA365: A Tip-a-Day By and For Evaluators “is dedicated to highlighting Hot Tips, Cool Tricks, Rad Resources, and Lessons Learned for evaluators” with the goal of featuring “a post a day from and for evaluators around the globe.” You can read it, and you can submit contributions yourself. One of the recent posts was by our own evaluator, Carey Tisdal, on “practical applications of theory.” You can read it here.

Here’s to even better exhibitions in 2011 – and best wishes to ExhibitFiles members and supporters!