Archive for the 'User Assessment' Category

What difference does an exhibition make?

Friday, August 1st, 2008 by Wendy Pollock

We were happy to see two new ExhibitFiles case studies posted by participants in the PI Summit, held July 25-26 in Washington, D.C. Liza Pryor of the Science Museum of Minnesota wrote about Science Buzz, and Elizabeth Fleming of the Museum of Life and Science in Durham, North Carolina, wrote about Flip It, Fold It, Figure It Out. Like a number of other exhibitions described in earlier case studies, both were funded by the National Science Foundation (NSF), a U.S. federal agency.

As anyone in the informal science education field with NSF funding now knows, documenting the impact of the work is an increasingly high priority. NSF issued a Framework for Assessing the Impact of Informal Science Education Projects (PDF) earlier this year to guide grantees, and future proposals will need to address impacts laid out in this publication.

But defining intended impacts can be a challenge in the rich and multifacted world of informal, lifelong learning, and assessing whether or not we’ve achieved these impacts can be even tougher. In a later reflection added as a comment about her case study, Liza notes one challenge she faces in assessing the impact of Science Buzz: “We’re working on our summative evaluation,” she says, “but we don’t have anything to compare our data TO. We’ve got the data from the Pew internet study, but it’s not too helpful. I’m particularly interested in studies of online communities. What’s a decent participation rate? Any way, without resorting to discourse analysis, to figure out what people are learning?”

Maybe this online community can give Liza some help. In fact, we’ll be posting soon about how we’re thinking about this in relation to ExhibitFiles itself.

Testing usability

Wednesday, April 18th, 2007 by Wendy Pollock

We were at Museums & the Web in San Francisco last week, getting feedback on ExhibitFiles in the usability lab. Many thanks to Michael Twidale and Paul Marty, who led the process, and Richard Urban, who played the role of user. We’re taking their comments into account in some last-minute adjustments we’re making to the beta site before opening next week. Meanwhile, the core group that’s been advising on development has started setting up accounts and posting case studies and reviews (and reporting bugs), so new users will be able to explore before jumping in. We already have a mix of old and new, large and small, funky and classic, science and art, with more in the works. We hope to see you on the site, Monday, April 23, when ExhibitFiles goes live.

April 23 ExhibitFiles BETA launch

Thursday, March 22nd, 2007 by Wendy Pollock

We’ve set the date. On April 23rd the ExhibitFiles website will be publicly available. Over the last nine months, we’ve shared part of the process with you via this blog. Now, in a month, you’ll be able to see (and participate in) ExhibitFiles for yourself. We appreciate all of the comments and suggestions we’ve received during this process; they have helped us improve the site’s design and functionality. Randi Korn & Associates will be carrying out a formal evaluation of the site during the summer, but meanwhile we hope you’ll continue to send your comments.
Once ExhibitFiles launches, we’ll continue to publish news here for the first few months as we consider whether and how to continue the blog. Meanwhile, we look forward to seeing you online in April!

East Coast Design Meeting

Wednesday, March 7th, 2007 by Jim Spadaccini

Site Structure 4.0
It’s been a while since we’ve posted anything on the blog, but we wanted to bring you up-to-date on our design process. In mid-February, we met at ASTC headquarters in Washington for a day-long meeting to review the ALPHA site.

Our core team has been working with the ALPHA build for the last few weeks, and their comments have helped us fine tune the structure of the site as work toward BETA. (The latest site diagram above is available as a PDF.) One of our main concerns has been to make this site as responsive to the needs and ways of working of exhibit practitioners — including time constraints. So we’ve been simplying and clarifying while working out bugs. Work has begun on the final phase which will lead to a BETA release. We’ll be publishing that release date soon.

ExhibitFiles structure emerges

Monday, October 23rd, 2006 by Jim Spadaccini

This site diagram (site diagram v2 PDF) illustrates the relationship between the various components of the proposed ExhibitFiles site. Since one of the key functions of the site is to “include shared records of exhibition descriptions,” the ability to add, browse, and search these records is a major part of the site structure. So far the design process has focused on the elements that make up these individual records. (See blog posts, “Personal Profiles” and “Case Studies and Reviews“).

Beyond the specifics, our discussions have focused on what is the “right amount” of information. Too much and users won’t want to add case studies or reviews, too little and we might be omitting important information. Obviously, we need to meet the needs of the members of the ExhibitFiles site. A major and ongoing concern has been how can we best encourage active participation.

Wendy’s last post, Tapping the wisdom of the crowd, explained in detail the concept of drawing ISEN-ASTC-L messages directly into the ExhibitFiles site. This is not the only place that ExhibitFiles members will have opportunities to converse. Comments (and ratings) can be added to any case study or review. Trying to anticipate conversation in a complex social site is difficult, so we intend to see how things go and make some adjustments as the site is launched and these conversations begin.

In our Berkeley design meeting, the issue of “ratings” was discussed and, for the most part, participants were concerned that employing such a system could trivialize contributions to the site. It was agreed that rating systems such as the starred Yahoo! ratings for news stories would not be appropriate in this environment.

As an alternative, we looked to favorites as way to sort reviews, case studies, and other content on the ExhibitFiles site. As members add items to their own favorites, a record of that addition will be associated with that item, so reviews or case studies that are favorited by many users could be presented in some way (e.g., most favorited). Also, each member’s list of favorites can be useful for others, in the same way the del.icio.us bookmarks can be made available to all members.

As our discussion progressed, we revisited the concept of “ratings.” This was due in part to addition of the ISEN-ASTC-L component. Having a way for ExhibitFiles members to help sort the messages that are most relevant to the site is essential, since not all of the messages on the Listserv are directly related to exhibit design. The example of Digg in which visitors give a simple thumbs up or thumbs down to content items as way to sort was discussed, as was Amazon’s Was this review helpful to you? Yes or No feature.

We’re looking to add a similar feature: “Did you find this useful? Yes or No.” This simple Yes/No rating along with favorites will help “score” the Listserv messages as well as case studies and reviews. While ratings won’t be the only way to browse (and search?) the ExhibitFiles content, having some methods for letting the community decide what information is most useful is important. We hope to have a large collection of records and conversations about exhibit design, so community members will need various methods to find the materials that are important to them.

After months of discussion and review we’re finally getting to the point of building out components of the ExhibitFiles site. While the pace will pick up dramatically over the next few weeks, we’re still very interested in your comments and ideas. The current push is for our first build to be ready in January 2007. We will be taking a look at the site again in late spring once the first members begin to contribute and use the site. As we design and develop elements for the ExhibitFiles, we’ll be posting page grids and other diagrams on this blog. As always, we welcome your comments and questions.

Personal Profiles

Monday, August 28th, 2006 by Jim Spadaccini

Since ExhibitFiles will be a community-built website, it was decided early on to require a membership (it will be free) for authoring and commenting. Since our design workshop, we’ve been actively exploring how we might structure “personal profiles” within the ExhibitFiles community.Based on the comments from the design workshop and our group of advisors, we’ve outlined the following elements as part of ExhibitFiles profile:—-NameWhat I do, where I workWhat I’ve done in the past (multiple-no more than five)City, State, CountryEmailExhibitFiles password (repeat password)Website(s) (multiple-no more than three)More about me (text area)My image (upload a JPEG or PNG)Resume and/or portfolio (upload a PDF, doc, or URL)What I’ve added to ExhibitFiles (automatically populates once a user adds content to the site)Things I’ve bookmarked in ExhibitFiles (automatically populates once a user bookmarks files in the site)Terms of Use, Privacy Policy, Email Use Agreement (check box)—–The purpose of the profiles is to allow ExhibitFile members to connect with each other, but also to learn from one another. We’re hoping that the “What I’ve added to ExhibitFiles” and “Things I’ve bookmarked in ExhibitFiles” will provide additional ways to access the “records” (the content of the ExhibitFiles site).Finding a balance between enough information and too much is an ongoing challenge in planning for the site. Too much and no one will want to fill in lengthy forms, not enough and we may be leaving out valuable information.The profiles (and language used) are intentionally informal and brief. We’re hoping that ExhibitFiles users will post additional information (resumes and porfolios) and/or link to additional sources using the “Website(s)” element. We’re not looking to recreate LinkedIn (who were part of our Competitive Analysis), but rather to create profiles that have a balance between professional in and personal information.As always, if you have any comments or suggestions, please feel free to post them here.

Front-end study: a commitment to the common good

Monday, August 14th, 2006 by Wendy Pollock

Exhibits people have been giving generously of their time to advise about development of the ExhibitFiles site. Earlier this year, 17 of them spoke with interviewers from Randi Korn & Associates about the exhibit development process, the resources they draw on in their work, and what they’d find useful in this new site.

Most striking to me is their extraordinary generosity of spirit. Most said they would be willing to share descriptions, photos, evaluations, lessons learned, final budgets, and other material. The reasons for their willingness to share: a commitment to collegiality and collaboration and, to quote the report, “a desire to build a learning community and avoid reinventing the wheel.” We look forward to hearing from others about what they would find useful, and ways they hope to contribute.

Front-End Study (PDF 163K)

Design Workshop

Wednesday, July 26th, 2006 by Jim Spadaccini

As part of the ExhibitFiles design process, we conducted a workshop at the Lawrence Hall of Science, Berkeley, California, June 13-14, 2006. Most of the workshop participants were exhibit developers, the primary users of the website.

We focused on potential features of the ExhibitFiles site. Card sorting, free-listing, and a lot of discussion over the two days really helped move our thinking along.

We’ve captured some of the results from this meeting in a Design Workshop Document (880K PDF). As always, your comments and questions are welcomed.

Learning from the ISEN-ASTC-L list

Friday, July 14th, 2006 by Wendy Pollock

The ISEN-ASTC-L email list is one of the resources science center and museum people have been using over the last 10 years to share information and offer mutual support. The list now has 1,330+ subscribers in more than 20 countries (most in the United States and Canada). To help us understand what kinds of questions come up around exhibits, we asked Randi Korn & Associates to analyze postings from a six-month period. We wanted to make sure we’re taking into account the real problems people run into in their work–not just whether or not someone has produced an exhibition about a topic like infrared radiation (to use a recent example), but more technical issues like inexpensive alternatives to heat-imaging cameras. We’re using this analysis as we specify content requirements for the exhibition records that will serve as the core of the ExhibitFiles site.

Analysis of ISEN-ASTC-L postings PDF

ISEN-ASTC-L: subscribe and search archives

User Needs Assessment Summary

Friday, July 7th, 2006 by Wendy Pollock

Early planning for the ExhibitFiles site was guided by conversations with colleagues, brief surveys, and analysis of listserv records. Results are summarized here:

User Needs Assessment Summary PDF