Pictures of Sound

Part of Exhibition: Wild Music

Topic: Physical Sciences Subtopic: Vibration & Waves

Case Study

of an Exhibit

by David Bailey

Published on June 25, 2007, Modified on September 11, 2013

  • Description and goals

    One of the toughest challenges we had working on the Wild Music exhibition was adhering to our goal of creating exhibits that were accessible to visually impaired people. Knowing that the primary activity for many of the exhibits would be listening to sounds, it seemed those exhibits would be inherently easy to make accessible. The major hurdles, however, would involve the design and layout of the user interfaces. We did devise a few tactics to help non-seeing people navigate their way around the exhibits to find pushbuttons, Braille labels, audio description handsets, etc. Some things worked out well for visitors. I’ll discuss what did and did not work in another posting. Of course, some of the exhibits we intended to build did not readily lend themselves to being accessible. Pictures of Sound was an inherently challenging exhibit, because of its content.

    Pictures of Sound———————-

    Pictures of Sound introduces the “spectrogram”, a graphic image that shows a 3D plot of the intensity of frequencies that make up a sound, over time. (see picture #1) The really cool thing about spectrograms is that they allow us to see the various frequency components of a complex sound that we can’t extrapolate using our ears alone. Depending on the sound, the information a spectrogram contains can be quite surprising, and visually interesting. For many, however, these images can be a little confusing to understand at first. Since we use spectrogram images throughout the exhibition, we felt we needed to create an exhibit that focused on how to read them.

    More on Spectrograms——————

    Spectrograms, unless you are familiar with them, can seem a little cryptic. First of all, spectrograms are 3D graphs, but they are usually depicted in two dimensions. The placement of frequencies ( vertical axis) over time (horizontal axis) is fairly easy to grasp. It is loosely analogous to reading Western music notation. Higher notes are placed higher on the scale, and lower notes are placed lower on the scale. Of course, the stanza represents the time-line. However since spectrograms are 3D graphs, there is a third dimension to represent, intensity. How is this done in a 2D graphic?

    The intensity of frequencies in a spectrogram is represented either by a color spectrum (warm to cool colors, or vice versa), or a tonal spectrum (black to white, or vice versa). One end of the spectrum represents higher intensities, while the other end represents lower ones. We use both color and tonal spectrogram images throughout the exhibition.

    Can the visitor read spectrograms without seeing them?————-

    The obvious challenge here was to figure out how to get across the notion of what a spectrogram is without necessarily having to use one’s eyes. As part the accessibility research we did for Wild Music, we made a visit to the Minnesota State Services for the Blind office to learn about available tools and techniques we could implement to help visually impaired people navigate the exhibition more easily. During the visit, I was intrigued to see one of their volunteers gluing pieces of string and various textured bits and pieces onto a piece of thin plastic. I asked her what she was doing, and she explained she was converting two-dimensional charts and diagrams from a high school textbook into raised ones that could be touched and interpreted by a blind student. This gave me an idea. What if we created spectrograms that were literally three dimensional, where intensity could be represented by height, instead of being represented by color or tonal spectrums?

    Building The Prototype———————————

    I went to work building a prototype that had two spectrogram images mounted on the tabletop, side by side, for comparison (see pic #2). Each spectrogram had a corresponding audio track that could be heard when the visitor pressed a button. Also, located above each spectrogram was a chaser light that followed along and indicated where on the spectrogram’s time-line the audio playback was. Each spectrogram image was covered with a piece of clear acrylic. On top of the acrylic, I used dimensional paint (a.k.a. Puff Paint) to trace over the information contained in the spectrogram. Since the sound intensity of the frequencies shown in the spectrograms varied a great deal, I decided to limit the intensity scale to three levels (low, medium, high). Each level was not only represented by the height of the paint, but also by the color. The different colors helped seeing people interpret the graph more easily. The idea worked quite well, and Wally Waranka, our blind advisor who worked with us throughout the building of the entire exhibition, was pleased to be able to correlate patterns he felt on the raised surfaces with the audio playback. The dimensional paint worked great as a prototype medium, but of course, there was nothing to prevent the paint from being picked off. Also, achieving any kind of real accuracy in the height of the paint was extremely difficult at best. Assuming we would figure out a way to model the spectrogram intensities in a more durable and accurate way, we proceeded to create a more finished prototype for display in the ASTC booth at the 2005 ASTC convention.

    Testing at ASTC—————————————-

    When we debuted this prototype at the 2005 ASTC convention, we had generally positive feedback from attendees, but unfortunately, I don’t recall a single person with low vision or blindness trying out the exhibit. What did happen, though, was the serendipitous meeting of Steve Landau, research director of New York based Touch Graphics. Steve’s company produces the Talking Tactile Tablet (TTT), an award-winning device that is used in both education and exhibit contexts. Users place one of a large number of raised-line and textured (tactile) overlay sheets on the TTT, and then interact with a computer by pressing down on parts of the tactile map, diagram, or other illustration, to hear descriptive audio relevant to the part of the image where they pressed. Using this system, it is possible to construct many entertaining and educational applications that are fully accessible to individuals with visual impairments (see After visiting the Touch Graphics booth and learning about the TTT, I had Steve come over to the ASTC booth to take a look at our prototype. After a bit of discussion, Steve was confident that the plastic overlay sheets he produces for his TTT system would work for our needs. He also indicated that he could print the original two-dimensional spectrogram image right on top of the raised areas, so that would be readable by sighted people as well. (see photo #3)

    Final Design—————————

    The visitor is able to choose from eight different cards containing spectrogram images on them. The spectrogram cards highlight bird songs, various animal calls, and the human voice. Visitors are instructed (via graphics or the audio description handset) to choose a card, and insert it into a slot on the tabletop slant. It is then read and identified by a computer. When the visitor presses a pushbutton, the computer selects and plays the appropriate audio track. Also, above the spectrogram card is a chaser light that follows along and indicates where on the time-line the audio playback is. Visitors are encouraged to follow along with their hand, so that they can feel the raised portions of the spectrogram image. The 2D rendition of the spectrogram image is also superimposed over the raised areas.

    Lessons Learned——————-

    We’ve just ended the three month long debut run of Wild Music here at SMM. It will begin its second venue at the North Carolina Museum of Natural Sciences in Raleigh at the end this month (June 07). Presently, the final evaluation data for Wild Music is still being compiled. Once this data is available, I will post an update regarding the overall visitor feedback on Pictures of Sound.

  • Exhibit Opened: March 2007

  • Location: Saint Paul, None, United States

  • Estimated Cost: $10,000 to $50,000 (US)

  • Website(s):

Latest Comments (1)

great story

by Kathleen Mclean - June 27, 2007

Thanks David for this really thoughtful and descriptive case study of the Pictures of Sound exhibit. It really describes the complexity of the process and the importance of being focused on outcomes with users. I’m going to get everyone on my current exhibition team to read it and talk about it. K

Log in to post a response.