Home > Website of the Week Archive Detail

 << Prev 9/30/2007 Next >>

## Hark...I Hear A Mathematical Function Sing!

Ever hear about "Sonification"...no pun intended. It uses non-speech audio (i.e. tones) to "convey information or perceptualize data," serving as a viable alternative to visualization techniques. It is not a new technology, as auditory displays are already in common use in situations that require a constant awareness of some information (e.g. vital body functions during an operation)...or even a Geiger counter for measuring radiation.

Current researchers are investigating uses of sonification to help the visually-impaired in diverse ways. One example is image sonification, which is the basis of The vOICe (emphasize "Oh I See"). Developed by Peter Meijer, a Dutch physicist, vOICe plays the sound of visual input from digital cameras, eyeglasses, or mobile phones. As a vertical line scans the image, the image's features are turned into sound by representing vertical position as pitch, horizontal position as time within the sweep, and brightness as volume.

Admittedly, the user must learn how to interpret the sounds, even for simple geometrical objects. But, after practice, users claim they can experience depth in the 2-D images.

The above introduction is a long way around to inviting you and your students to visit a vOICe website. Things to do there:

• Play with the Java demonstration of sounds of visuals that you draw. For example, draw geometrical objects such as a square, a triangle (equilateral, scalene, rotated, etc.), or even a 3-D visualization of a cube (with sides shaded in). Draw, hide, and then play a shape...see if students can learn to use sound to identify the shapes.
• Play with the sounds of a graph for a (x,y) function. Listen to the tones, where time is the x-axis and pitch is the y-axis. Note: You also can do this with either Mathematica (using Play [f,{t,0,tmax}]) or Matlab (using (y,Fs) where y is the function vector and Fs is an optional parameter for sample frequency).
• Play with the musical sounds generated by a Bach motif, an image of Sierpinski's gasket, or even John Conway's Game of Life.
A Caveat: Though these sounds are of value to a visually-impaired user, more information is needed. For example, shifts in a function (i.e. f(x) vs. f(x)+10) may sound the same, as no base-line axis values are established....making a nice analogy to derivations of indefinite integrals (Ye Olde "+c" phenomena).

Finally, read Ray Girvan's article, which gives a more complete description of sonification. Focus on its uses in music (e.g. Sonification Sandbox) and steganography. For example, like people who play songs backwards, someone discovered a demonic face in the spectrograph of Aphex Twin's song Windowlicker...but when the "sound view" was adjusted using a logarithmic scale, it was merely an intentional portrait of Aphex Twin himself. Other artists mentioned by Girvan include John Dunn/Mary Ann Clark's Life Music (sonification of protein data), Bob Sturm's Music from the Ocean (sonification of water waves), and Marty Quinn's Seismic Sonata (sonification of the 1994 Northridge California earthquake). And finally, approaching science fiction, read about the use of sonification as "human augmentation." An example is Leonard Foner's Visor (part of a wearable computer) that sonifies radiation detected by a head-mounted spectrometer that replicates human vision....and can see through camouflage.