This is what data sounds like
When Penn State music technology professor Mark Ballora was given his first synthesizer in the early 1980s, a connection with the digital realm of music was fused.
Almost 25 years later, that relationship is helping to score music using data created by occurrences such as an earthquake or even the Big Bang.
The technique is called sonification, and Ballora is using his musical background to steer the technique into science communication, which he hopes can inspire young scientists to be the innovators of the future.
Ballora grew up in Mill Valley, Calif., which is just across the Golden Gate Bridge from San Francisco. He studied theater arts at the University of California, Los Angeles.
After he graduated in 1984, he headed to New York, where he worked on Wall Street using “fancy word processors” to create charts for bankers.
During his time on Wall Street in the ’80s, he began composing scores for modern dance routines and honing his music writing skills. That was when Ballora discovered that he could connect his synthesizer to his Apple Macintosh computer and use a musical instrument digital interface to record, edit and play back music.
He bought the gear, connected his synthesizer and fully embraced the digital world. He also discovered that recording went beyond music.
“I used to go around my East Village apartment sampling things like the door creaking, or I’d make a lot of oatmeal and squish through it with my hands to get a squishy, oozy sound,” Ballora said. “I immediately loved coming up with stuff like that.”
While exploring the medium, Ballora was also attending graduate school at New York University. At the same time, Leon Glass, a physiologist at McGill University in Montreal, Canada, was attempting to make sound using heart rate variability data sets. Glass reached out to Ballora and asked him to help.
“I said yes. Then I asked what it was all about,” Ballora said. “It just seemed too cool not to do, so I changed my thesis to sonification.”
What is sonififcation?
When analyzing the digital data generated by the sound wave of occurrences, such as an earthquake or a celestial event, scientists can use the set of numbers that outlines the waveform and listen to a literal sound of that event.
“So here’s an acoustic wave traveling through the Earth and the thought is, ‘Hey why not take that sequence of numbers and wrap it in an audio file header and you can play it like an audio file,’ ” Ballora said. “That’s sometimes called audification, which is taking the files and listening to them.”
The audio generated usually lasts about one second and often translates as a chirp or a beep.
Audification is what Chad Hanna, assistant professor of physics at Penn State, and the Laser Interferometric Gravitational-Wave Observatory team used in 2016 to hear the gravitational wave chirp that is created by two black holes colliding and merging, Ballora said.
Sonification is the process of taking the same set of data from an event and mapping it into an instrumental range. Ballora accomplishes this by transposing and multiplying the data so the numbers serve as frequencies.
“It’s less literal and more creative because you have to design a sound, which is what I like doing anyway,” Ballora said. “It’s like playing with synthesizers again, but now I’m designing a synthesizer that’s playing a data set.”
The technique does have its detractors, Ballora said. The pushback usually focuses on the numbers losing their value when they are transposed and multiplied, which he said is valid, but he believes the contour and the behavior of the data is the same.
Another complaint he has fielded focuses on the creative control he employs when developing an instrument to represent the sound of the data he has mapped.
Sonification is just an alternate modality to illustrate the behavior and to make the science more consumable.
Mark Ballora, Penn State music technology professor
“People have told me, you can’t say this is the sound of the data if your making up the sound of an instrument,” Ballora said. “They’re correct, it isn’t, but so what? Sonification is just an alternate modality to illustrate the behavior and to make the science more consumable.”
The Hart of his creativity
In 1979, Ballora attended his first Grateful Dead concert. He saw the band play several times over the next 10 years, but aside from being drawn to the improvisational nature of the music, he identified most with the “Drums and Space” portion of the show.
During the second set of a Grateful Dead show, Mickey Hart and Bill Kreutzman, the drummers for the band, would take over the concert for about 15 minutes, using standard drums and digital equipment.
“That fed my sonic pallet. It fed my tastes,” Ballora said. “I was never one of the people who left during ‘Drums and Space.’ That’s what I went there for, and I loved that open-ended weird stuff. It was extremely influential.”
When the band started to play large arenas and stadiums in the early ’90s, Ballora said he drifted away from the scene, but in 2011 he had a “serendipitous” interaction with a chapter from his past.
The Rhythm Devils, a percussion band formed by Hart and Kreutzman, was playing a show at The State Theatre in downtown State College. He went to the show, and at an information desk was a flier that detailed Hart’s work with Nobel Prize-wining physicist George Smoot on a film called “Rhythms of the Universe.”
The film focused on humankind’s yearning to understand the cosmos, and one of the tools used to tell the story was sonification. The flier had contact information and after being connected with Hart, Ballora began working on the film.
The collaboration led to a working relationship that brought Ballora back to California for the 75th anniversary of the Golden Gate Bridge.
Hart held a concert near the foot of the bridge at Crissy Field to celebrate the structure just miles from where Ballora grew up.
“He was using a model of the bridge that the Exploratorium Museum out there had built for him,” Ballora said. “He could shake and pluck and do all of these things to trigger different sounds. I created sounds for that and it was an incredible experience.”
The future of his craft
Ballora’s next collaboration is a result of a seed grant he was awarded this year by the Keck Futures Initiative, a program of the National Academies of Sciences, Engineering and Medicine with the support of W.M. Keck Foundation, which focuses on innovative, cross-disciplinary research and inquiry.
The goal of the initiative is to advance science communication, Ballora said, and each year the research focuses on a theme. He will be part of “discovering the deep blue sea.”
Beginning in 2018, Ballora will team up with marine biologist and cellist Heather Spence. The two will produce a performance art piece that combines Spence’s underwater recordings and cello playing with Ballora’s sonification of her data.
The goal of the project is to create art that can communicate the science of the ocean and tell the story of its declining health in a way that is more appealing to kids, which Ballora said is the direction inwhich he hopes to take his work.
“Get the kids. Get them to think you can listen to science as well as look at it,” Ballora said. “Let them make the innovations in 40 or 50 years, maybe they’ll be the ones to do it, so let’s plant those seeds now.”
During the Central Pennsylvania Festival of the Arts in July, his work was featured at one of the Penn State booths. One day, Ballora said a man stopped by with his 10-year-old son, who strapped on a set of headphones and watched all 22 minutes of “Rhythms of the Universe.”
After about 15 minutes, Ballora said the boy’s father looked at him perplexed and asked, “What should I do?”
“I laughed and told him, nothing at all. Let’s just leave him alone,” Ballora said. “Then I thought, ‘This is it, this is for him.’ ”