LondonR lightning talk: Audiblization / sonification of data

It looks like the next LondonR meeting on 10 September 2013 will involve a series of 5 minute lightning talks rather than a few half hour slots. I have proposed “Audiblization / sonification of data: what are people doing and is R a good launchpad for it?”

Wot no subwoofer? Image courtesy of Matthew Grover.

This is at such an embryonic stage that it’s pretty comparable to graphs in the 1770s. There are even two terms in common usage, although it feels like consensus is moving towards ‘sonification’. We don’t even have a William Playfair yet, popularising it with good ideas. But some people are giving it a go and I’ll show a few examples. What I’m particularly interested in is using it as a communication tool for data and statistical patterns. There may be an exploratory role too but I’m unconvinced at present. R is well placed for this given the ease with which it can interface to other packages. Some people are keen on Python for much the same reason. There is at present a well-established R package tuneR which will handle some basic audio editing, and a package audio that does some I/O, but anything fancier will be best handled externally. There are, of course, powerful C++ libraries like CLAM, opening up the possibility of Rcpp-powered on-the-fly synthesis. With tuneR, for example, you work on a sound object and then end by writing it to the hard drive as a .wav file. Then, and only then, can you hear it.

Open-source synthesis packages like SuperCollider and PureData are powerful but essentially they speak a totally different language to stats people. It’s a bit like the old days where programming a GPU involved pretending your data was an image. We need data-oriented interfaces or this will remain the preserve of oddballs like me who have somehow acquired experience of stats and programming and sound art or music. It will also remain hard to convince your boss that what you’re spending time on is not just messing about!

 

Advertisements

4 Comments

Filed under R

4 responses to “LondonR lightning talk: Audiblization / sonification of data

  1. There’s already 2 packages specifically for sonification: playitbyr and audiolyzr.

    They both have a long way to go, but we’ve started the journey!

  2. Hi Ethan, nice to hear from you. I enjoyed your significance article on the subject (http://www.significancemagazine.org/details/magazine/2731281/Listening-to-uncertainty-Information-that-sings.html)

    Yes you’re absolutely right, I’ll be sure to cover those packages. I particularly admire the work behind the scenes making the ggplot-inspired syntax for playitbyr. That lends itself to a design where R objects are passed to external software to control various aspects of synthesis. That’s a nice design but I wonder whether it works for people who know data but not sound? Do they know what sonic parameters to assign different objects to? Have you had any feedback shedding light on their experiences?

    • I haven’t heard a huge amount of feedback about that, though that’s a very good point.

      Ideally, an R sonification package would free the user from thinking about too many parameters and create pretty good sonifications by default. My sense of the state of sonification literature, though, is that the best practices aren’t very well-known at this point, so it’s difficult to know what those defaults are.

      That said, there are some things that we do know, and I haven’t dug all those nuggets out of the literature and brought them into the package yet. I did put in the package almost all the suggested sonifications from the chapter on statistical sonification in The Sonification Handbook.

      What would your ideal R interface to sonification look like? I’d love to hear more perspectives on this.

      • Hi Ethan

        For me personally, it’s going to be different to the average number cruncher because I’ve spent many hours working with audio editing software in the past. I feel less comfortable doing the synthesis in an external piece of software because it’s one more thing to learn and rely on, but I accept that is a curmudgeonly attitude and it’s likely to be the most efficient route. I would like something for building and perfecting that sends the sound to a player of some kind and keeps it looping (perhaps just a shorter chunk between specified start and end points). If I then update the code, it will tweak the sound and keep it rolling, so it’s close to a real-time experience. Having said all that, perhaps the best interface is to have another package (Pure Data maybe?) call R, not the other way round. Then you could leverage the functions in PD and just have R do preliminary data processing, which is what it’s really good at.

        Having said all that, someone who’s into data but not yet audio is going to prefer an R-based interface even if it is slower and clunkier.

        The number one consideration is freeing people up so they can think about the aesthetics of it, the clarity of the patterns and message, etc. If they can tweak things quickly and hear the effect, it’s going to help that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s