[ Strata Mini Maker Faire -- photo by Pinar Ozgar for O'Reilly Media ]
While my selective experience of the conference left terms like Hadoop and Map Reduction just as mysterious to me as they were before Strata, I still gleaned a general sense of the technologies at work within this growing data deluge.
But first, we get a welcome sense of perspective from day 2 keynote speaker Mark Madson of Third Nature. He reminds us, with a somewhat unexpected and engaging history lesson, that our present "information overload" certainly is not the first of its kind. Everything from the printing press and the Dewey decimal system could be considered as precursors to the processing and organizational systems for big data developing today -- interesting thoughts to keep in mind while encountering the rest of the conference's pitches and promises.
Arnab Gupta's more forward-looking treatise on "Man + Machine" taps into an age old discussion about our relationship with technology, which thus far has been a "history of innovation substituting human labor". We usually like to interpolate Skynet and malicious robots in a dystopian future from there, but Gupta argues that the next evolution will be towards the enhancement humanity.
It took 3 weeks for online gamers to crack the code of the AIDS enzyme model, a feat that no computer or researchers had been able to accomplish in more than a decade. A mortgage default map of California could have helped predict the 2008 housing bubble burst, if someone using the tool had the curiosity to test what would change if unemployment went up. The fact remains that humans are really brilliant at things machines are not, and vice versa. While data is the key word, it is the process of making sense of it and distilling meaningful signals that matter more and more.
[ AIDS enzyme model cracked by online gamers using FoldIt ]
Meanwhile, new tools for data visualization stirred up their own macrocosm of hype within the conference.
[ Protovis template styles, including famous Florence Nightingale chart and Napoleon's march information visualization ]
[ Static images of a few of D3's interactive visualization options: Voronoi diagram above, and force-directed graph below ]
Oftentimes, it's not picking the machines or software, but a putting together the right people that becomes crucial to success. The Guardian's Alistair Dant takes us through the process of building a new interactive piece from the ground up, and how it is made possible by the variety of skills in his team -- the usefulness of a motion graphics person to whip up a video to present ideas to managers and editors, development and flash experts that can creatively work around unusual problems, and having someone with legal experience never hurts. Technology is only as good as the talent that wields it, and it can have as much influence on the pace and shape of the creative process as has on the end product.
[ Guardian interactive piece that visualized tweets during the 2010 World Cup ]
Presented with such powerful and seductive communications software, I worry how easy it is to use them as a crutch or lean towards novelty rather than make critical decisions that best articulate your message. It would be interesting to compare the work from template-ready Tableau and D3, versus more general purpose "creative coding" libraries, like Processing. With the flexibility comes more power, though building a piece from scratch (or something closer to it) certainly is not the pragmatic choice for everyone. In any case, by understanding the constraints and opportunities around each tool, even the simplest ones, an elegant solution can be reached.