<-- Advertise here.

Hi, I am an interactive editor at msnbc.com who worked on the U.S. presidential election maps. Now that the election is over and I am well rested, I want to look back at what we have worked on all these months at msnbc.com. A few people wrote me asking about the process of creating a large scale project in a fast paced environment. Here is a longer answer to that question. Since this post will be a bit long, I will break it into several sections.

dataExplorer_infosthetic.jpgClick on image to see full size.

Pre-election content

We put a huge amount of effort into the Decision '08 Dashboard data explorer map. Many people had been preparing interactive content for our election coverage for months before I arrived. By the time I started in July 2008, most of the core functionality of the map had been developed. My first assignment was to support other developers and designers, by preparing data and writing small utility classes. Next, I created the polling data display and helped with designing the fundraising data display.

We wanted to map every data set we could find that came with geographic information. There were at least 10 data sets we wanted to map, but we launched our politics front map with only six data sets. We added user opinion, polling, candidate appearances and fundraising data over the next several weeks.

Most of our data sets were huge and we needed the technology to support and serve them to a large audience. This is where the team that builds our publishing platform came in. They built a new, user-friendly database tool just in time for us to use for the election.

The amount of information that the maps show is enormous. We have state and county level demographics data such as age, race and education from the U.S. Census and voting history of states and districts back to the 1980's. We also mapped individual contributions to the presidential candidates as well as every single candidate for a seat in the Senate and House of Representatives. Not a lot of our competitors have that kind of depth. In fact, I can't recall one that has this much variety of information either.

Of course, there are a few more things we wish we had done, but given the deadlines that we had, I think we did a good job.

Continue reading the post ».


Election night plans

While I was not heavily involved with the design and implementation of the pre-election maps, my responsibilities changed as we started working on the election results maps. With Chris Cast, a fellow producer from msnbc.com, we was responsible for planning and designing the pre-election maps. We spent some time thinking about the data, and trying to come up with the best way to visualize it. We also thought about what the competitors might do. Chris was more heavily involved with designing the mock-ups while I thought about the code that we would have to write to make things happen.

The Creative Director of msnbc.com, Ashley Wells, had a desire to dominate the web with msnbc.com's election coverage (not too shabby for a goal =)). To make that happen, we had to tackle a few challenges. First, we had to make sure our servers could handle the traffic that was expected on election night. We also had to come up with an effective way to distribute our content around the Internet.

To meet this very ambitious goal, we did a few things. First, msnbc.com's web operations team team spent a great deal of time optimizing our server capacity and stability. Second, we created maps for our homepage and the homepage of MSN that could drive traffic deep into our site where more results and maps were displayed. Third, we made portable maps that could be embedded on other people's Web sites and blogs that also linked back to more detailed results. Last but not least was the results map on the Decision '08 Dashboard. We also worked with MSN global to create a flexible version that they could translate into many different languages and place on MSN global homepages and news channel pages. The results of several months of work look like this:

allMapsPanorama_infosthetic.jpgClick on image to see full size.

The implementation

While the whole designing and planning process was still going on, I started to build the code foundation upon which most of our election night maps relied. The Politics front results map was the only map that used a slightly different code base. Although I adapted a lot of the code that had been written for our pre-election maps, this job was a challenge for me. It was a challenge because I had to optimize our maps to reduce their file sizes and make sure they only requested the data that they needed. Although we were confident about the servers' ability to handle the predicted traffic, we wanted to minimize the amount of database queries that the maps made, especially because we were making the calls every two minutes to ensure users had the lastest results. In the end, we eliminated a couple of minor features so that we didn't put unnecessary stress on the system.

Another challenge we faced was the MSN US map. We had to be extra careful with the file size because of the amount of traffic that would hit the map. Chris Cast and my manager Paige West, who is the interactive director at msnbc.com, spent a great amount of time optimizing the map. Things were a bit easier with MSN global. I was quite happy after seeing the screenshots of the different localized versions.

msnGlobalPanorama_infosthetic.jpgClick on image to see full size.

The biggest problem that we faced throughout this project was with the data. There were a lot of inconsistencies and special cases in the results feed. It probably took 30% of our development time dealing with special cases. It was hard to check the data ahead of time because of its size and the low rate at which these inconsistencies occurred, so until we actually started testing with sample data, we didn't realize how many problems it had.

Testing testing testing...

The team spent a lot of time testing the maps to make sure that all the features worked appropriately. In the last few weeks leading up to the election, we participated in several transmission tests and dress rehearsals in which both the editorial and the technical staff at NBC and msnbc.com practiced for every conceivable scenario on election night. We put together a prioritized list of bugs and spent the last couple of days fixing them. Although the bugs list was relatively big, we managed to fix every single one that we found. With five people working on fixing bugs at the same time, the list disappeared in almost no time =).

As more and more people began to see the finished product we had a few requests for additional features or improvements to the existing functionality. Although these requests came late in the game, the team managed to push them through by working in shifts over 24 hrs for several days leading up to the election.

frontPagesPanorama_infosthetic.jpgClick on image to see full size (1.2MB).

The results

We released the portable maps a few days before the election and hoped that they would get picked up by the social media. We also distributed these maps to a few NBC affiliates, some small newspapers, and other media outlets that did not have the resources to develop this kind of data-based map. Because we considered this project to be highly experimental in nature, we had fairly reasonable expectations. In the first few days after release, the amount of installs grew quickly, but on the day of the election the number of installs skyrocketed.

The maps on our site and all the MSN sites went up in the afternoon of the election day. We continued to make small improvements right up until the maps went live, but over all I think it went pretty well. This fact was later confirmed by the traffic stats. Here are a few excerpts from that report.

"Nielsen Online shows that for election day msnbc.com was #1 in total minutes and time per person, doing the best job at meeting users' needs and holding their attention."

"The combined total in page views on msnbc.com was nearly half a billion: 471 million."

"Interactive maps and the one-stop Decision '08 Dashboard drove over 100 million page views on 11/4."

"Our best-of-breed news widget was shared by users all over the web, driving an unprecedented 12 million widget page views, and driving over 8 million click backs to msnbc.com. The widget was used by approximately 2 million unique visitors over the 2-day period."

About the author: Vu Nguyen is a graphic designer who works and lives in Seattle. He currently works as an interactive editor at msnbc.com. He has previously worked as a graphics editor at The New York Times. Vu's personal blog and portfolio can be found at www.biofusiondesign.com.

Do you have a story about a visualization? Contact infosthetics.


Interesting post-mortem, thanks!

May I ask how you optimized the size of the county maps? Were they hand-drawn? One thing I find when working with county-level GIS data is that it's quite difficult to compress it down to something of reasonable size for the web, even if you use sophisticated tools to thin the data.

Sat 15 Nov 2008 at 6:48 AM

Wonderful writeup, I would love to see more of these. Doing journalism with maps and data and code is hard work, we should all share what we learn!

Sat 15 Nov 2008 at 8:52 AM

Nice overview and beautiful stuff. I love how scalable they are. 24+ hour shifts! Yuck, but it looks like it paid off.

Sat 15 Nov 2008 at 8:53 AM

MtotheThird: there are a lot of things you can do to optimize the swf file and the data. We used a tool that can reduce the amount of points it takes to draw a map. There are also a few techniques that you can use to shave off a few Kb in the swf file. You can use less fonts and use the omit tracing option in the publishing settings. That would help to save a few kbs. You can also choose to use CSV files for your data instead of XML because XML files tend to be bigger due to the extra XML tags in the file. You can actually write your data file in a binary format and compresses it with zip and then use ByteArray in flash to read the zip file. I haven't tried that myself, but that would be probably one of the better ways to reduce data size.

Chris Blow and Eyevariety: thanks for the kind comments.

Sun 16 Nov 2008 at 3:43 AM

This was an amazingly good read, thank you. I'm actually gonna re-read this and take notes, I'd never thought about doing a project on this scale before and I really appreciate the amount of detail you went into here. Very generous of you.

Sat 22 Nov 2008 at 2:01 PM

@Justin: Thanks for the comment.

Due to request, I am writing a tutorial on how to optimize flash project file size. I am also thinking about writing tutorials about how to setup and create a data driven interactive infographics with flash/flex. So, if any of you are interested, email me, I will let you know when I finish it.

Sun 23 Nov 2008 at 5:43 AM
Commenting has been temporarily disabled.