Usually, watching a music video online isn’t that different from watching it on TV. But when we at Tool were asked to make a video for the Cold War Kids song I’ve Seen Enough (coldwarkids.com/iveseenenough/) we decided to do something a little special.
Each of the four band members was filmed on a darkly lit stage, playing different versions of the song on different instruments. When viewing the resulting video online, fans can chop and change between these versions – or even switch off individual members entirely – providing the ultimate remix experience.
Here, we’ll explain how our team pulled off this uniquely challengingaudio-visual experiment.
Planning the shoot
To achieve a filmic effect, we shot at 24 frames per second and kept everything dark on the set. This also meant we didn’t have to worry about fading things out, fading the edges out and blending them into the background. Because it was so black we could have put in a faded edge or a drop shadow or something, but it all worked so perfectly with the black edging that all you can see is the lit-up band member and their instrument.
We focused on not having a lot of movement. The bass player moved around quite a bit in the original version and that did affect the quality. The longer they stayed in one position, the more detail we could get on all those pixels. The black backgrounds also helped keep things simple conceptually, helping the end user to focus where we wanted – the different versions of the music and the actual performance.
The key to this video was small movements. It’s not like we told the musicians to stand still: it was more about the way we shot it. We didn’t have the camera moving around – it was stationary – and we blacked out the floor. This way we didn’t have too much detail, because even drawing the keyframe, all the video was done with one keyframe per second, allowing jumping to that point.
In Flash, you need to have a keyframe that tells it all the information on the frame, in case you jump to that point. If you had a keyframe every 30 seconds, say, or no keyframes, you wouldn’t be able to jump around at all – you couldn’t do what we did. But when you have a keyframe, it’s like taking a screenshot of that exact moment in time.
Intros and outros
It was about 30 seconds before the audio was cued, so we instructed everyone to come in, pick up their guitar and start tuning it, getting ready. That was roughly 30 seconds. Then the band would hear their cue, which would indicate that the music was going to start in about five seconds. The intro of the song was always such that they knew when to come in, and the outro was around 30 seconds of them hanging around, putting their instruments down or whatever. Those times were all the same; we did cut that down in the end, though. Regarding compression strategy, we started off with some heavy compression, using the H.264 encoding on the Flash videos. The thinking was to get the smallest file size because we were dealing with four band members and four different versions of the song; that’s 16 different videos of about three minutes each.
At first, we actually tried doing all 16 videos, playing them, queuing them up all at once. That was the most seamless approach. Everything was running and always in sync because it all started at the same time and they all played exactly in sync. This worked great – until we got to a certain point where the CPU couldn’t handle it, and the browser would crash.
File size
When we switched to the four versions, or four videos playing at once instead of all 16, we had some sync problems switching between different versions. That’s where a lot of the problems were prevalent with the heavy encoding. To counter this, we switched to the least CPU-intensive decoder, SorensonSpark, but we were also just using the regular Flash Media Encoder. And that made a huge difference in playback.
The CPU dropped considerably and that bumped up the file size, but only by about 28 per cent, which wasn’t too bad because we were able to compress these quite a bit. The MP4 compression was really dark with the H.264. When we went to the other ones, we tried the On2VP6 Pro and it was just brighter using those two codecs instead of the MP4 H264 decoder. In the end, we used Sorenson Spark.
For the audio, we took it down in mono: it sounded great with 22 and 16. We brought the data rate down to 48 or 40 kilobits per second and it still sounded pretty decent – that helped the file size a bit. We tried to get the smallest file size using Sorenson Squeeze and trying to use the H.264 encoding that would give us the best compression. That turned out to give us a decent file size, but it would really spin up the CPU quite a bit. As a result, we would get some crashing when we were playing back the videos, because we were playing all four all at the same time and trying to decode them.
Taking this approach meant that we’d need people to have fast machines for it to work properly.
Syncing videos and audio
When the user came in on one of the different versions, say for the singer, between the acoustic version and the reggae version, it would load up and randomly pick the different versions. And once everything was buffered, it would start to play it back.
We came up with a number of tricks to sync it up. If the user switched to the different version, say the reggae version, it would stop the playback of the video/audio the user was just playing and then load up the reggae version and pause it as soon as it was in the buffer and was able to get the events back. It would pause that, seek to go to a second in advance of the current time with the rest of the videos, do a loop there on a set interval loop and wait until the rest of the videos were at the same time. Then it would start to play that and catch up, so there was a margin of error there of about less than a 10th of a second. And if it missed that, if it didn’t load that information in time, it would miss that syncing and try to load two seconds in advance. And then if it missed that again, then it would load three seconds in advance and just increase that until it was able to buffer it in time.
Hybrid streaming
Using the FLVMDI metadata injector For the pseudo streaming solution, I used the FLVMDI metadata injector. It goes through the videos, take out all the keyframes fromeach one, determine the byte position, put them into an array andthen embed them into the metadata of the flv, meaning when the flv is loaded in there, it knows if it has to jump to the three-minute mark. As a result, it knows it has to jump to the 2MB mark, which says get me this chunk of data. This can’t be done natively; it requires the flvmdi to do that. This meant looking at other solutions, there’s lighty http which is just a built-in server that will embed that metadata right into the flv on the fly, so it doesn’t have to be run separately. For this project, however, it wasn’t necessary.
Hybrid streaming solution
The default functionality of playback in Flash is that it’s progressive, so you don’t have to download the whole thing before you start playing. The problem is that if you have more than one video and you’re trying to sync them up, it’ll try to download everything up to that point. If you’re at the three-minute mark and you click the reggae version for the singer, it’ll try to download the three minutes prior to that point and then start playing. What YouTube and other sites do – there are a lot of scripts out there for this now – they actually put in the byte position of that time. This give you the ability to jump to the three-minute mark. It knows that it’s at the 2MB mark, so it just uses a PHP script to find that position in the file and it jumps right to it. This way, you don’t have to download anything before that. It’s kind of like live streaming using Flash Media Server – which we did try out, but we didn’t like it because there was a lag.
In the end, we used the hybrid or pseudo streaming method that YouTube uses where it fixes any point. Using that pseudo streaming solution, it was kind of a hybrid of those solutions where you could jump to any point without downloading anything. That’s what the Flash Media Server offers up, but you could also download it to your cache, which is what the Flash Media Server doesn’t do. So, it would be buffered in cache ready to go and you could preload it. That’s what we did in the second version. We just allowed someone to click on a preload version and they would sit there and wait for the 15 or 16MB to load up before they even started it. That way, when you clicked between different versions, it would be a lot quicker.
It was using the same baiting mechanism, where it would jump to a time a second in advance. However, because it was already preloaded, it was always consistent, and would always hit that spot.
So, for example, if you’re watching 15 seconds and you click on the reggae version, it would jump to 16 seconds, pause it, and loop 30 times a second or so, until it saw that the current time of everything else was up to the same time and it would play that video.
Another key to this project was having one central timer. There are a couple ways to do this. The solution we came up with in the end was to have a video that’s just a timer video.
There are actually five videos playing in the background. That timer video we made by taking the singer, it didn’t matter, just any version there that would be in synch, so it’s exactly the same length, exactly, everything, and not encoding any of the audio and taking the screen size down to like the very minimum – 10 x 10, or something like that. So it was all black, and a very small file size, maybe 32K.
Percentage loader
We inserted a real time percentage loader in the testing version. That’s just the buffer filling up. When you click on something that is just showing the buffer. We played around with different buffer sizes. There’s a tradeoff. The longer you buffer, the longer it takes to load. And this means that you would likely miss the synch mechanism. But the more you buffered, the better chance you have of having it not stutter and get out of synch after. In the end, we only did a second or two. That buffer wasn’t super important because the bit rate of all this, each one, was around 220 kbps.
This means a person with 1 mbps connection, which is a typical, regular bandwidth for cable modems, would be about that. In theory, they should be able to handle it, like watching a single high bandwidth video; these are four low bandwidth videos.
Dustin brings 11 years of digital marketing experience and an undeniable passion for the interactive space. Since joining Tool in 2009, Dustin has been responsible for leading the growth of the company’s digital business, overseeing the development of interactive live-action content that merges the worlds of technology, art and advertising. Most recent signature projects include the “Touching Stories” iPad app, which was unveiled in a five-day Fast Company exclusive; “David on Demand” featuring Leo Burnett’s David Perez; and the award-winning interactive music video “I’ve Seen Enough” for Cold War Kids.
Before joining Tool, Dustin was Co-Owner & Managing Partner, at digital agency Spacedog, with offices in LA + NY. In 2008, Dustin helped to negotiate Spacedog’s acquisition by Mendelsohn Zien and Hakuhodo, the second largest advertising holding company in Asia.
Working with clients such as Sprint, Starbucks, Project (RED), Doritos, Citibank, Nike and others, Dustin has overseen over 150 digital media projects across all digital platforms. His projects have been recognized with numerous awards, including a Cannes Bronze Lion, two Webbys, AICP Next, Bronze Clio, four FWA SOTD’s, MTV VMA “Breakthrough Video” nomination, an OMMA and a Davey Award.
Dustin maintains an active presence in the interactive advertising industry serving as a judge for the Webby’s, Davey, FWA Mobile and .net Awards; speaking at interactive conferences such as the Yahoo! Creative, 2009 Boards Summits and 2010 Creative Review Click Conference SF; holding membership in the International Academy of Digital Arts & Sciences; and serving on the committee for AICP Digital. 
