Q: What is it like to have the process of video encoding or transcoding bring you to tears?

In 2005, we built a system to do the first amateur ATSC HDTV broadcasts. We broadcast MIT sporting events and the January "Integration Bee" (with play-by-play and color commentary, etc.) on the MIT cable TV system. (http://broadcastengineering.com/newsrooms/mit-sportcast-plan-calls-hd-telecasts, http://sportcast.mit.edu)

The plan called for us to build our own live HD switcher with wipes and other transitions, score overlay and ATSC encoder -- a commercial system was several hundred thousand dollars at that point. We had four HDV cameras connected via Firewire to a head-end computer, and used an array of 1U servers with nVidia cards to render an arbitrary scene with OpenGL (using a pipeline through four machines to paint the video from four cameras on the frame), then overlay the score and time, and finally encode with A/52 audio in ATSC-conformant MPEG transport stream and send out in QAM256 over the MIT cable system (http://web.mit.edu/sportcast/checkout/).

Some of the technical challenges proved especially difficult and I was up all night with them before a game. Getting four Firewire HDV cameras to talk to the same computer was a real pain, since these cameras generally have cheap chipsets and all want to talk on the same broadcast channel. (We got this working, but it was quite brittle since unplugging one camera would freeze the whole Firewire bus for about 1 second. In year 2 we switched to connecting the cameras to small Mini-ITX computers and then running them to the switcher over UDP on Ethernet.) Showing up at 10 a.m. at a volleyball game and having to explain to my colleagues that we STILL didn't quite have working video was painful!

The most challenging part was writing the MPEG systems stream encoder that would be ATSC-complaint -- in other words, writing a program to stitch together Dolby AC-3 audio and MPEG-2 video that would actually PLAY in sync on a real store-bought television without glitches. (The MPEG-2 video elementary stream was compressed by libavcodec, and the audio by liba52, but getting a compliant audio-video multiplex is a different story.)

This was difficult because those TVs do not exactly give you a lot of helpful diagnostic information. If you do it wrong, you see glitches, but these can be very rare (like once every 20 minutes!) and it's not like you get a debugging trace. 

ATSC has a lot of requirements you have to comply with, like you can't send a frame of video more than 0.5 seconds before it will be displayed, and if you break these you will see undefined behavior from the TV.

Getting all those pieces put together, so we could actually WATCH our sports broadcasts on a real HDTV in 2005 via the cable TV system without hiccups, was very satisfying and a great payoff for all our work. After spending umpteen all-nighters doing it and breaking numerous promises to my friends and colleagues to have it running earlier, I am sure I was brought to tears (of exhaustion and/or joy) when it finally worked.