Testing the New Windows Media Encoding Profiles
With this as background, let's jump to the next topic, which is whether the improved quality produced by the command-line options changes the balance of power in the codec world.
Who's Number 1?
Some background before jumping in. In the original codec comparison I prepared for StreamingMedia.com, I tested five different categories, from 56Kbps postage-stamp video to 500Kbps 640x480 video. For this article, I duplicated those testing procedures, but only for the 500Kbps category, primarily to save time, but also because the 500Kbps category was both highly relevant and challenging from a compression perspective.
In my study, RealNetworks' RealVideo rated the highest in all tested categories. However, in the survey I mentioned at the top of this article, I visited over 40 sites, and found no RealVideo streams. This doesn't mean that RealVideo isn't out there, but the company has refocused on consumer sales, along with sales of servers, broadcast hosting and content delivery. For example, in their last reported quarter (Q3, 2006), Real's revenue from consumer products was $82 million compared to $11 million from technology products. Throw in the $761 million anti-trust settlement received from Microsoft in October, 2005, and Real appears to be doing quite well, thank you very much, but RealVideo is on very few short lists for codec selection. For this reason, I did not include RealVideo in this comparison, though I have little doubt that they would have won again.
Instead, I compared the Windows Media Video Advanced Profile file produced via Ben Waggoner's batch file with a Flash file encoded with On2's Flix Pro and an H.264 QuickTime file produced with Apple Compressor. I also included a legacy Windows Media Video file from the first survey for comparison purposes.
The test file contains categories for business, high motion and entertainment. I rated each codec on a scale of 1-4 based upon four criteria, still frame quality, color quality, temporal quality, which assessed motion-based artifacts, and smoothness, which essentially looked for dropped frames. In calculating the total ranking in each category, I gave full weight to still-image and temporal quality, and half weight to color and smoothness, reflecting my belief that viewers find still-frame and motion artifacts more noticeable and objectionable than dropped frames or color fading.
Overall, Flash edged out Apple's H.264 for top quality, 1.39 to 1.72. However, Flash scored 106 total points to 100 for WMV-9 AP and H.264. It's nice being first and all, but with this small a difference, you have to wonder if anyone will notice. As it turns out, analyzing each category separately put a different spin on the issue. So let's jump in.
Business
This segment is comprised of six primarily talking-head clips. The Windows Media Video Advanced Profile swept all components of this category, courtesy of very sharp images and rock-solid backgrounds.
H.264 had problems with both image quality and temporal quality, specifically background noise, which is illustrated by Figure 6, an image that I've darkened to highlight the artifacts in the back wall. The Apple image in the bottom right shows significant bleeding and noticeable background blockiness, while the Flash frame, on the bottom left, showed significant banding, or grouping of pixels into noticeable bands. In comparison, the Advanced Profile video on the upper right shows the smoothest wall of all.
Figure 6 (below). The Windows Media Video – Advanced Profile (upper right) showed the clearest image and least noticeable background artifacts.
WMV9-AP came out on top with 53 points, with Flash coming in second at 50—a 6% difference—and WMV9 coming in third at 44—a 17% difference. In my view, a points difference of 10% or less is probably unnoticeable absent side-by-side comparisons, so is commercially irrelevant. Between 11-20%, viewers may suspect that quality isn't up to par, and beyond 20%, most will notice and start to wonder what's up.
Accordingly, I would ignore the 6% difference between Flash and WMV-AP, particularly because still frame quality was identical and because by choosing an appropriate background, you can minimize these types of background artifacts. In this regard, avoid backgrounds with wide-open spaces on a bright, reflective surface, like the one shown in Figure 6. Either add low-detail pictures, posters, or similar objects, or drop the lights on the back wall to reduce the reflection.
On the other hand, at a 28% difference between WMV-AP and H.264 (38 points), producers using this format should consider bumping their data rates significantly to produce equivalent still-frame quality, and manage their backgrounds to minimize temporal artifacts.
Action
The action category contains seven high-motion clips including golf, kickboxing, skateboarding, martial arts, and the pita clip shown in Figure 3 that has tortured codecs since I shot the video in Israel back in 1999. In a surprising turn, Apple won this category (and the next) with good still-frame quality and fewer temporal artifacts during real-time playback than its nearest competitor, Flash, though the difference between the two, at least in this category, was not commercially meaningful—39 points for H.264 and 37 points for Flash, a 5% difference.
On the other hand, WMV-AP's performance (29 points, or a 26% difference) was significantly below that of both H.264 and Flash, primarily due to blockiness in the still frame comparison, and lots of motion artifacts. A good example of the still frame issue is shown in Figure 7, where blocks are noticeable in the two WMV images, while the Flash and H.264 images are clear.
Figure 7 (below). Both WMV clips are much more blocky than Flash and H.264.
