Review: Moscow State University Video Quality Measurement Tool
In a recent project, I compiled PSNR and SSIM scores (higher scores are better), mostly because they were the most familiar, and VQM (lower scores better) to identify true quality differences. What does VQMT tell us about our tests in this case? As you can see in Table 1, in the top comparison of 640x360 files at 600Kbps, the High profile delivers better quality in all three metrics, though PSNR is only 2 percent better and SSIM is less than 1 percent. At 7.5 percent, however, the difference in VQM score feels potentially meaningful. At 1200Kbps, the SSIM and PSNR results are even smaller, though the 4.8 percent difference in VQM again feels potentially meaningful.
Table 1. Objective quality metric scores produced by VQMT. Lower scores are better in VQM, higher scores better in SSIM and PSNR.
How meaningful? Let’s have a look. Once you run a test, VQM opens the Results visualisation screen shown in Figure 2. As you can see, there are two graphs, the top graph representing the entire file, and the bottom graph representing the smaller portion of the file shown as the darkened region in the top graph. To navigate to particular frames, you drag the Video slide beneath both graphs to the left or right, or use the left and right arrow keys, with the frame number (1202) shown to the right of the slide and in both graphs.
Figure 2. VQMT’s Results visualisation screen, showing how the score varied frame by frame over the duration of the two videos. This is the VQM test so lower scores are better.
This is the VQM comparison for the 600Kbps files, so lower scores are better, and the red line (first processed) is for the Baseline profile and the blue line (second processed) is for the High profile. The graphs show how the VQM scores varied for the two files over the duration of the file, with the VQM score for that particular frame shown in the Values box to the right of the Legend box on the bottom left of the screen.
For most of the file duration, the red and blue lines are fairly close, indicating that there’s not a significant quality difference at those locations. However, the red hump around frame 1202 shows a region where the quality of the Baseline-encoded file compared poorly to the quality of the High-profile encoded file. To see the quality difference, drag the slider to frame 1202, and push Show frame on the bottom right of the Results visualisation screen. This opens the viewer shown in Figure 3.
Figure 3. The VQMT frame viewer showing the Baseline and High encoded files. High is clearly superior here.
Using the radio buttons on the bottom of the screen, you can toggle through the Original and two encoded files, or you can use the 1, 2, and 3 keys on your keyboard to do the same. This makes it very easy to compare the compressed files with the original, or with each other. In Figure 3, for example, you can see that the guitar player’s pinstriped shirt and several other regions are clearer in the High profile file than in the Baseline. Using the scale slider (set to 269 percent in Figure 4), you can zoom into the image, and then toggle through the inputs, which is incredibly useful when analysing larger resolution files.
What conclusion do I reach on the Baseline vs. High profile issue? Well, in Figure 2, you can see that for every red spike there’s a blue spike, indicating a region where the Baseline profile outperformed the High Profile. Figure 4 shows comparison frames from one of those regions, where Baseline-encoded quality clearly exceeds the file encoded using the High profile. Overall, there’s no doubt that High profile quality exceeds Baseline, though like objects in the mirror, the quality might be closer than you think. I still recommend going with a single set of files, though reasonable minds can certainly differ. One final caveat; while still image comparisons are very useful, you should always compare the files during real-time playback to identify motion artifacts.
Figure 4. Here’s one region where the Baseline encoded file offered superior quality to the High profile file.
Overall, after 24 years of encoding and encoding comparisons, VQMT has changed the way I evaluate my encodings. If you need to make serious decisions about codecs, encoding tools or compression configurations, you’ll find it invaluable. Note that Moscow University offers a free trial worth downloading and experimenting with, though it only analyses files of less than 720p resolution.
This article appears in the Winter 2014 issue of Streaming Media European Edition as "Review: Moscow State University Video Quality Measurement Tool."
Related Articles
Any good compressionist needs tools to help reveal critical data about encoded files, from codecs and resolution to GOP structure and quality levels. Here's a look at some of the most useful.
18 Mar 2020
Video quality measurement is more crucial than ever. In this buyers' guide, we'll identify the most commonly used metrics and discuss tools that deploy them.
20 Mar 2019
Reach viewers no matter where they are. Check out these strategies for reaching the most devices with the smallest number of files.
06 Mar 2013
Get advice from Sorenson, Zencoder, and Encoding.com: When companies should encode and transcode their own content, and when it makes sense to send video to the cloud.
28 Sep 2012
Our annual test of the leading desktop encoding programs.
10 Mar 2011