Buyers' Guide to Quality Analytics 2017
Delivering high-quality video requires effective quality checks from ingest to playback on the viewer’s computer or device. Accomplishing this requires multiple suites of tools from objective quality benchmarking to Quality of Experience (QoE) measurement. Though the distinctions between some of these categories are occasionally blurred, in this article I identify the major categories of tools and why you need them, some prominent tools and vendors in the space, and factors to consider when choosing a solution.
Note that lists of tools are meant to be illustrative, not comprehensive. If you’re a potential buyer, you should do your own research to identify all tools in the category. If you’re a vendor who wasn’t mentioned, contact the author to make sure you are considered next time.
Objective Quality Benchmarking Tools
Objective quality benchmarking tools help you make major decisions about encoding configurations, codecs, and encoding tools. For example, when building an encoding ladder, you should benchmark the quality differences relating to decisions like data rate, bitrate control, and encoding profile and/or preset. When choosing an encoding tool or cloud service, you should compare output quality before making a final decision, as it does vary by encoder and service.
The major tools in the space include the Moscow State University Video Quality Measurement Tool, SSIMWave Video QoE Monitor (Figure 1), Tektronix Picture Quality Analyzers (PQA), and Video Clarity ClearView. All work by comparing a compressed file with the original and computing one or more quality metrics like peak signal-to-noise ratio (PSNR), structural similarity (SSIM), SSIMplus, picture quality rating (PQR), and/or difference mean opinion score (DMOS). These metrics let you assess the actual quality difference produced by that extra 500Kbps in your top 1080p stream, or determine whether the encoding tool or cloud service that claims premium video quality really delivers.
SSIMWave QoE Monitor’s file visualization screen
When comparing tools, first identify your price point. Fully configured with batch capabilities, some tools cost close to $20,000, which is outside the budget of most producers. In contrast, you can compute simple PSNR calculations with FFmpeg, which is obviously free. All of these tools will offer both GUI and batch operation, which is essential to running these tests and which can often be very time-consuming.
Next, identify the metrics supported by the tool. The Moscow State University tool can produce over 20 metrics, but none are recognized as being as advanced as the SSIMplus rating from the SSIMWave tool, or the PQR and DMOS ratings from the Tektronix tool. By advanced, I mean that the ratings produced by the tool should have a higher correlation with subjective ratings than metrics like PSNR or SSIM. That said, despite having helped produce its own quality metric (video multimethod assessment fusion; VMAF), Netflix still refers back to PSNR for many of its quality decisions as a reality check.
Intuitively, we know that video that looks great on an iPhone might look awful on a 60-inch 4K TV, which makes device-specific scoring a useful feature. Attention-weighting, which weights quality in the primary region of interest more than quality at the edges, where most users won’t notice issues, is also useful.
Make sure the tool doesn’t have any time-related limitations. Most test files will be 5 minutes or longer, but occasionally you may want to check your assumptions with a full-length movie. Surprisingly, not all tools can handle these longer files. Finally, you should work with the tool before buying to make sure that the GUI and batch processes work well for you. Moscow State University offers a free trial program limited to files smaller than 720p, while most other companies can supply limited time demo programs.
Automated File-Based QC Tools
The tools mentioned in the first category solidify your encoding functions; the tools in the quality control (QC) category ensure the integrity of the files produced by your encoding and packaging systems. In a perfect world, these functions will be available both pre- and post-encode: there’s no sense encoding a file with 5 minutes of unplanned audio silence, or an aspect ratio different than what you’re expecting. After encoding and packaging, you want to make sure your files are in the proper format, pass all required regulatory checks, and are properly linked and accessible, which is complicated by the thousands of media files often created for adaptive bitrate delivery.
Tools in this category include standalone tools like Interra Systems’ Baton (Figure 2), Tektronix Aurora, and Venera Technologies Pulsar. Most workflow systems, like Telestream Vantage, can perform some QC functions with internal tools, like Telestream’s Vidchecker program, plus integrate with the Interra, Tektronix, and Venera systems.
Interra Systems’ Baton automated QC tool works from ingest to archive.
The first consideration here is whether the system can perform QC checks on the formats you’re producing, like ABR formats HTTP Live Streaming (HLS), Dynamic Adaptive Streaming over HTTP (DASH), Smooth Streaming, and the like. Does it simply check manifest files and file locations, or does it also verify details like keyframe alignment? If you’re distributing content via DPP, Netflix, or CableLabs, make sure the tool can certify these packages. By now, most tools should be able to check for loudness and sequences that can trigger photosensitive epilepsy (PSE) as well as the integrity of subtitles and closed captions, but verify this before making a purchase. You should also determine if the tool can correct loudness and PSE-related problems, or simply report them.
Next, determine whether the tool can check files where they are located, either locally or in the cloud. Check if the solution is installed locally or if it’s cloud-based (the latter obviously is easier to scale up or down to meet demand), and also make sure it can be used from multiple offices if your production capabilities are decentralized. If you are working in multiple offices, check for relevant language support for the user interface (UI) and QC reports. If desired, make sure the system can be run from inside your own firewall, as opposed to solely as software as a service (SaaS).
Since you’ll likely want to integrate QC capabilities with your encoding and/or media asset management functions, check how this would be handled. Determine how the system can alert you when there is a problem, look for the tools to help you verify and correct the problem. Beyond a report identifying the problem and its location, many tools include a player that lets you scrub through a flagged video, with markers identifying specific problem areas.
Check pricing options, which vary significantly by vendor. For example, most Venera products offer pay-per-use pricing, which is a fabulous option if you’re a production house that publishes only a few titles irregularly through the year. If you’re a high-volume shop, check system throughput and the associated cost before comparing pricing, since throughput will obviously determine how large a system you will need.
QoS and QoE
QoS and QoE are often used interchangeably, but while they are related, they’re completely distinct. Simply stated, QoS, or Quality of Service, refers to the performance of the network, while QoE, or Quality of Experience, relates to the quality actually experienced by the user. While it’s unlikely (if not impossible) to have consistently high QoE when QoS is bad, it is entirely possible to have a subpar QoE even when network performance is stellar.
Related Articles
Video quality measurement is more crucial than ever. In this buyers' guide, we'll identify the most commonly used metrics and discuss tools that deploy them.
20 Mar 2019
Counting viewers has always been a difficulty for streaming publishers, and today's multiplatform world makes the issue more demanding than ever. The answer is to radically rethink video analytics.
31 Oct 2016
With Parrot's demand ratings, broadcasters and OTT providers can measure the popularity of content streamed from any service.
28 Oct 2015
A research report finds the European OTT video market poised for strong growth, and sees big differences between East and West.
09 Sep 2014