Quality Control Roundup: Catch the Mistakes Before they Go Live
Interestingly, the audio and video quality-related tests differ greatly between the integrated and stand-alone systems. Most significantly, neither of the two stand-alone systems that I tested can perform comparative quality analyses such as PSNR or SSIM. The obvious reason is that they don’t have the original file to compare to the encoded file since the file was encoded by a different program typically running on a different computer system.
I don’t have a bias one way or the other regarding comparative versus stand-alone tests that might identify certain levels of blockiness, pixelation, or blurriness. Obviously, however, before buying a system, you have to feel comfortable that the qualitative checks it performs can meaningfully separate the wheat from the chaff.
How Does the System Report QC?
Most QC systems, whether integrated or stand-alone, can pass machine-parsable XML results data to an encoding or MAM system to further drive the workflow. Beyond this, most systems provide multiple forms of feedback that humans can use to identify and resolve issues. For example, Tektronix offers See and Solve test results that let you drill down into the file until you see a thumbnail image of the erroneous frame, while Baton provides extensive PDF reports. Both systems also let you play back the file after encoding, Tektronix using the open source VLC Player and Baton via the Baton Media Player, which lets you quickly click to problem areas and view the related frames in the media file itself (Figure 5).
Rhozet can integrate QC checks within the encoding workflow, simplifying the review and resolution of any issues. Specifically, all files flagged with QC issues appear in a separate tab in the Workflow System, where you can review the errors as thumbnails or in the video file itself, and you can approve, reject, or escalate the problem to a higher grade. Telestream Vantage can similarly set triggers for operator review for any files flagged at any point in the production workflow.
How Do I Integrate QC?
How and where you integrate your QC system depends upon whether the QC system is integrated or stand-alone. With an integrated QC system, you integrate the system with your MAM or the equivalent. With a stand-alone QC system, you integrate QC functionality with your encoding system and MAM.
Either way, you’ll typically have three alternatives: manual jobs, watch folders, and integration via an application programming interface (API). You create manual jobs through the program’s interface, identifying the files to analyze, the tests to perform, and actions to be taken upon success or failure.
With watch folders, you identify a watch folder and the associated tests and post-test action items, which typically will include copying the encoded files to different folders to move to the next distribution step. If you use the API, you pretty much can define the tests, notifications, and end actions through the entire workflow. For most producers, manual jobs will help debug the system, with automat- ed operation, whether via watch folders or the API, driving the bulk of day-to-day analysis.
How Do I Buy QC?
OK, let’s consolidate this information into a few questions or areas of focus to examine before buying.
- The basics—price and operating system. Be sure to determine how pricing is scaled. For example, the base version of Cerify can only analyze a single stream at a time, which may work in small shops, but it might be insufficient for large-scale operations.
- Integration. If using a stand-alone QC sys- tem, determine how it will interface with your encoding system, and recognize that you probably don’t want to be a pioneer here. For example, Digital Rapids Transcode Manager has hooks into both Cerify and Baton, so if you’re buying a stand-alone system to use with Transcode Manager, give priority to one of these systems. Plot out where the QC functionality will fit into your production workflow and determine the necessary post-encode action items and reporting feedback. If you’re considering using an integrated encoding/QC system, figure out how it will integrate with your MAM. Also ask how QC failures are integrated into the encoding workflow, with the ability to review errors manually and then either approve or reject the files—a valuable feature.
- High-level workflow. For simple ingest-test-encode-test cycles, any system will do from the workflow perspective. If you want your encoding/QC system to perform triage on input files, you should plot out what tests you want to perform and how you want the system to react to them. Then, between your encoding, QC, and MAM systems, under- stand how you’ll make this work.
- Format/quality/container tests. If you have any particular quality control tests that you need to run, make sure you identify those up front. In particular, identify which quality-related tests you trust the most and determine if the candidate systems offer these.
- Reporting. Identify the reports that you need for the integration that you’ll be using (manual, watch, API) in both human and machine-readable formats.
- Trial version. It’s always good to test before you buy.
- Scalability. If your needs are currently large or if they may grow, you should understand how the QC system can scale to meet increasing demands.
Now, with this as a background, let’s jump to the mini-reviews.
Tektronix Cerify Version 7.1.1.18
Tektronix Cerify is a stand-alone QC program that starts at $10,000 for a version that can process a single file simultaneously, with a version that can process two simultaneous files costing $18,000. You drive Cerify via a browser-based interface. Cerify’s basic building block is a job, which marries a media set and a profile. A media set defines the files to be tested, which you can manually select or designate via a watch folder, while the profile defines the video-related, audio-related, and container-related tests and the post-test actions (Figure 1).
Video quality tests can include tests for too many black frames at the start, at the end, or during the video; letterbox and pillarboxing; luma limit violations; and RGB component violations. Note that Cerify uses the same testing algorithms here as in Tektronix’s other hardware and software analysis tools, so the results should match tests produced by those other products. That’s a nice benefit if you use other Tektronix products elsewhere in your production workflow.
Cerify’s primary subjective H.264-related quality measure looks at blockiness and rates your video on a scale from 0 (poor) to 100 (excellent). In analyzing the frames, Cerify looks for hard edges that correlate with macroblock boundaries. As with most QC tools, you choose both a quality rating and duration, so, for example, you can kick out a video if any 10 frame region averages lower than an 80 quality rating.