HEVC, AV1, VVC: How to Make Sense of 2019's World of Codecs
On March 27, 2019, Luxembourg-based Sisvel announced two new patent pools offering licenses on patents “relevant to the VP9 and AV1 specifications.” The pools apply solely to consumer display devices, like smartphones, computers, and TVs, and consumer non-display devices, like set-top boxes, dongles, and graphics cards. The standard rates for VP9 are €0.24 for display devices and €0.08 for non-display devices, while the rates for AV1 are €0.32 and €0.11, respectively. (The rates in U.S. cents are similar.)
The pools represent IP from JVCKENWOOD Corp., Koninklijke Philips, Nippon Telegraph and Telephone Corp., Orange, and Toshiba IPR Solutions. According to a Q&A with the Sisvel CEO, the Sisvel pools won’t charge for content, but haven’t ruled out charges for software-based playback. (Author’s note: As detailed at Streaming Learning Center, the author is providing some editorial consulting services to Sisvel.)
When AOMedia announced the release of AV1, it also announced the formation of a legal defense fund for any patent-related litigation, so the Sisvel pool is hardly surprising. No one has sued anyone yet, so the defense fund hasn’t kicked in. The next major development will be when Sisvel files a patent list, which should occur by late summer or early fall 2019. Once the list is filed, other IP professionals can start to review both the veracity of the patents and whether they apply to AVI and VP9. Until then, the only real effect of the Sisvel announcement is that all potential users are on notice that AV1 may not be royalty-free.
AV1 BROWSER SUPPORT
Since Google and Mozilla are both AOMedia members, it’s not surprising that Chrome and Firefox support AV1 playback. Throw in support from the Opera browser, and AV1 now plays in 35.28% of all browsers, according to Can I use (Figure 3), with more coming soon. That is, in March, Google announced that Android Q, now available in beta, “introduces support for the open source video codec AV1. This allows media providers to stream high quality video content to Android devices using less bandwidth.” This seems to indicate that support will be playback-only, not for recording.
Figure 3. As of May 2019, AV1 played in Chrome, Firefox, and Opera, accounting for 35.28% of all browsers.
In November 2018, Microsoft released the beta AV1 Video Extension, which allows AV1 playback on Windows 10 devices. No word on when this will be added to a Windows 10 update for broader distribution or if or when AOMedia member Apple will add AV1 to its computers or devices.
Hardware support for AV1 encode/decode wasn’t scheduled to appear until mid-2020, about 2 years after the bitstream freeze, and it appears to be on track. Note that browser-based playback has proven to be more efficient than many pundits predicted, which bodes well for AV1 adoption for browser-based playback. In October 2018, StreamingMedia.com reported that 1080p playback of an AV1-encoded YouTube video on Chrome consumed about 20% of CPU on an HP ZBook notebook powered by a 2.8 GHz Intel Xeon E3-1505M v5 CPU. A few days later, I reported on my blog that playback of the same video on the same notebook on Firefox consumed about 10% to 15% of CPU resources (Figure 4).
Figure 4. AV1 playback on an HP ZBook powered by a 2.8 GHz Intel Xeon E3-1505M v5 CPU
On May 23, 2019, Mozilla announced the integration of the dav1d decoder into Firefox, which should drop the required CPU resources even further. Mozilla also reported that 11.8% of video playback in Firefox Beta used AV1, up from 3% in March and 0.85% in February. Much of this traffic likely comes from YouTube, which posted an AV1 playlist in September 2018. In addition, Google software engineer Steven Robertson promised at Demuxed 2018 that YouTube would distribute 1TB per second of AV1-encoded video by the end of October 2018. Netflix also released some AV1-encoded video for public consumption in September 2018.
As we reported in 2018, however, Netflix encoded its 1080p video to 6.7Mbps, while YouTube was at 5.1Mbps, so neither company was pushing the quality envelope. Although Facebook claims that AV1 saves 51% when compared to x264, and 32.5% when compared to VP9, the company hasn’t publicly stated that it is shipping AV1-encoded video. At this point, a year after the bitstream freeze, we don’t know if any company is achieving the bitrate savings reported by Facebook at any kind of scale.
We do know that encoding times have dropped dramatically. When StreamingMedia.com first looked at AV1 encoding back in August 2018, encoding times were 45,216 times longer than real time. This dropped to 147 times longer than real time in tests from March 2019. Just before the 2019 NAB Show, Intel and Netflix announced SVT-AV1, which they claim is capable of real-time 4K/60p 10-bit encoding when running on Intel Xeon Scalable processors and Intel Xeon D processors (Figure 5).
Figure 5. SVT-AV1 is reportedly capable of real-time 4K/60p 10-bit encoding on certain Intel CPUs.
What’s the quality hit? At this point, it’s unknown, but the qualitative difference between SVT-AV1 and the AV1 version shipped by AOMedia should diminish over time. Here’s why.
According to the BBC, AV1 was built with Google’s VP9 specification as a base. To this, different “tools” are proposed by various AOMedia members for inclusion in the codec. Necessarily, at least at first, the successful tools are bolted into the codec architecture as opposed to integrating for maximum encoding efficiency. This is why AV1’s encoding times skyrocketed during the codec development process.
Intel’s SVT was designed to enable “the encoder core to be split into independently operating threads, each thread processing a different segment of the input picture, that run in parallel on different processor cores, without introducing any loss in fidelity.” This speeds encoding on multiple-core CPUs like Intel Xeons. However, the porting of AV1 to the SVT architecture starts with the core functions and adds different tools over time. For this reason, version 1 of any SVT codec won’t produce the same quality as a mature version of the same codec, but should catch up over time.
So, over the last 12 months, AV1 has demonstrated encoding and playback efficiency and substantial browser support, but also the prospect of royalties on the horizon.
VVC
VVC is the next-generation (after HEVC) standards-based codec developed jointly by MPEG and the ITU. The quality target is 30% bitrate reduction over HEVC at the same perpetual quality. Although the codec isn’t scheduled to ship until the end of 2020, some quality comparisons have already started to arrive.
The findings from the aforementioned BBC study are shown in Figure 6, with JEM standing for the VVC Joint Exploration Model and HM the reference HEVC model. As described in the 2018 International Broadcasting Convention paper titled, “An Overview of Recent Video Coding Developments in MPEG and AOMedia,” the study included both objective and subjective tests.
Figure 6. The BBC found HEVC (HM) and AV1 to be about the same, with VVC (JEM) substantially better than both.
At HD resolutions, the BBC found AV1 to be only 7% more efficient than HEVC, with VVC 33% more efficient than HEVC. At UltraHD resolutions, AV1 was only 2% better than HEVC, while VVC was 27% better. For those who care about such things, the BBC is both a member of AOMedia and a licensor member of the MPEG LA HEVC patent pool.
When reconciling these findings with Facebook’s, note that the BBC used reference encoders for its VVC and HEVC testing, which are encoders that use every tool in the codec. As such, encoding times are too lengthy for commercial use. In addition, the BBC used standard test clips that promote comparability among different test generations, in essence, testing encoders no one will ever use for production encoding with clips no consumer will ever watch. In contrast, Facebook analyzed 100 of its most popular test clips using actual production encoders, which makes it a completely different test.
On the licensing front, one would assume that contributors to VVC include many of the same companies in the HEVC and H.264 pools. To help avoid the issues experienced with HEVC, a group of companies formed the Media Coding Industry Forum (MC-IF). As stated in a press release, “MC-IF will initially focus on establishing VVC and related standards as well-accepted and widely-used standards, for the benefit of consumers and industry.” One key goal is to “Provide a forum for, and encourage, the discussion of issues related to the licensing of intellectual property rights relevant to the deployment and use of these Standards, in the furtherance of the Purpose.” While laudable, MC-IF has no teeth, so the impact of the organization remains to be seen.
A Future Decided by ... Patent Attorneys?
To a great degree, the codec future will be dictated by AV1’s royalty-free status, which AOMedia could secure by proving that member companies truly reinvented the wheel with AV1 or by licensing technology as needed. Remember that Google used this latter tactic to cease the formation of a VP8 patent pool by MPEG LA back in 2013. Or perhaps AOMedia can garner enough control over AV1’s IP to distribute AV1 with a fair and reasonable royalty.
If not, it’s likely that many of the same IP owners behind H.264, HEVC, and VVC own technology used by AV1. This suggests a truly dystopian future in which IP owners attempt to impose HEVC-like business terms on AV1, VVC, and future codecs. You’d like to think they learned their lesson, but 6 years, 4 months, and 4 days after the release of HEVC, they haven’t seemed to.
[This article appears in the Autumn 2019 issue of Streaming Media Europe Magazine as "2019: A Brave New (Codec) World."]
Related Articles
With the recent launch of version 1.0, SVT-AV1 appears to have caught up with libaom in quality, with very definite performance advantages. Its two-pass rate control is tested and proven. If you're creating an AV1 encoding workflow today that emphasises encoding speed and quality, SVT-AV1 should definitely be on your short list.
22 Aug 2022
Streaming Learning Center Owner & Chief Blogger Jan Ozer discusses the role the Alliance of Open Media's influence plays in the current codec landscape, and how it impacts the adoption of AV1 and VVC in particular as the streaming world increasingly moves beyond H.264.
31 May 2022
Who's contributing to the two VVC patent pools, and will the combined royalties accelerate or discourage adoption? We sat down with two industry experts and the CEOs of both MPEG LA and Access Advance to get their insights.
01 Mar 2022
The terms are out—20 cents per unit for hardware/paid software and 5 cents per unit for free software—but there's lots of fine print and plenty of questions remain, including those about the contributors that aren't in either the MPEG LA pool or the Access Advance pool.
01 Feb 2022
VVC today can be both useful and usable; let's hope that VVC IP owners can formulate a royalty policy that delivers the same.
08 Dec 2020
Within 24 months, hardware support appeared, encoding became affordable, and AV1 became a much more realistic competitor to HEVC. Here's how the currently available AV1 codecs measure up.
05 Oct 2020
Royalties for 2,000 patents in AV1 from at least 12 companies will be charged via Video Coding Licensing Platform administrator Sisvel, which promises "reasonable and transparent" cost
10 Mar 2020
BBC R&D finds that AV1 produces better low-bitrate quality than HEVC, but the codec picture will get even muddier in 2020 as MPEG fast tracks VVC, MPEG-5 EVC, and LCEVC
24 Jun 2019
It's not easy, but it can be done, as long as you have the right tool and know the tricks. Here are eight steps to success using Softvelum Nimble Streamer.
20 Mar 2018
With Apple joining the Alliance for Open Media, there's been a lot of talk about AV1 vs. HEVC. But we no longer live in an either/or world, and can now give viewers what they want with minimal cost and effort compared to years gone by.
17 Jan 2018