The State of Video Codecs 2022
In this 2022 codec update, I'll identify the most significant announcements from the last year relating to H.264, VP9, HEVC, AV1, Versatile Video Coding (VVC), Low Complexity Enhancement Video Coding (LCEVC), and Essential Video Coding (EVC). Codecs are many, but space is limited, so I'll be brief. If you feel like I missed a significant event, please add a comment about it in the online version of this article.
H.264
H.264 is the closest we have to a universal codec, which is why it was surprising that in Bitmovin's "Video Developer Report 2021," H.264 usage actually dropped from 92% in 2020 to 83% in 2021 among survey respondents. This data was backed up by predictions from Rethink Technology Research's "Media & Entertainment Transcoding Workload and Device Royalty Forecast 2020-2030," which has H.264 usage dropping from around 80% of all traffic in 2020 to about 25% of traffic in 2030. We all knew H.264 usage had to decline at some point, and it appears that the decline has started.
To set the stage for the rest of this article, I've shared a chart from the Bitmovin report (see Figure 1), which shows the new codecs that survey respondents plan to add in 2022. Note that respondents include representation throughout the video encoding and delivery ecosystem, so even if few publishers deploy VVC or EVC streams in 2022, encoding and other services that will ultimately deploy these codecs appear to be in the process of adding them to their technology stack.
Figure 1. Codecs that developers plan to add in 2022, from the Bitmovin “Video Developer Report 2021” (go.bitmovin.com/video-developer-report)
VP9
One of the most significant challenges for implementing new codecs is the additional complexity they introduce, which translates to increased encoding costs. This is particularly true with large-scale services like YouTube, since they need advanced codecs to reduce streaming costs and avoid "breaking" the internet. YouTube has long used VP9 for 4K and high-volume content, but has encoded in software, which uses 5x more computer resources than H.264.
In April 2021, Google announced the release of Argos video-transcoding chips, which deliver a 20x–33x improvement in compute efficiency compared to its previous server setup. Each chip includes 10 encoder cores that "can encode 2160p in real time, up to 60 FPS using three reference frames," according to Data Centre Dynamics. As discussed in a Streaming Learning Center article from August 2021, YouTube now uses VP9 for videos with as few as several thousand view counts, with AV1 reserved for videos with 3.5 million or so view counts.
On the royalty front, Google has long maintained that VP9 is an open source and royalty-free codec; Sisvel, with its VP9 patent pool, disagrees. In December 2021, Sisvel announced that Panasonic signed a licensing agreement for its usage of VP9, bringing the licensee total for both the VP9 and AV1 pool mentioned below to 30.
HEVC
There's not much to report about HEVC, which remains king of the living room, particularly as it relates to High Dynamic Range (HDR) content. Although there's been lots of chatter about AV1 and other codec support in Dolby Vision, HEVC remains the only 10-bit codec supported in a Dolby Vision profile. As discussed later, although Netflix did make a big, AV1-in-the-living-room announcement, it also stated that "another exciting direction we are exploring is AV1 with HDR," meaning that HDR isn't there yet.
However, HEVC made no inroads in the traditionally weak browser market, with Can I Use reporting only a 19.52 global browser availability rate, almost exclusively in Apple products. In contrast, Can I Use reports that AV1 is available on 72.89% of all browsers (with Apple the major omission), and VP9 is available on more than 95% of all browsers.
The lack of browser support for HEVC doesn't hinder premium content distributors, which can use apps with HEVC playback on most platforms, but it keeps HEVC unusable for publishers targeting broad, browser-based playback. The lack of HEVC support in Chrome and Firefox 9 years after its completion also bodes poorly for native browser support for newer standards-based codecs like VVC, EVC, and LCEVC.
AV1
The most significant AV1-related announcement in 2021 was Netflix's blog post "Bringing AV1 Streaming to Netflix Members' TVs," which essentially declares that the living room is ready for AV1. A short list of supported devices—six certified Samsung smart TVs, PS4 Pro, select Amazon Fire TV devices, and select Android TV devices—published in The Verge seemingly contradicts that claim, as does the aforementioned lack of HDR support in Netflix's current AV1 offering. There is an HDR10+ AV1 draft specification and talk of Dolby Vision support, but it's tough to be king of the living room without fully implemented HDR support on a relevant critical mass of devices, which AV1 doesn't currently seem to have.
Also related to TV, an update to the European DVB television specification is expected in early 2022, and it will include three codecs: AV1, AVS3, and VVC. This, along with content support from Alliance for Open Media (AOMedia) members, would seem to ensure that AV1 will be included in most (if not all) TV sets built to adhere to the upcoming standard.
On the mobile front, David Ronca, formerly of Netflix and now with Facebook, reports in a YouTube video that the hardware AV1 decoders he expected (and AOMedia predicted) would deploy in 2020 were pushed back, "so we're probably not going to see hardware introduced at any meaningful level until 2023/2024 and…we're looking at 2026/2027 before hardware decoders reach critical mass in the mobile market" (see Figure 2). Instead, Ronca was hopeful that software decoders could provide efficient 720p60 decoding "for a majority of Android devices by 2024." This, plus Apple's lack of AV1 support, dims the AV1 mobile picture considerably, although note that Netflix has been delivering AV1 to Android phones since early 2020.
Figure 2. In a YouTube video with John Porterfield, David Ronca shares the bad news about hardware AV1 decoding on mobile platforms.
In contrast, distributing standard dynamic range AV1 in the browser is pretty much a done deal. However, presumably due to the cost of encoding, YouTube limits AV1 deployments to videos that seem destined for view counts of more than 3.5 million or so, as previously mentioned. As a general rule, you'd have to assume that if YouTube (and Netflix) can't afford AV1 for videos with lower viewer counts, most other producers probably can't either. As reported at Streaming Learning Center, there are several AV1 hardware encoding alternatives coming, which could make AV1 encoding affordable for all.
In the WebRTC market, Google announced an AV1 encoder for Chrome, Cisco added an AV1 encoder to Webex, and Millicast announced an AV1 encoder for its WebRTC production environment. Clearly, codecs optimized for 720p at 2Mbps have little relevance for 4K premium content. Perhaps there's hope for efficient software-based AV1 encoding nonetheless.
AOMedia claims that, like VP9, AV1 is open source and royalty-free, while the Sisvel AV1 licensing program seems to indicate otherwise. In February 2022, Sisvel reported that its total number of AV1 and VP9 licensees had reached 30.
This takes us to the three new MPEG codecs.
VVC
In terms of quality, I compared VVC to AV1, HEVC, and H.264 in late 2020, and in late 2021, I compared VVC to these codecs plus EVC and LCEVC. The summarized version is that VVC is the most efficient codec in the land, although it's also the most complex for encoding and decoding. I tested Fraunhofer's VVC implementation in both articles; those interested in a more complete view of VVC quality should check out Moscow State University's 2021 report.
In terms of deployments, standards-based codec adoption involves a number of milestones, particularly for those such as VVC and EVC, which will likely require hardware-accelerated decoding on mobile and living room platforms. Royalties must be set, then chips must be developed and integrated into retail products for deployment and, sometime later, reaching critical mass.
In terms of chips and integration, AV1 is the most recent codec to move through this cycle. For reference, AV1 was finalized in 2018, the first TVs and OTT devices appeared in 2020, and the market achieved critical mass, at least for Netflix, in late 2021. So, at best, it's a 2-year cycle from finalization to first retail product and then another year until a relevant critical mass of products is available to target.
VVC was finalized in June 2020. On the royalty front, in early 2021, the Media Coding Industry Forum announced that there would be two patent pools, administered by Access Advance and MPEG LA. Access Advance announced its royalty terms last year, and MPEG LA announced theirs in February.
In terms of hardware decoding, two implementations have been announced. In July 2021, Allegro announced its AL-D320 video decoder semiconductor IP core supporting the latest VVC/H.266 format. According to Allegro, the decoder is "immediately available for integration by SoC vendors," so actual chips with the decoder won't be available until after this deployment. In November 2021, MediaTek rolled out the Pentonic 2000, which the company claims "is the first commercial 8K TV chip with Versatile Video Coding (VVC) H.266 media support." MediaTek further states that the chip will support Dolby Vision out of the box and that it expects smart TVs with the chip to launch in the global market in 2022.
Assuming that MediaTek is correct, we should expect the first TVs with VVC to ship in 2022, which means a critical mass of addressable devices will be available sometime in late 2023 or 2024, and sooner for greenfield deployments in which the service provider dictates the set-top box provided.
What about software decoding on computers and mobile devices? Certainly, there are workarounds to the anticipated lack of browser support by Chrome and Firefox, and app developers can use whatever codec they want in their apps. The issue, of course, is the required CPU horsepower and the impact on battery life. There are several intriguing reports of completed trials from companies like Youku and Tencent and other documented trials of software decoder performance from the Joint Video Experts Team (JVET) that look promising, but, with the exception of AV1, codecs are very seldom deployed on mobile platforms without hardware decoding.
New codecs also benefit from inclusion in TV-related standards, and both VVC and LCEVC were integrated in Brazil's recent TV 3.0 project (see Figure 3). As mentioned earlier, VVC will also be included an upcoming DVB specification.
Figure 3. Brazil’s next-gen TV set will include both VVC and LCEVC.
What does this add up to? It's hard to live in the U.S. and not have a U.S.-centric view that AOMedia content suppliers like Netflix, YouTube, Amazon, and Hulu will dictate codec adoption, and it's unlikely that any of these companies will support VVC in the near term. However, it's a big world out there, and companies such as Alibaba (which owns Youku) and Tencent have their own cloud distribution systems and appear to favor VVC. Certainly, there's been a much more diversified effort behind VVC than EVC or LCEVC. All that said, it's difficult to imagine any scenario that would push VVC support to the top of the 2022 to-do list of any engineering manager who's not working for an absolute top-of-the pyramid streaming company.
LCEVC
As an enhancement codec, LCEVC is different from VVC and EVC. It's more lightweight for encoding and decoding, so it really doesn't need dedicated hardware for either function (see go2sm.com/testingevc). V-Nova, the company behind LCEVC technology and its biggest promoter, has solved most of the deployment issues for mobile and browser-based computer playback. For example, V-Nova can supply playback in HTML5 players like HLS.js, Shaka Player, and Video.js and has already published LCEVC-capable apps for iOS and Android platforms.
In early 2021, V-Nova released its royalty structure, which applies a capped, low per-user fee on the streaming service actually using the codec and is free for encoder and decoder vendors. This places the royalty burden on the company receiving the bandwidth savings and should simplify adoption in the encoding and playback ecosystem, although probably not with AOMedia members.
You can read various LCEVC-related announcements, mostly involving evaluation testing and MPEG verification testing, at lcevc.org. Other news includes integrations with Harmonic, Southworks, and Red5 Pro. In my article on LCEVC testing, I tested LCEVC with x265 as a base layer, and it delivered as promised, providing better quality and superior encoding/decoding efficiency than x265. I plan to test LCEVC with AV1 as a base layer for a webinar, and this pagecontains a link to the webinar and any downloads.
EVC
EVC is often the forgotten sister of the three codecs released by MPEG in 2020. VVC has many fathers who benchmark, promote, produce trial deployments, and create encoding and decoding hardware. LCEVC has one very loud parent, V-Nova, that keeps it in the spotlight. In contrast, EVC has three main contributors—Samsung, Huawei, and Qualcomm—which seem more focused on finalizing and improving the technology than marketing and deploying it. If you scan the pre-June 2021 activity for EVC and compare it to VVC, you'll get a sense of the comparative commercialization-related activity for each codec. I could find no announcements that EVC-enabled chips were imminent or even in the works.
EVC was finalized in April 2020, and there's been no announcement of the royalty policy. True, the contributors agreed to a 2-year time window when creating the codec, but sooner is always better when it comes to royalty policies, and with only three contributors, one assumed that the 2-year goal was an outside limitation, not the actual target.
I tested EVC in December 2021, along with VVC and LCEVC, and the open source encoder and decoder provided very good quality and encoding/decoding efficiency. The technology was impressive, but building a better mousetrap only gets you so far without aggressive productization. Certainly, Samsung, Huawei, and Qualcomm can bring substantial marketing resources to bear, but we've seen no evidence of that happening as of yet.
[Editor's note: Jan Ozer consults with Sisvel on marketing and related matters, and he occasionally performs testing and other related services for V-Nova.]
[This article has been revised to correctly reflect David Ronca's comments about hardware decoding in mobile]
This article first appeared in the 2022 Streaming Media Industry Sourcebook European Edition.
Related Articles
CodecWar is an analysis service created by ViCueSoft, the developer of codec analysis tools VQ Analyzer, VQ DVK, and VQ Probe. As currently configured, the site's ideal users are researchers who are looking for a convenient way to compare codecs using relevant datasets and codec developers who are looking for a structured way to benchmark their codecs against others. In contrast, it's not a particularly convenient way for streaming producers to run experiments to optimise their encoding parameters, although it could grow into this.
28 Aug 2023
Welcome to your codec update for 2023. I'm here to help you decide whether it's time to go all in on AV1, VVC, LCEVC, or EVC or whether it's better to stick with H.264, VP9, and HEVC. Along the way, I'll include 2022-2023 updates to identify significant changes that have occurred since last year.
27 Mar 2023
Jan Ozer put EVC, VVC, and LCEVC through the paces, checking each for not only encoding quality, encoding complexity, and playback efficiency but also power consumption. Each one has its pros and cons; read on to find out how they all performed.
29 Mar 2022
Who's contributing to the two VVC patent pools, and will the combined royalties accelerate or discourage adoption? We sat down with two industry experts and the CEOs of both MPEG LA and Access Advance to get their insights.
01 Mar 2022
The terms are out—20 cents per unit for hardware/paid software and 5 cents per unit for free software—but there's lots of fine print and plenty of questions remain, including those about the contributors that aren't in either the MPEG LA pool or the Access Advance pool.
01 Feb 2022
Independently validated to deliver higher quality at up to 40% lower bitrates, LCEVC enhances compression and computational efficiency of any codec on any device
02 Nov 2020
Unlike other cutting-edge codecs coming to market, LCEVC will have an immediate impact on the streaming landscape. Here's how to use it.
13 Mar 2020
Companies and Suppliers Mentioned