How to Deliver Resilient Streams at Scale
Guaranteeing a satisfying end user experience, whether you’re delivering content live or VOD, requires resiliency, ensuring that the stream doesn’t break down regardless of the scale, bursts, or other fluctuations in delivery demands. The challenges are different for live and VOD, with live proving significantly more challenging in most instances.
Mark de Jong, Chairman, CDN Alliance, discusses these key challenges and how to address them with Michael Demb, VP, Product Strategy, TAG VS, and Bob Hannent, Principal Architect: Technology Operations, DAZN, in this clip from Streaming Media Connect 2023.
De Jong asks Demb to discuss some of the most important elements of resilient live streams beyond taking a multi-CDN approach.
Demb emphasizes that VOD streaming is not at all like live streaming because when streaming live, there is very little room for mistakes. “If I have to choose my top three differentiators from a pure distribution perspective,” he says, “one will be the latency. In VOD, there is no such thing as latency. [Then] scalability, and of course quality assurance. How do you make sure that you're delivering the right quality of video? So to ensure that live streaming works smoothly, real-time monitoring is key. You need to know exactly what is happening with the streams at every moment. If something starts going wrong, you need to catch that fast and fix it even faster. That's critical. So in terms of scalability, live events, especially sports…they can be huge and unpredictable. Spikes in viewership can start from 10 viewers and grow to 10,000 and maybe even 10 million. And you don't know how many users you're going to have. So that means you need a really flexible setup, real-time performance measurements, and that can quickly identify those bottlenecks and make the switching decisions.”
Demb says that there can be embarrassing and unsatisfying user experiences if there is too much latency during live streams, which can result in occurrences such as the final scores of a match appearing on the phones of users before they actually see the winning goal. “So our customers invest a lot of effort in reducing their streaming latency and ensuring it stays low,” he says. “And TAG is helping them to measure those latencies and make those decisions. TAG just recently rejoined the Streaming Video Technology Alliance (SVTA) to help identify the best methods of measuring latency in live streaming. SVTA has published a number of papers on the subject that were part of that work in the past. And soon we'll have a cloud-based test environment to test real live use cases for low latency streaming and measurement.”
Regarding content quality, Demb says, “So you need to make sure that this video stream gets to your audience at the highest quality possible without any delays or buffering. And that means, first of all, choosing the right format codec to reduce the bitrate as much as possible without losing quality. And for that, you need the monitoring solution in place as well. So, having the right tools with very practical and actionable KPIs is the key. For example, quality degrades. You want a monitoring tool that can tell you exactly what happened, point you in the right direction, and also help understand the root cause of that, not just tell you the quality dropped from five to three. That's not very actionable. So you don't want some actionable KPIs and actionable alarms that will point you to where the problem is.”
Hannent agrees with these points and notes that sports fans are far less forgiving of live-streaming errors than VOD viewers. “There is nothing worse than the anger of a sports fan who isn't able to watch their content,” he says. “Maybe if you fail to publish Game of Thrones properly, there will be an issue, an equivalent perhaps, of passion. I'm not convinced. I think 99% of the time, if you can't watch the piece of VOD you want, you'll find something else. If you have decided the appointment to view is that game, you've got to deliver it. You can get the CDN infrastructure lined up and you can get the video chain sorted, but stampedes on your APIs, stampedes of trying to get CDN links, stampedes of trying to get DRM licenses. If you have got people all clicking on the tile at the same time, or worse still, you have a glitch in the CDN, and people rotate, or your system rotates synchronously…I've seen 40,000 requests in two seconds to the backend and going from a baseline of less than 500. So that kind of ramp up.”
Hannent mentions that autoscaling is not a panacea. “Autoscaling does not work most of the time, and that's a really generic statement, a broad statement, but you have to be really careful about autoscaling,” he says. “But we have multiple regions located around the world, so that load can shift if something happens.”
He speaks of “headends” and how DAZN effectively uses them. “We have two main headends,” he says. “Two secondary and a disaster recovery headend. By headend, I mean the encoder and packager and all that stuff. So that is a very high level of resilience, and all that needs to be monitored as well. Are you monitoring the level to the level that you need on that? And there's another element: Do you want them to be coupled or decoupled? How do you want to ensure that impact on one system doesn't have an impact on another?”
Watch full sessions from Streaming Media Connect November 2023. We'll be back in person for Streaming Media NYC on May 20-22, 2024. More details here.
Related Articles
End-to-end workflows for live streaming at scale are complex affairs, with troubleshooting challenges at each stage when delivery breaks down. An expert panel from Paramount, Amagi, TAG VS, and Nomad Technologies discusses key best practices for troubleshooting large-scale streams when the pressure is on.
22 Feb 2023
How much streaming reliability and quality are worth trading for ultra-low latency, and when is one at a premium over the others? Amagi's Brian Ring, Dolby,io's David Hassoun, Nomad Technologies' Adam Miller, Paramount's Corey Smith, and Norsk CMO Eric Schumacher-Rasmussen discuss in this panel from Streaming Media West 2022.
07 Feb 2023
The best chunk size for low-latency streaming is dependent on a number of factors based on different use cases, and there is often a need for some compromise and tradeoffs in quality or speed. Nadine Krefetz, Consultant, Reality Software, Contributing Editor, Streaming Media, asks three industry experts what their chunk size preferences are for their requirements.
27 Sep 2022