Our team has extensive expertise in implementing a wide range of delivery protocols in the Nimble Streamer software media server — from conventional HLS and RTMP to WebRTC and our proprietary low-latency SLDP protocol. We’ve also recently implemented HTTP/3 in Nimble and even contributed a related fix to lsquic library.
We have also developed our own players for SLDP (browser and mobile) and a WebRTC WHEP player, giving us a comprehensive view of the pros and cons of each protocol from both transport and playback perspectives.
As technologies evolve, we are transitioning to the next generation of delivery protocols and playback mechanisms. This means that we’ll need to make changes to both the delivery and playback components of our streaming pipeline.
New playback
Let’s begin by reviewing the current state of end-user browser playback.
There are three major methods for video playback in the browser:
- MSE/MMS – The player receives content and feeds it in chunks to the browser’s playback engine, which then decodes and displays it. This approach gives limited control to the player and it’s limited to the codecs supported by MSE on the platform. There’s no strict definition of chunk size, so small chunks can disrupt playback. Features like zapping and rewind may also be problematic.
- WebRTC – Playback is fully handled by the browser’s internal WebRTC stack. Developers have minimal control, and limitations include restricted codec options and limited buffering configuration.
- WebCodecs – This method allows the player to decode frames using any codec available on the platform.
We will start by using WebCodecs in combination with our SLDP protocol.
With WebCodecs, the player has full control over the processing pipeline:
- Audio and video is synchronized manually before playback.
- Jitter buffer is configured as needed.
- Playback speed can be adjusted – be slower or faster if needed.
These capabilities are great for debugging purposes as we have just a single point of failure, which is the decoder. The rest of pipeline is under our control, so if some stream is not played properly, we can always check what is wrong at each step of the way – whether it’s video, audio, decoding, display etc.
So using WebCodecs-based player brings more options to those who will use it on their website, and allows to improve the usability by web developers, and the overall user experience. We’ll open source this new WebCodecs-based player so developers can customize the pipeline for their own use cases.
Adding WebCodecs to SLDP allows us to focus on the playback side while using a transport we’re already familiar with.
But that would be just the first step, the next one will be the delivery.
New delivery
Now here’s what we have on the transport side of end-user transport pipeline.
- HLS/DASH – Chunk-based protocols that are currently industry standards. They rely on TCP and large buffers, making HLS the mainstream delivery method despite higher latency.
- WebRTC – A real-time protocol offering full control over the transmission pipeline for both client and server.
- WebSocket-based protocols (e.g. SLDP) – Overall, a good delivery method, however it has “head of the line” problem. These are often paired with MSE/MMS for playback, with its aforementioned drawbacks
- Media-over-QUIC – Considered the future by many, but still far from standardization or broad implementation.
- WebTransport – A QUIC-based transport protocol built on HTTP/3.
We plan to implement WebTransport as part of this initiative in Nimble Streamer and the player. While it’s not yet supported on Apple devices and will be in the standardization process for years, its advantages are promising. For instance, HTTP/3 allows seamless transitions between networks (e.g., switching between cellular and Wi-Fi) without interrupting playback.
We already added WebTransport protocol support into lsquic library as part of this initiative.
The game plan
We’ve already begun work on the new player. The current implementation successfully plays content, proving the concept and allowing us to move forward. You can see current state of our player in this github repo. So our development is done in two phases.
Phase 1: Create a web player with WebCodecs. It will use our SLDP protocol over WebSockets as the transport and it will decode and display content using WebCodecs.
SLDP provides configurable latency, supports adaptive bitrate and other useful features, which have attracted a large customer base. This makes it a solid starting point for our WebCodecs implementation. Our new player will have same advantages like configurable latency and much more.
We’ll also extend HTTP/2 and HTTP/3 support to MPEG2TS over HTTP and Icecast, enhancing playback options for those formats. E.g. this will make them playable more smoothly when a user is switching between networks on-the-fly.
Phase 2: Add support for WebTransport, incorporating the benefits of HTTP/3 and QUIC. WebSockets will remain available as a fallback for broader compatibility. This will be implemented in both web player and Nimble Streamer. Nimble Streamer will support WebTransport output alongside WebSockets.
Our ultimate goal is to enable cross-platform playback with the lowest possible latency, using the most effective transport technologies currently available.
We’ll keep informing the audience about further progress of our journey. Stay tuned to our blog and social media to learn more.