GStreamer

GStreamer is a general multimedia processing framework and has a very broad adoption throughout various operating systems, media players, embedded devices, server products, and browser engines. It is a vital technology that manages the real-time transport of audio and video data in various applications.

The GStreamer Real-Time Transport Protocol (RTP) in Rust project is transitioning critical components of the GStreamer multimedia framework from the programming language C to Rust for enhanced security, maintainability, and sustainability. This low-level protocol is the base of the widely-used Real-Time Streaming Protocol (RTSP) and WebRTC (Web Real-Time Communication) standards. The implementations of these protocols form core components of the GStreamer framework.

RTP is responsible for delivering the actual audio and video data. Its sibling protocol, the Real-Time Control Protocol (RTCP), provides control and feedback on the quality of the data delivery. RTCP is a crucial component in real-time communication systems, contributing to the smooth and synchronized delivery of multimedia content while providing mechanisms for participants to monitor and adapt to network conditions.

The current GStreamer implementations of RTP/RTCP, RTSP, and WebRTC, as well as most other components of GStreamer, are written in the C programming language, and contend with common issues as a result of this. This has lead to a stream of recurring security issues which require maintainer time and attention to be fixed. In order to make this situation more sustainable and attract more contributors, the GStreamer community has already begun to implement dedicated modules and components in Rust, a memory-safe alternative to C.

Despite how important it is to re-implement foundational parts of the framework in a memory-safe language, these components are complex, and the project has been unable to find private-sector stakeholders and partners to support this work.

GStreamer is omnipresent, seamlessly embedded in TVs, smart speakers, video doorbells, CCTV cameras, IoT devices, phones, watches, servers, drones, Blu-ray players, drilling equipment, and desktop/mobile applications. From elevators, cars, ships, trains, and planes to the International Space Station, GStreamer is used in entertainment, security, and even medical devices like surgical microscopes for intricate surgeries.

Its API stability, feature-set, and extensible design have led it to become a fundamental part of other frameworks such as NVIDIA Deepstream, Intel OpenVINO/DLStreamer, Amazon Kinesis, Qt, OpenCV, and many more. GStreamer's hardware-accelerated video support has also established it as a standard in embedded Linux.

GStreamer also has greater significance within the broader context of media frameworks. In today's internet and World Wide Web landscape, media frameworks play a crucial role. However, some video and audio standards remain under the control of private corporations. Investing in the GStreamer project improves the long-term security and maintainability of an open framework for audio and video transport. A free and open source streaming framework (in contrast to privately-controlled ones) preserves the freedom of choice in communication and media broadcasting solutions.

STF has commissioned work from Centricular to enhance the GStreamer multimedia framework by implementing critical components in Rust for improved security, maintainability, and sustainability. These include RTCP parsing, RTP session handling, real-time communication enhancements, and comprehensive support for audio and video formats, ensuring GStreamer's continued versatility and compatibility across a range of applications.

1. Re-writing Foundational GStreamer Components in Rust
   • Real-Time Control Protocol (RTCP) Parsing and Real-Time Transport Protocol (RTP) Session Handling.
   • RTP/AVPF profile support.
2. Real-Time Communication Enhancements
   • GStreamer RTP Jitterbuffer and Synchronization in Rust.
3. Comprehensive Audio and Video Support
   • RTP Payloaders and Depayloaders in Rust for basic audio and video formats.
   • Including RTP support for H.264, H.265, VP8, VP9, and container/meta formats.
4. Re-writing Real-Time Streaming Protocol (RTSP) Handling in Rust
   • GStreamer RTSP protocol client element in Rust, focusing on essential features, live playback, and RTSP 1.0 support