GPAC, Toolbox for Interactive Multimedia Packaging, Delivery and Playback
GPAC was born 10 years ago from the need of a lighter and more robust implementation of the MPEG-4 Systems standard , compared to the official reference software. It has since then evolved into a much wider project, covering many tools required when exploring new research topics in multimedia, while keeping a strong focus on international standard coming from organization such as W3C, ISO, ETSI or IETF. The goal of the project is to provide the tools needed to setup test beds and experiments for interactive multimedia applications, in any of the various environments used to deliver content in modern systems: broadcast, multicast, unicast unreliable streaming, HTTP-based streaming and file-based delivery. The project is used in many contexts, from commercial video services production chains to academic research projects, teaching and lab sessions; it allows fast designing of multimedia prototypes, from content production (creation and delivery) to interactive playback (such as augmented reality, video streaming, 2D or 3D scalable drawing) – more details are available in our paper .
Architecture and tools
Most of GPAC functionalities are gathered in a core library called libgpac, allowing building a wide range of applications on top of GPAC without depending on the media player or on the media packagers. Figure 1 shows various GPAC tools and their relationships with other OSS multimedia softwares. All multimedia capabilities needed by the player (access to video display, cameras, sensors such as accelerometers) are abstracted into platform-specific modules.
The core library is written in extremely portable C code, and all modules required by the player have been ported to a wide variety of platforms, from desktop OS (Windows, Linux, OSX) to mobile devices (iOS, Android, Windows CE) or embedded systems (various SH4 and ARM boards, set-top boxes …), as shown in Figure 6. Under constant construction, GPAC is build and bug-checked on a daily basis, thanks to the Buildbot project.
Packaging of Interactive Multimedia Content
- Altering the data layout to enable progressive download and play of the file,
- Streaming a file over RTP  unicast or multicast,
- Creating hint tracks, thereby allowing streaming servers such as the open-source Darwin Streaming Serverto stream the file over RTP without knowledge of the media format,
- Creating and optionally streaming an MPEG-2 Transport Stream  from a set of ISOBMF files.
Figure 3 shows one usage of GPAC tools to generate an MPEG-2 Transport Stream containing interactive BIFS scenes generated live, compatible with the T-DMB standard; the resulting multiplex is streamed over RTP or using HTTP Live Streaming format, and is compatible with traditional MPEG-2 TS player without T-DMB support such as VLC. Recent additions to MP4Box concern the packaging of media content for Adaptive HTTP Streaming, such as HLS, Smooth Streaming or HDS, with a special focus on the new MPEG-DASH standard: different bitrate/resolution versions of the content can be processed together by MP4Box to generate a set of “segmented” files used by these streaming protocols, as explained in . The generated files have been successfully tested with existing MPEG-DASH and other HTTP adaptive streaming players, as well as with the experimental implementation of Media Source Extension (Chrome browser). The segment generation process can be automated in MP4Box to simulate an adaptive streaming live session.
Playback and Rendering of Interactive Multimedia Content
The second set of tools in GPAC deal with content playback and interactive application rendering. The software includes a player, referred to as Osmo4 or MP4Client (command-line version), whose main design is to present media data together with rasterizing vector-based media descriptions, in a synchronized and interactive way. The player may use hardware acceleration such as OpenGL whenever possible (for video composition, 2D or 3D image synthesis).
The player supports many audio and video formats, mainly relying on third party open source software such as ffmpeg, OpenSVC, XVID. It also supports native processing and rendering of scene description formats such as X3D/VRML, SVG and BIFS, thereby enabling a wide range of applications: cartoons, interactive video portals, immersive 3D worlds, augmented reality, “follow-me” applications, etc … It can also perform multi-stereoscopic rendering for many 3D displays types, from old-style anaglyphs to side-by-side or interlaced multiview (as illustrated in Figure 4, and can easily be configured for new displays using OpenGL shaders.
The player has also been used to showcase adaptive video playback through scalable video decoding, highly immersive worlds on 3D displays, augmented reality using embedded camera, streaming media portals (RTP/RTSP, web radios, progressive download, MPEG-2 TS over IP or DVB4Linux, HTTP Adaptive Streaming). Client-side scripting is possible in ECMAScript through Mozilla SpiderMonkey engine, and some usual bindings are provided (DOM subset, XMLHttpRequest support). Plugins can also expose their functionalities through scripting (for example UPnP/DLNA stack can be completely controlled from script).
Our future works will focus on more flexibility in the various tools present in GPAC to simplify creation of streaming media applications by third-party developers, as well as new tools to help the research community in areas such as hybrid networking, ultra-high definition displays and immersive environments.
Being an open-source project, GPAC would not be GPAC without its user community, which we would like to thank for their support, suggestions, bug spotting and fixing, and positive feedback during these years! Especially, many thanks to all software developers who joined our team at Telecom ParisTech and all our students.
- ISO/IEC 14496-1: MPEG-4 Systems
- ISO/IEC 14496-12: ISO Base Media File Format
- RTP: A transport protocol for real-time applications
- ISO/IEC 13818-1: MPEG-2 Systems
- Stockhammer, T. 2011, “Dynamic Adaptive Streaming over HTTP: Design Principles and Standards”, In Proceedings of the second annual ACM conference on Multimedia systems (MMSys11), ACM, New York, NY, USA
- J. Le Feuvre, C. Concolato, J.C. Moissinac. “GPAC: open source multimedia framework”. In Proceedings of the 15th international Conference on Multimedia, Augsburg, Germany, September 25 – 29, 2007. MULTIMEDIA ’07. ACM, New York, NY, 1009-1012. DOI= 10.1145/1291233.1291452