Introduction

Taken from the GStreamer main page:

What is GStreamer?

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.

Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface.

GStreamer is released under the LGPL.

For more information you can go here

Architecture Overview

GStreamer is a library written in C which utilizes GObject and GLib extensively. It provides language bindings in multiple languages, including Python and C++. It has support for multiple platforms including Windows, Linux, Android, iOS and OSX.

Design overview

GStreamer handles the processing of media with the concept of a media pipeline. A pipeline is made up of a series of elements that handle the creation, processing and presentation of different kinds of media data. Media can be of any form, be it audio, video, text or any other type of data. Because elements know what kind of data they can accept as input and what data they produce, communication between elements can be negotiated even dynamically during runtime, allowing for complex and dynamic pipelines.

As previously stated, a pipeline comprises of a series of elements. There are 3 types of elements:

• Sources: They produce media data. For example, this can be generated, read from disk, from the network or from a device.
• Filters elements. These take data and perform certain operations on the data. These can take multiple forms and can have one or more import and one or more outputs.
• Sinks: These are data endpoints for a media pipeline. This can be for instance, a device, like a screen of a speaker or it can be output into a network.

Elements use Pads to communicate data to one another. Pads connect to one another and handle the data transfer between elements as well as the resolution of what format the data being transfered will have. An example of a simple pipeline is the following:

In the example we see 3 elements: The source is colored in red, the filter in green and the sink in blue. In this example we are capturing images using a source that can read from a v4l2 device. The data is being output to X using a sink that handles image presentation via X. So what does the filter do? if we look at the data flow going in and out of the filter we see that there is information on the type of data being passed between the elements. In this case we will see that the format going into the filter is YUY2 and the output is BGRx. X expects data to be in xBGR, however the device in this case cannot produce data in that format. We need an element that will transform the data into the correct form for the sink to show it. For this reason we use a 'videoconvert' element.

Installation

Installing GStreamer on Linux can be done using the package manager, in our case apt. All you need to do is:

apt-get update
apt-get install gstreamer1.0

This will install the GStreamer core elements and tools. However, you will need the actual elements to start development. There are 4 additional packages for that:

apt-get install gstreamer1.0-plugins-base
apt-get install gstreamer1.0-plugins-good
apt-get install gstreamer1.0-plugins-ugly
apt-get install gstreamer1.0-plugins-bad

These packages have a large range of elements that may not all be necessary for your application. These packages are made up of smaller packages, each containing a small subset of common elements. If you know which elements you need you can always look for the specific packages containing the elements you need and install those instead.

Command line tools

GStreamer comes with a series of command line tools for aiding in developing multimedia applications. We will go over a couple:

gst-launch

This allows you to test different pipelines by passing a string representation of the target pipeline. The example show before was generated from the following command line:

gst-launch-1.0 v4l2src device=/dev/qt5023_video1 ! videoconvert ! ximagesink

Elements are connected with exclamation marks ('!'), elements with spaces between each other are not connected. in the aboive example there is the additional string: device=/dev/qt5023_video1. Elements have properties that can change their behavior. Properties always follow the element they are affecting. In this case, we need the v4l2src to point to the correct device to perform video capture. In the QTec camera the pipeline was run on, that was at /dev/qt5023_video1.

##gst-inspect

So far we have seen elements, properties, data negotiation. However we have not seen how we can look up this information. For this we have gst-inspect:

gst-inspect-1.0 v4l2src

Whoa! that is a lot of information! The information for an element is similar to the information that is on the official GStreamer plugin documentation. so you can always use that instead. There is a lot of details, too much for this tutorial, but a simple overlook:

• Factory and Plugin details: Information like version number, author, installation dir are located here.
• Pad templates: These are the formats that the element can support. The key things to remember starting off: data leaving an element comes from a source, or 'src'. Data entry points to an element are 'sink's. Pay attention to the details of the supported caps. Just because a specific cap may be supported does not mean it will! E.g. for v4l2src there is a wide range of properties. However, it is the device that it connect to that limits the range of options.
• Pads: These are the actual pads that the element provides.
• Element properties: The list of properties that can be changed for the element.
• Element Signals: Special signals that the application can connect to.

#For more information

For a more detailed overview, frequently asked questions, tutorials and the developer manual, you should check the official documentation. If you feel like trying to write an element of your own, we suggest starting with the Advanced GStreamer Tutorial.