The GBD IoT Framework - andy3471/GBD GitHub Wiki

As a live audio stream is routed through the GBD framework, the GBD library analyzes the audio signals and delivers beat counts to Linux POSIX shared memory. The IoT maker can then use this beat count information to produce synchronized music-to-light events.

To iterate, GBD delivers realtime music DSP analysis data such as beat counts (in addition to routing the PCM signals), nothing more. Creativity and complexity of the sync'd light patterns is the responsibility of the maker.

The GBD framework is primarily designed for the IoT maker industry and community working with commodity hardware.

The GBD framework currently consists of:

• A music DSP analysis library and a GBD server which always run on the same host. This is the GBD core and is based on the GNU/Linux platform.
• A GBD client which routes audio signals from a user's audio player to the GBD library via the GBD server. As such, the GBD client is also referred to as a PCM plugin. The GBD client and the user's audio player may run on a different host machine from that of the GBD core, or may even execute on a separate machine each. In this respect, these modules need not be GNU/Linux based.

The Raspberry Pi was selected as the commodity hardware platform for the GBD core given the board's popularity with the (IoT) maker community. Nevertheless, certain practical limitations with the Raspberry Pi board dictate the following architectural setups:

• The GBD PCM plugin first routes the audio player's signals to the GBD DSP library which analyzes the signals and dumps beat count information in Linux POSIX SHM.

• The audio signal is then routed to the soundcard of the machine hosting the GBD PCM plugin.

• It is up to the IoT maker to implement the light pattern sequencing logic using the GBD beat count info, as well as the physical I/O device drivers, Light/LED interface circuitry, etc.

• Practical limitations of the RPi On-board audio chipset either dictate employing the economy GBD setup, or require using the standalone RPi GBD model with an additional GPIO PiHat soundcard:

  • With the audio routing in the GBD setup, realtime beat detection requires very low audio output latencies if the display of detected beats is to be in sync with the audio. Regardless of how accurate the beat detection is, delays in audio output of a few tenths of a millisecond (e.g. 0.01s), i.e. w.r.t display of beat detection, will produce results that are out-of-sync with with the audio stream and, thus, a poor end-user experience. In other words, the user will see the detection of a beat before the actual beat is heard.

    The RPi on-board audio chipset was not designed for low-latency audio. Even when using a low-level ALSA PCM plugin such as hw for deterministic ALSA period lengths (i.e. ALSA output latency), the on-board device-driver/hardware still performs some buffering of its own which results in additional latency with a noticeable amount of jitter and, say, simply adding some latency "ballast" for the beat analysis does not solve the problem. Also see raspberrypi forums, linuxaudio.org, Raspberrypi, etc for a more detailed discussion.

    Note that the latency issues apply to both the 3.5mm line-out and the HDMI on-board interfaces. Feel free to experiment and prove us wrong! Also recall that the 3.5mm line-out on-board device was not meant for Hi-Fi audio.

  • DJ setups using external hardware mixers not only demand low-latency (both input and output), i.e. a high degree of responsiveness to the DJ's actions, but also will typically employ physical audio routing connectors such as some form of a line-out-to-line-in adapter/cord. The RPi on-board soundcard does not include support for line-in. Alternative RPi audio input sinks for external sound sources such as Bluetooth audio or (most) USB line-in adapters may be fine for home/ regular use, but the input latencies incurred make them impractical for a "party" DJ setup.

  PC's/Laptops generally come equipped with low-latency and HiFi audio interfaces. Alternatively, for a standalone RPi solution, various GPIO (I2S/I2C) PiHat/addon soundcards are available for high quality and low-latency audio. Some of these GPIO H/W addon solutions support inputs as well as outputs, ranging from 2 channels (in/out) to 8 channels out and 6 channels in. See elinux.org, PiHats for a list of available RPi GPIO PiHats. Also see comments in linuxaudio.org, Raspberrypi.

• Economy GBD Setup Issues: TCP vs. UDP

  GBD currently uses TCP rather than UDP sockets for transmission of raw PCM signals between the GBD client (PCM plugin) and GBD server. Although UDP is typically the ideal transport-layer protocol for timely and realtime delivery of signals, robust beat detection also requires reliable delivery of signals. While it is possible to implement heuristics at the application layer for "reliable UDP", e.g. checks for packet sequencing and re-ordering, the bandwidth used for the GBD audio signal routing in a LAN setup does not really warrant this extra code complexity and debugging ... performing generic RT beat detection in a decent manner is already hard enough. A simple check with a network-bandwidth monitor reveals that an average of only 300KB/s is used up with TCP. For instance, see section Network Bandwidth Usage.

  The downside of TCP reliability is that on the rare/unlikely occasion that the LAN "glitches" (notoriously with cheap WiFi chips/access-points), underruns will occur, i.e. momentary breaks in live audio playback. The good news is that these instances are either rare or almost completely avoidable, e.g.:

  • Configure the RPi, or the GBD PCM plugin host, as an access point for a "dedicated" WiFi LAN.
  • Use a decent access point and/or avoid high congestion if the hotspot is a shared resource.
  • Use Ethernet if you suspect and observe network glitches on a bandwidth monitor. For example, see section Network Bandwidth Usage.

  Nevertheless, future development may include switches for UDP-based protocols such as RTP.

For the economy GBD setup, the device hosting the audio application first streams PCM via BT to the PC/laptop. The BT PCM is then routed to the GBD client (PCM plugin) which then routes the audio to the remote GBD server on the RPi via WiFi/Ethernet using TCP/IP.

Otherwise, for standalone RPi setups, the BT audio is streamed directly to the RPi. The entire GBD framework resides within the RPi.

With this approach, the audio player is remote w.r.t GBD framework. An example of this setup in action is presented in GBD with BlueALSA.

Here, a web server provides an interface to a local instance of an audio player for a remote user. In other words, the audio player is remote w.r.t user. Hence, the user's device does not stream PCM but instead sends commands to the web server which then (spawns and) communicates with the audio application. The PCM generated by the audio application is then routed accordingly depending on whether the GBD setup is economy or standalone.

An example configuration can be seen in GBD with MPV/Lua Web Interface.

• Since a Bluetooth setup inescapably involves wireless streaming of audio signals, it is inherently prone to interruptions of sound during playback due to electrical interference, signal loss, etc. Web configurations, on the other hand, provide a way around this problem since the audio player resides locally w.r.t the web server, i.e. no wireless streaming of audio signals is involved -- at least between the audio player and GBD client (PCM plugin). An implicit plus of the web UI approach is that the user device need not have an audio player installed.

• The web based configuration presents power consumption savings for the user's (battery powered) device. Although WiFi generally consumes more power than Bluetooth, the user device for the web based GBD setup only sends requests to the server when necessary. For instance, the device can request the server to initiate playback for a 10hr long playlist (with the audio tracks stored on the server side, of course) and then disconnect. With the Bluetooth setup, the remote audio player device has to constantly stream audio signals for the entire duration.

• An extra advantage with the web based approach is the increased range for user connectivity. WiFi has greater wireless range than Bluetooth. Ethernet makes greater connection distances possible than is with a (single and "residential") WiFi radio. Wired LANs are also much less subject to electrical interference than their WiFi counterparts.

• Nevertheless, the Bluetooth setup allows for seamless audio system integration with the GBD framework. This can be an important factor for the end user (consumer) who expects things to "just work". Firstly, the native audio player on the device eliminates the need for, say, a 3rd party mobile-app or (clumsy) web browser interface. Secondly, once the devices have been paired/bonded, connecting two Bluetooth devices that are within range of each other is fairly automatic/trivial. On the other hand, connectivity and management of (fixed) IP address assignments in WiFi setups may (occasionally) require some configuration effort by the end user. A headless RPi device in these situations makes things even more difficult.

Essentially, any GBD setup where some form of Analog-to-Digital Conversion (ADC) was performed between the audio source and the GBD core qualifies as a capture configuration. For example:

• A GBD setup involving PCM obtained via line-in or on-board mic is by definition a capture setup.

• On the other hand, a GBD setup involving Bluetooth audio capture via an A2DP S/W service running on the RPi BT audio sink, and where the source of the PCM is a remote digital audio player processing a pre-recorded digital audio track is to be viewed as a playback configuration rather than a capture setup.

For reliable GBD operation, playback configurations are generally recommended over capture setups:

• Noisy environments: The PCM obtained in a capture setup involving a microphone will contain noise signals picked up from the surrounding along with the desirable music signals. In contrast, since playback configurations involve only the PCM of a recorded audio track, these GBD setups will work identically in any environment.

• Capture Volume Settings: Careful calibration of capture volume settings is crucial for reliable realtime GBD operation. Establishing proper system volume settings in capture setups can be quite tricky.