Home - acfr/comma GitHub Wiki

Table of Contents comma library: motivation a great divide real time data streams misc library overview downloads release 1.0.5 Linux Windows Building windows packages yourself building from source Linux more details on building from source Windows authors

Being a non-reflective language, C++ notoriously suffers of the great divide between strongly typed structures and serialized data representation.

Representing C++ objects as XML, JSON, CSV, telecom protocol packets, etc, may require a number of libraries, often with convoluted, coupled, and opaque interfaces.

The core purpose of the comma library is to cross this divide in a simple, generic, semantically clear, non-intrusive, and decoupled way.

The comma library serves well 70 per cent of common use cases. The practice shows that the remaining 30 per cent tend to never happen.

comma is not intrusive and imposes minimum infrastructural coupling if any on the code.

The name of the library was suggested by Cedric Wohlleber.

comma was written to handle heavy streams of data in real time but also be immediately useful for offline data processing. It makes implementation of communication protocols easy. Often it allows handling third-party data on the fly. Its tool kit of command-line utilities helps to slice and dice offline and real-time data, both structured (e.g. json) and unstructured (e.g. csv).

Additionally, comma implements various generic concepts that seemed to be missing in other open-source libraries at the time of its implementation.

The core concepts of comma are visiting and visitors. Visiting and visitors are fully decoupled. It is easy to write custom visitors applied to arbitrary C++ classes.

Libraries implementing the core concepts:

• csv: comma-separated data visitors and streams
• visiting: implements core concepts of non-intrusive visiting of arbitrary classes

Other libraries:

• application: helper classes: command line options, simple signal handlers
• base: integral type definitions, exceptions
• bash: bash utilities for parsing command line, process handling, bash variable manipulations, etc
• containers: cyclic buffer, etc
• dispatch: generic double dispath
• io: helper classes and utilities for socket streams, files streams, data publishing, etc
• math: floating point comparison, intervals, cyclic values, etc
• name_value: name-value style visitors and streams (e.g. xml, json)
• packed: classes for easy definition of communication protocol packets
• string: split, join, and other minor string utilities
• xpath: class for xpaths like "/hello/world", "a.b.c[3].d" and operations on them

Python modules:

• csv: python implementation of csv streams

See the doxygen documentation for more: http://acfr.github.com/comma

If building from source, the latest version is usually stable and contains lots of bug fixes and new features. Get that if you want:

 git clone https://github.com/acfr/comma.git

If building from source:

 git clone https://github.com/acfr/comma.git -b 1.0.5

Currently, you need to build release from source (see below)

Pre-built installer: http://perception.acfr.usyd.edu.au/libraries/packages/win/32/comma-1.0.5-win32.exe

1. Download and run the executable, which will unpack include, lib, bin and cmake files.
2. It is recommended to add the full bin path (e.g. c:\libraries\comma\bin) to your system path environment variable.
3. Open a command window (after changing your system path) and type 'csv-from-bin -h', if you see the usage for this app then the installation was successful.
4. The comma applications should all work from the standard command window, however, we suggest installing cygwin (http://www.cygwin.com/) and using that terminal in order to run more complicated pipelines, which may also use linux style utilities and concepts (e.g. cut, grep, mkfifo, etc). This has the added advantage that identical shell scripts can be used in both linux and cygwin environments.
5. Cygwin treats carriage returns differently to linux by default, so for full compatibility with linux bash scripts, add the following two lines to your cygwin ~\.bashrc file:

 export SHELLOPTS
 set -o igncr

See also using comma on windows

You may want to create your own windows packages of comma, e.g. if you built it from the latest source and want to pass to someone as a package.

Here's how.

These pre-built packages are built using Visual Studio 2010 and NSIS, all via cmake/cpack.

1. You will need to install the Nullsoft Scriptable Install System (NSIS). There's nothing to build, just download the installer: http://nsis.sourceforge.net/Main_Page
2. Follow the instructions on installation from source for windows below: i.e. obtain the comma source using git, run cmake, load the .sln into visual studio, and build it.
3. To create the pre-built package, simply find the "PACKAGE" project in Visual Studio (default on the left window pane, e.g. where BUILD and INSTALL projects are located. Right click "PACKAGE" and in the menu that appears next to the mouse, click "BUILD".
4. Once completed, this will have created a package executable file like those available for download from this site. The file is located in your top level build directory.

Currently, only building from source.

Requires boost, at least 1.42; and cmake. There are a couple of optional useful utilities that require libzmq v2.2.0

Run git clone specific release (tags like 1.0), if you don't want the latest stuff.

Something like:

 mkdir ~/src
 cd ~/src
 git clone https://github.com/acfr/comma.git
 mkdir -p ~/build/comma
 cd ~/build/comma
 ccmake ~/src/comma
 make
 sudo make install

You should build comma 'out of source', meaning (as above) the build directory is not a subdirectory of the source.

more details on building from source

Use git, cmake http://www.cmake.org/ and Visual Studio (VS9 and later should work).

You may find 'tortoise git' useful to pull the source: https://code.google.com/p/tortoisegit/ but any git client will do the job.

Build out of source, make a build directory that is not a subdirectory of the source.

Run cmake and point it to the top level source folder (that contains CMakeLists.txt) and select your build environment (e.g. Visual Studio).

Configure and point to the installed locations of your dependencies such as boost.

If you don't need to link to zeromq, don't bother, but if you want to, it compiles on windows with version 2.2.0: http://www.zeromq.org/distro:microsoft-windows. Download the installer, make sure to turn on the "source" option during the installation (off by default) and once installed, add the zeromq bin path to your "path" system environment variable.

Configure, then Generate and cmake will generate a solution file (comma.sln) for Visual Studio. Double click to load it.

In Visual Studio, build the "BUILD_ALL" project.

If successful (no errors), build the "INSTALL" project and the binaries will be placed in the install path chosen in cmake (by default, it installs to 'program files').

Add the bin path (e.g. ...program files\comma\bin) to your system path. To test the installation, refer to the windows downloads section above.

Vsevolod Vlaskine

Cedric Wohlleber