csv_logger - bruno-beloff/scs_dev GitHub Wiki

docs > software repositories > scs_dev > commands > file-based data management

DESCRIPTION

The csv_logger utility is used to provide continuous logging of data. For devices that are not internet-connected, this can be used as the method of data capture. For always-connected devices, it is recommended that the utility is used to provide a backup facility. This is because, in normal operations, data that is queued for publishing is held in volatile memory.

The operation of the csv_logger is specified using the csv_logger_conf utility - this specifies the filesystem location for logging, together with the logging mode of operation.

The csv_logger receives JSON data on stdin and writes this to the log file. The log file is named for its topic and the datetime of the first JSON document reception. Log files are closed - and a new log file opened - each day after 00:00 UTC. All logging datetimes are UTC, irrespective of the system or application timezone. Log files are stored in directories named for the year and month.

If the system ID is set (using the scs_mfr/system_id utility) then log files are prepended with the device tag. Otherwise, the log file name begins with the datetime.

Like the csv_writer utility, the csv_logger converts data from JSON format to comma-separated value (CSV) format. The path into the JSON document is used to name the column in the header row, with JSON nodes separated by a period ('.') character.

All the leaf nodes of the first JSON document are included in the CSV. If subsequent JSON documents in the input stream contain fields that were not in this first document, these extra fields are ignored.

If the --echo flag is used, then the csv_logger utility writes the received data to stdout. When the utility starts, the historic data API is queried to find the most recent record for the topic. The log reader child process then outputs any data on the device that is missing from the data store before outputting new records. Valid AWS API credentials and configuration are required.

Note: Note: echoing cannot begin until a network connection has been established.

SYNOPSIS

csv_logger.py [-a] [-e] [-v] TOPIC

Options
--version	show program's version number and exit
-h, --help	show this help message and exit
-a, --absolute	absolute topic path (default is find from AWS project)
-e, --echo	echo stdin to stdout (mediated by storage and byline)
-v, --verbose	report narrative to stderr

EXAMPLES

./socket_receiver.py | ./csv_logger.py -e climate

FILES

dependent on csv_logger_conf

DOCUMENT EXAMPLE - INPUT

{"tag": "scs-ap1-6", "rec": "2018-04-04T14:50:27.641+00:00", "val": {"hmd": 59.6, "tmp": 23.8}}

DOCUMENT EXAMPLE - FILE CONTENTS

tag,rec,val.hmd,val.tmp
scs-ap1-6,2018-04-04T14:50:38.394+00:00,59.7,23.8

BUGS

If any filesystem problem is encountered then the reader child process is stopped, and the utility directly echoes stdin to stdout from then on. No attempt is made to restart the reader process.