aws_topic_history - bruno-beloff/scs_analysis GitHub Wiki

docs > software repositories > scs_analysis > commands > retrieving historic data

DESCRIPTION

The aws_topic_history utility is used to retrieve stored data from the South Coast Science / AWS historic data retrieval system. Data can be retrieved by start or start + end localised datetimes, or by a days / hours / minutes timedelta back in time from now.

Where start and end datetimes are used, messages are retrieved such that:

start datetime <= message rec datetime < end datetime

The --latest mode returns the most recent document, or none if the topic has never received a publication. A --latest-at mode returns the most recent document, or none, prior to, or at the given datetime.

The --rec-only flag causes only the rec fields on the documents to be returned. This results in much faster data retrieval, and is useful if sampling continuity is being tested.

Note that no check is made for the existence of the topic - if the topic does not exist, then no error is raised and no data is returned.

Information on the cloud service used by this utility is at the data API wiki.

The --credentials flag is only required where the user wishes to store multiple identities. Setting the credentials is done interactively using the command line interface.

SYNOPSIS

aws_topic_history.py [-c CREDENTIALS] { -l | -a LATEST_AT [-b BACK-OFF] | -t { [[DD-]HH:]MM[:SS] | :SS } | -s START [-e END] } { -p HH:MM:SS [-m] [-x] | [-w] [-f] } [-r] [{ -v | -d }] TOPIC

Options
--version	show program's version number and exit
-h, --help	show this help message and exit
-c CREDENTIALS_NAME, --credentials=CREDENTIALS_NAME	the stored credentials to be presented
-l, --latest	the most recent document
-a LATEST_AT, --latest-at=LATEST_AT	the most recent document before ISO 8601 datetime
-b BACK_OFF, --back-off=BACK_OFF	maximum look-back period (seconds)
-t TIMEDELTA, --timedelta=TIMEDELTA	starting days / hours / minutes / seconds ago, and ending now
-s START, --start=START	ISO 8601 datetime START
-e END, --end=END	ISO 8601 datetime END
-p CHECKPOINT, --checkpoint=CHECKPOINT	a time specification as **:/05:00
-m, --min-max	include min and max in addition to midpoint
-x, --exclude-remainder	ignore documents after the last complete period
-w, --wrapper	include storage wrapper
-f, --fetch-last	fetch the last available hour of data if the requested data is unavailable
-r, --rec-only	retrieve only the rec field
-v, --verbose	report narrative to stderr
-d, --debug	report narrative and request / response to stderr

EXAMPLES

aws_topic_history.py south-coast-science-dev/production-test/loc/1/gases -t 1 -v -w

DOCUMENT EXAMPLE - OUTPUT

Without -w flag:

{"val": {"hmd": 68.4, "tmp": 12.3}, "rec": "2019-01-11T12:15:36Z", "tag": "scs-bgx-401"}

With -w flag:

{"device": "scs-bbe-401", "topic": "south-coast-science-demo/brighton/loc/1/climate", "upload": "2019-01-11T12:15:36Z", "payload": {"val": {"hmd": 68.4, "tmp": 12.3}, "rec": "2019-01-11T12:15:36Z", "tag": "scs-bgx-401"}}

RESOURCES

2: Security

Data API
Data dashboard
Data formats
ISO 8601
https://github.com/curl/curl