aws_topic_history - bruno-beloff/scs_analysis GitHub Wiki
docs > software repositories > scs_analysis > commands > retrieving historic data
DESCRIPTION
The aws_topic_history utility is used to retrieve stored data from the South Coast Science / AWS historic data retrieval system. Data can be retrieved by start or start + end localised datetimes, or by a days / hours / minutes timedelta back in time from now.
Where start and end datetimes are used, messages are retrieved such that:
start datetime <= message rec datetime < end datetime
The --latest mode returns the most recent document, or none if the topic has never received a publication. A --latest-at mode returns the most recent document, or none, prior to, or at the given datetime.
The --rec-only flag causes only the rec fields on the documents to be returned. This results in much faster data retrieval, and is useful if sampling continuity is being tested.
Note that no check is made for the existence of the topic - if the topic does not exist, then no error is raised and no data is returned.
Information on the cloud service used by this utility is at the data API wiki.
The --credentials flag is only required where the user wishes to store multiple identities. Setting the credentials is done interactively using the command line interface.
SYNOPSIS
aws_topic_history.py [-c CREDENTIALS] { -l | -a LATEST_AT [-b BACK-OFF] | -t { [[DD-]HH:]MM[:SS] | :SS } | -s START [-e END] } { -p HH:MM:SS [-m] [-x] | [-w] [-f] } [-r] [{ -v | -d }] TOPIC
Options | |
---|---|
--version | show program's version number and exit |
-h, --help | show this help message and exit |
-c CREDENTIALS_NAME, --credentials=CREDENTIALS_NAME | the stored credentials to be presented |
-l, --latest | the most recent document |
-a LATEST_AT, --latest-at=LATEST_AT | the most recent document before ISO 8601 datetime |
-b BACK_OFF, --back-off=BACK_OFF | maximum look-back period (seconds) |
-t TIMEDELTA, --timedelta=TIMEDELTA | starting days / hours / minutes / seconds ago, and ending now |
-s START, --start=START | ISO 8601 datetime START |
-e END, --end=END | ISO 8601 datetime END |
-p CHECKPOINT, --checkpoint=CHECKPOINT | a time specification as **:/05:00 |
-m, --min-max | include min and max in addition to midpoint |
-x, --exclude-remainder | ignore documents after the last complete period |
-w, --wrapper | include storage wrapper |
-f, --fetch-last | fetch the last available hour of data if the requested data is unavailable |
-r, --rec-only | retrieve only the rec field |
-v, --verbose | report narrative to stderr |
-d, --debug | report narrative and request / response to stderr |
EXAMPLES
aws_topic_history.py south-coast-science-dev/production-test/loc/1/gases -t 1 -v -w
DOCUMENT EXAMPLE - OUTPUT
Without -w flag:
{"val": {"hmd": 68.4, "tmp": 12.3}, "rec": "2019-01-11T12:15:36Z", "tag": "scs-bgx-401"}
With -w flag:
{"device": "scs-bbe-401", "topic": "south-coast-science-demo/brighton/loc/1/climate", "upload": "2019-01-11T12:15:36Z", "payload": {"val": {"hmd": 68.4, "tmp": 12.3}, "rec": "2019-01-11T12:15:36Z", "tag": "scs-bgx-401"}}
SEE ALSO
scs_analysis/aws_byline
scs_analysis/aws_topic_origin
scs_analysis/aws_upload_interval
scs_analysis/cognito_user_credentials
RESOURCES
Data API
Data dashboard
Data formats
ISO 8601
https://github.com/curl/curl