aws_topic_history - bruno-beloff/scs_analysis GitHub Wiki

docs > software repositories > scs_analysis > commands > retrieving historic data


DESCRIPTION

The aws_topic_history utility is used to retrieve stored data from the South Coast Science / AWS historic data retrieval system. Data can be retrieved by start or start + end localised datetimes, or by a days / hours / minutes timedelta back in time from now.

Where start and end datetimes are used, messages are retrieved such that:

start datetime <= message rec datetime < end datetime

The --latest mode returns the most recent document, or none if the topic has never received a publication. A --latest-at mode returns the most recent document, or none, prior to, or at the given datetime.

The --rec-only flag causes only the rec fields on the documents to be returned. This results in much faster data retrieval, and is useful if sampling continuity is being tested.

Note that no check is made for the existence of the topic - if the topic does not exist, then no error is raised and no data is returned.

Information on the cloud service used by this utility is at the data API wiki.

The --credentials flag is only required where the user wishes to store multiple identities. Setting the credentials is done interactively using the command line interface.

SYNOPSIS

aws_topic_history.py [-c CREDENTIALS] { -l | -a LATEST_AT [-b BACK-OFF] | -t { [[DD-]HH:]MM[:SS] | :SS } | -s START [-e END] } { -p HH:MM:SS [-m] [-x] | [-w] [-f] } [-r] [{ -v | -d }] TOPIC

Options
--version show program's version number and exit
-h, --help show this help message and exit
-c CREDENTIALS_NAME, --credentials=CREDENTIALS_NAME the stored credentials to be presented
-l, --latest the most recent document
-a LATEST_AT, --latest-at=LATEST_AT the most recent document before ISO 8601 datetime
-b BACK_OFF, --back-off=BACK_OFF maximum look-back period (seconds)
-t TIMEDELTA, --timedelta=TIMEDELTA starting days / hours / minutes / seconds ago, and ending now
-s START, --start=START ISO 8601 datetime START
-e END, --end=END ISO 8601 datetime END
-p CHECKPOINT, --checkpoint=CHECKPOINT a time specification as **:/05:00
-m, --min-max include min and max in addition to midpoint
-x, --exclude-remainder ignore documents after the last complete period
-w, --wrapper include storage wrapper
-f, --fetch-last fetch the last available hour of data if the requested data is unavailable
-r, --rec-only retrieve only the rec field
-v, --verbose report narrative to stderr
-d, --debug report narrative and request / response to stderr

EXAMPLES

aws_topic_history.py south-coast-science-dev/production-test/loc/1/gases -t 1 -v -w

DOCUMENT EXAMPLE - OUTPUT

Without -w flag:

{"val": {"hmd": 68.4, "tmp": 12.3}, "rec": "2019-01-11T12:15:36Z", "tag": "scs-bgx-401"}

With -w flag:

{"device": "scs-bbe-401", "topic": "south-coast-science-demo/brighton/loc/1/climate", "upload": "2019-01-11T12:15:36Z", "payload": {"val": {"hmd": 68.4, "tmp": 12.3}, "rec": "2019-01-11T12:15:36Z", "tag": "scs-bgx-401"}}

SEE ALSO

scs_analysis/aws_byline
scs_analysis/aws_topic_origin
scs_analysis/aws_upload_interval
scs_analysis/cognito_user_credentials

RESOURCES

2: Security

Data API
Data dashboard
Data formats
ISO 8601
https://github.com/curl/curl