aws_upload_interval - bruno-beloff/scs_analysis GitHub Wiki
docs > software repositories > scs_analysis > commands > Retrieving historic data
DESCRIPTION
The aws_upload_interval utility is used to determine the end-to-end health of the data infrastructure. Specifically, it reports on the time difference between a sense document being created on a device, and the same document being ingested by the data storage system. Time difference is represented as:
DAYS-HOURS:MINUTES:SECONDS
The aws_upload_interval utility relies on the data format provided by the aws_topic_history utility - when invoked with its -w ("wrapper") flag, the aws_topic_history output includes an "upload" field, indicating the datetime at which the document was processed. The sense time is provided in the "payload.rec" field. The aws_upload_interval utility reports on the difference between the two datetimes in seconds.
The utility accepts input from stdin. If an input document is empty or is not well-formed JSON, the document is ignored. If the document is well-formed JSON, but does not contain the required "upload" and "payload.rec" fields or the fields are not in ISO 8601 format, then an error is reported, and the aws_upload_interval utility terminates.
SYNOPSIS
aws_upload_interval.py
EXAMPLES
aws_topic_history.py unep/ethiopia/loc/1/climate -m1440 -w | aws_upload_interval.py
DOCUMENT EXAMPLE - INPUT
{"device": "scs-bbe-501", "topic": "unep/ethiopia/loc/1/climate", "upload": "2018-12-24T13:07:03Z", "payload": {"val": {"hmd": 36.5, "tmp": 25.5}, "rec": "2018-12-24T13:07:01Z", "tag": "scs-bgx-501"}}
DOCUMENT EXAMPLE - OUTPUT
{"upload": "2018-12-24T13:07:03Z", "rec": "2018-12-24T13:07:01Z", "offset": "00-00:00:02"}
SEE ALSO
scs_analysis/aws_topic_history