sample_stats - bruno-beloff/scs_analysis GitHub Wiki
docs > software repositories > scs_analysis > commands > filtering and aggregating data
DESCRIPTION
The sample_regression utility provides a statical report on a single specified field for documents provided in the input data stream. The specified field(s) normally represent an error or difference between two sources.
If the specified field(s) are not present in any of the input documents, the sample_regression utility terminates. If a field is present but cannot be interpreted as a float, that document is ignored.
The output is a single JSON document. In standard mode, report fields are:
- Document count
- Minimum
- Mean
- Median
- Maximum
- Variance
- First standard deviation
- Second standard deviation
- Third standard deviation
In analytic mode, report fields are:
- Minimum
- Mean
- Maximum
- Lower boundary of 1st standard deviation
- Lower boundary of 2nd standard deviation
- Lower boundary of 3rd standard deviation
- Upper boundary of 1st standard deviation
- Upper boundary of 2nd standard deviation
- Upper boundary of 3rd standard deviation
- Amplitude by 1st standard deviation
- Amplitude by 2nd standard deviation
- Amplitude by 3rd standard deviation
A minimum of two input documents are required.
SYNOPSIS
sample_stats.py [-t] [-p PRECISION] [-a] [-r] [-v] PATH1 [PATH2 .. PATHN]
| Options | |
|---|---|
| --version | show program's version number and exit |
| -h, --help | show this help message and exit |
| -t, --include-tag | include the device tag |
| -p PRECISION, --prec=PRECISION | precision (default 6 decimal places) |
| -a, --analytic | analytic output |
| -r, --rows | output results for each path on a separate row |
| -v, --verbose | report narrative to stderr |
EXAMPLES
csv_reader.py -v Mfi_so2_vE_22Q1_results_err.csv | sample_stats.py -v -p 3 err.SO2.vE.Urban.22Q1 | csv_writer.py -v Mfi_so2_vE_22Q1_results_err_stats.csv
DOCUMENT EXAMPLE - INPUT
{"ref": {"gas": {"SO2": 4.1}}, "SO2": {"vE": {"Urban": {"22Q1": 1.9}}}, "err": {"SO2": {"vE": {"Urban": {"22Q1": -2.2}}}}}
...
DOCUMENT EXAMPLE - OUTPUT
standard mode:
{"tag": "scs-bgx-619", "val": {"VOC": {"cnc": {"count": 4320, "min": 397.7, "mean": 448.2, "median": 446.1,"max": 498.9, "var": 375.6, "stdev": 19.4, "stdev2": 38.8, "stdev3": 58.2}}}}
analytic mode:
{"tag": "scs-bgx-619", "val": {"VOC": {"cnc": {"min": 397.7, "mean": 448.2, "max": 498.9, "l3": 387.9, "l2": 407.3, "l1": 426.7, "u1": 465.5, "u2": 484.9, "u3": 504.3, "a1": 38.8, "a2": 77.6, "a3": 116.4}}}}
rows mode:
{"path": "val.NO2.cnc", "count": 60, "min": 13.1, "mean": 21.0, "median": 20.3, "max": 35.4, "var": 16.3, "stdev": 4.0, "stdev2": 8.0, "stdev3": 12.0}
{"path": "val.Ox.cnc", "count": 60, "min": 98.9, "mean": 110.6, "median": 111.2, "max": 117.5, "var": 17.5, "stdev": 4.2, "stdev2": 8.4, "stdev3": 12.6}
{"path": "val.NO.cnc", "count": 60, "min": 18.1, "mean": 28.1, "median": 25.4, "max": 69.7, "var": 92.0, "stdev": 9.6, "stdev2": 19.2, "stdev3": 28.8}
SEE ALSO
scs_analysis/sample_regression
scs_analysis/sample_slope