Report file format - dzzh/elfstatsd GitHub Wiki

Elfstatsd processes the log records and stores the aggregated statistics in the report files. Per each different set of processed log files, a separate report file is created. This report file is then used by elfstats-munin for sending these data to Munin.

As of elfstatsd-1.8.4, the report file is a text file having a number of sections containing key-value pairs.

Value U that occurs here and there is reported instead of 0. It is needed sometimes to report this value for telling Munin not to discard it from the graphs.

###Metadata

[metadata] section is used to report elfstatsd's state. Its keys are self-descriptive. An example [metadata] section is provided below.

[metadata]
daemon_invoked = 2013-04-23 09:02:12
daemon_version = v1.18.4
first_record = 2013-10-23 08:57:13
last_record = 2013-10-23 09:02:12

###Records data

Information about the total number of log records processed by elfstatsd in the most recent run is stored in records section. It contains the following keys.

• error - the number of records that were not parsed correctly due to their wrong format or other reasons.
• parsed - the number of successfully parsed records that match one of the regexes defined by VALID_REQUESTS setting and are not skipped.
• skipped - the number of records that are valid but match one of the rules in REQUESTS_TO_SKIP setting. They are not counted as parsed.
• total - the total number of processed records. Should equal to the sum of error, parsed and skipped values.

An example [records] section is provided below.

[records]
error = U
parsed = 49535
skipped = 520
total = 50055

###Request-related data

Elfstatsd is a regex-based parser of requests found in the access logs. Each of the valid requests processed by the elfstatsd is assigned with group and method identifiers that are extracted from the request's path. These identifiers are used for aggregation.

Each unique combination of group and method is considered a separate request. Each of these requests has its own section in the report file with a name formed following the template method_<group>_<method>. An extract from a sample report file containing three requests is presented below. It has requests view and purchase from a group shop and add-product request from manager group.

The keys in request-related sections are the following.

calls and stalled_calls contain the number of records containing the request in the processed log file and the number of records with latency exceeding user-defined threshold respectively.

shortest, longest and average represent the aggregated latencies statistics - the shortest, longest and average latencies for the given request that was observed within the tracked time frame.

pXX - user-defined percentiles calculated from the latencies distribution.

rcXXX - number of records matching the request that were responded with a given response code.

[method_shop_view]
calls = 11024
stalled_calls = U
shortest = 5
longest = 482
average = 9
p50 = 8
p90 = 12
p99 = 26
rc200 = 3011
rc404 = 8013
rc500 = U

[method_shop_purchase]
calls = 297
stalled_calls = U
shortest = 98
longest = 3100
average = 177
p50 = 115
p90 = 253
p99 = 1308
rc200 = 297
rc404 = U
rc500 = U

[method_manager_add-product]
calls = 93
stalled_calls = U
shortest = 1
longest = 1397
average = 59
p50 = 35
p90 = 97
p99 = 269
rc200 = 93
rc404 = U
rc500 = U

###Response codes

In addition to providing the response codes-related counters per each request, elfstatsd also reports the total number of matches per each response code that appears in the log (including skipped requests). These information is presented in response_codes section. An example of such section is provided below.

The daemon remembers the codes it processed in the previous runs and reports all of them even if no requests with the matching code was found in the current round. Example: in the previous round the daemon found five requests with response code 304. In this round no such requests were found. However, the daemon will still store rc304=U in the report. If not doing this, Munin will discard line 304 from the response codes graph.

[response_codes]
rc200 = 41205
rc204 = 172
rc301 = 2
rc303 = 119
rc404 = 8556
rc500 = 1

###Patterns

Elfstatsd performs additional extraction of named patterns found in the requests, if this is specified in PATTERNS_TO_EXTRACT setting. Aggregated data for these patterns is stored in [patterns] section. Per each named pattern, two values are reported, namely the total number of pattern matches (key <name>.total) and the total number of distinct matches (key <name>.distinct). An example [patterns] section is presented below.

[patterns]
muid.total = 21573
muid.distinct = 10676