Pipeline configuration (MongoDB, Splunk) - anodot/daria GitHub Wiki

Pipeline config

Pipeline ID - unique pipeline identifier (use a human-readable name so you could easily use it further)
Query file path - path to the file with search query
Count records? - to include the number of records as separate metric, default No
Measurement name - what property for counting records
(Advanced) Is 'what' property static? - if yes, then what property is constant (Measurement name), if no - get value for what property from data
Value columns with target types - key-value pairs separated with spaces, format column_name:target_type column2_name:target_type. A target type represents how samples of the same metric are aggregated in Anodot. Valid values are: gauge (average aggregation), counter (sum aggregation). If what property is not static - instead of an actual target type specify a property name where target type is stored
Measurement properties names - key-value pairs separated with spaces, format column_name:what column2_name:whar2. If what property is not static - instead of an actual what value specify a property name where it is stored.
Timestamp config
- Timestamp property name
- Timestamp property type unix
  - string (must specify format)
  - unix_ms (unix timestamp in milliseconds)
  - unix (unix timestamp in seconds)
  - datetime date type in elasticsearch
- Timestamp format string - if timestamp property type is string - specify format according to this spec. Test here
- Timezone - if the timestamp property type is string you can specify a timezone for it (e.g. Europe/London), UTC by default.
Required dimensions - Names of properties delimited with spaces. If these fields are missing in a record, it goes to error stage. Format - dimension1 dimension2
Optional dimensions - Names of properties delimited with spaces. These fields may be missing in a record. Format - dimension1 dimension2
(Advanced) Additional properties - additional properties with static values to pass to Anodot as dimensions. Format - key1:value1 key2:value2 key3:value3
(Advanced) Tags - tags. Format - key1:value1 key2:value2 key3:value3

If your JSON data has nested structure you can specify path to required field with / sign. For example:


    {
      "amount": 100,
      "transaction": {
        "type": "SALE",
        "status": "SUCCESS"
      }
    }

To specify these fields in pipeline config you can type transaction/type and transaction/status

Example:

> agent pipeline create
Choose source config (elastic_test) [elastic_test]: 
Pipeline ID (must be unique): test
Query file path: query.json
Count records? [y/N]: y
Measurement name: transactions_count
Value columns with target types. Example - property:counter property2:gauge []: amount:gauge duration:gauge
Measurement names. Example -  property:what property2:what2 []: amount:transaction_amount duration:transaction_duration
Timestamp property name: time
Timestamp property type (string, unix, unix_ms) [unix]: unix_ms
Required dimensions [](/anodot/daria/wiki/): transaction/type transaction/status
Optional dimensions [](/anodot/daria/wiki/): location
Created pipeline test
>
> agent pipeline create -a
Choose source config (elastic_test): elastic_test
Pipeline ID (must be unique): test
Query file path: query.json
Count records? [y/N]: y
Measurement name: transactions_count
Is `what` property static?: y
Value columns with target types. Example - property:counter property2:gauge []: value:agg_type
Measurement names. Example -  property:what property2:what2 []: value:name 
Timestamp property name: time
Timestamp property type (string, unix, unix_ms) [unix]: string
Timestamp format string: yyyy-MM-dd'T'HH:mm:ss.SSSZ
Timezone (UTC): Europe/London
Required dimensions [](/anodot/daria/wiki/): dim1 dim2
Optional dimensions [](/anodot/daria/wiki/):
Additional properties []: key1:value1 key2:value2 key3:value3
Tags []: key1:value1 key2:value2 key3:value3
Created pipeline test

Pipeline File config

Properties list

Property	Required	Property name in config file	Value type in config file	Description
Source	yes	source	String	Source config name
Pipeline ID	yes	`pipeline_id`	String	Unique pipeline identifier (use human-readable name so you could easily use it further)
Query file path	Yes	`query_file`	String	Path to the file with a search query
Static what	no	`static_what`	bool	If measurement_name is static
Count records?	no	`count_records`	bool	to include the number of records as separate metric, default false
Measurement name	no	`count_records_measurement_name`	String	`what` property for counting records
Value columns with target types	yes	`values`	Object	Key-value pairs (`target_type`). A target type represents how samples of the same metric are aggregated in Anodot. Valid values are gauge (average aggregation), counter (sum aggregation). If what property is not static - instead of an actual target type specify a property name where the target type is stored
Measurement properties names	yes	`measurement_names`	Object	Key-value pairs (`property:name`). If what property is not static - instead of an actual what value specify a property name where it is stored.
Dimensions	yes	`dimensions`	Object	Dimensions object
Timestamp	yes	`timestamp`	Object	Timestamp object
Additional properties	no	`properties`	Object with key-value pairs	Additional properties with static values to pass to Anodot as dimensions.
Timezone	no	`timezone`	String	A timezone of a timestamp field if its type is string, e.g. Europe/London, default UTC

`Timestamp` object properties:

Property	Type	Description
`type`	String	`string`, `unix` or `unix_ms`
`name`	String	Property name
`format`	String	Specify format if timestamp type is string

Required properties are type and name

`Dimensions` object properties:

Property	Type	Description
`required`	List of strings	These properties are always present in a record
`optional`	List of strings	These properties may be missing in a record

All properties are required

Example

{
    "source": "test_elastic",
    "query_file": "/path/to/file",
    "pipeline_id": "test",
    "count_records_measurement_name": "clicks",
    "count_records": true,
    "values": {"amount": "gauge"},
    "measurement_names": {"amount": "name"},
    "dimensions": {
      "required": ["ver", "Country"],
      "optional": ["Exchange", "optional_dim"]
    },
    "timestamp": {
      "type": "string",
      "name": "timestamp_string",
      "format": "M/d/yyyy H:mm:ss"
    },
    "properties": {"key1": "value1", "key2": "value2", "key3": "value3"},
    "tags": {"key1": "value1", "key2": "value2", "key3": "value3"}
}