Transformations - anodot/daria GitHub Wiki

• Transformation configuration
• Supported functions
• Change field values depending on a condition
• Set dimensions by applying a function to a value

Transformation configuration is a JSON object that might contain one of two keys: file or config.

The file field contains the path to a transformation CSV file, the config field contains the transformations itself in the comma-separated format. Example:

[{
    "pipeline_id": "pipeline",
    ...
    "transform": {
      "file": "/usr/src/app/agent-data/pipelines/cacti_transform.csv"
    }
}]

/usr/src/app/agent-data/pipelines/cacti_transform.csv file example:

country_code,AU,'country' == 'Australia'
country_code,RU,'country' == 'New Zeland'

File structure:

• Column 1 - field to write value to
• Column 2 - a value to write (can be a constant or result of one of the transformation function, see the list further below)
• Column 3 - a condition. When the result is true, the transformation will be done, if false - ignored. Specifying the condition is optional. If condition is not specified the transformation will always be applied

All transformations are applied in the order they appear in the file

Instead of a file transformation rules can be specified directly in the json config like this:

[{
    "pipeline_id": "pipeline",
    ...
    "transform": {
      "config": "dimensions/host_name,zeus.snmplabs.com,'dimensions/host_name' == 'value=zeus.snmplabs.com'"
    }
}]

You can use these functions to transform your data:

str:regExCapture(<string>, <regEx>, <group>)
str:replace(<string>, <oldChar>, <newChar>)
str:replaceAll(<string>, <regEx>, <newString>)
str:substring(<string>, <beginIndex>, <endIndex>)
str:toLower(<string>)
str:toUpper(<string>)
str:trim(<string>)
str:truncate(<string>, <length>)

==
!=
contains
startsWith
endsWith
matches

For combining multiple conditions && and || operators can be used as well as a negation operator !.

Please note that keeping spaces between literals is very important!

Condition examples:

"prop2" matches "[a-z]+"
"prop" == "val" && !(prop2' contains 'val2)
'prop' == 'val' || 'prop2' contains 'val2'

Input data:

[
  {"country": "Australia", "value": 1},
  {"country": "Russia", "value": 1},
]

Transformations file:

country_code,AU,'country' == 'Australia'
country_code,RU,'country' == 'New Zeland'

After the transformations are applied, the data will contain an additional field named country_code:

[
  {
    "properties": {"country_code": "AU", "country": "Australia", "target_type": "counter", "what": "count"},
    "value": 1
  },
  {
    "properties": {"country_code": "NZ", "country": "New Zeland", "target_type": "counter", "what": "count"},
    "value": 1
  }
]

Another option is to specify a resulting field name and the function the result of which will be set into it.

Input data:

[
  {"name": "host01 - memory.consumption - avg.day", "value": 3438.5},
]

Transformation:

title,"str:regExCapture(name, '(.+)\\s-\\s(.+)\\s-\\s(.+)', 2)"

This code means that the dimension title will be created (if it doesn't exist), and the function regExCapture() will be applied to the field name and the second match, which is memory.consumption, will be set to the title dimension

Result:

[
  {
    "properties": {"name": "host01 - memory.consumption - avg.day", "title": "memory.consumption", "what": "value"},
    "value": 3438.5
  }
]