Data Transforms - nyurik/vega GitHub Wiki

This wiki documents Vega version 2. For Vega 3 documentation, see vega.github.io/vega.

Wiki ▸ Documentation ▸ Data Transforms

A data transform performs operations on a data set prior to visualization. Common examples include filtering and grouping (e.g., group data points with the same stock ticker for plotting as separate lines). Other examples include layout functions (e.g., stream graphs, treemaps, graph layout) that are run prior to mark encoding.

All transform definitions must include a type parameter, which specifies the transform to apply. Each transform then has a set of transform-specific parameters. Transformation workflows are defined as an array of transforms; each transform is then evaluated in the specified order.

These workflows can be specified as part of either a dataset definition:

  "data":[
    {
      "name": "stats",
      "source": "table",
      "transform": [
        {"type": "facet", "groupby": ["x"]}
      ]
    }
  ]

or added to a mark's from property:

  "marks": [
    {
      "type": "group",
      "from": {
        "data": "fields",
        "transform": [{"type": "cross"}]
      },
      ...
    }
  ],

Many transforms accept data attributes, or fields (denoted below as "Field"), as parameters. Data field parameters are strings that describe either individual attributes (e.g., "price") or access paths (e.g., "price.min"). Data fields can also access array indices, for example "list[0]".

These transforms can be used to manipulate the data: to filter and sort elements, form groups, and merge different data sets together.

Data Manipulation Transforms:

• aggregate - Computes aggregate summary statistics (e.g., median, min, max) over groups of data.
• bin - Bins raw data values into quantitative bins (e.g., for a histogram).
• countpattern - Counts the number of occurrences of a text pattern, as defined by a regular expression.
• cross - Compute the cross-product of two data sets.
• facet - Organizes a data set into groups or "facets".
• filter - Filters elements from a data set to remove unwanted items.
• fold - Collapse ("fold") one or more data properties into two properties: a key property (containing the original data property name) and a value property (containing the data value).
• formula - Extends data elements with new values according to a calculation formula.
• impute - Performs imputation of missing values.
• lookup - Extends a primary data set by looking up values on a secondary data set.
• rank - Computes an ascending rank score for each data tuple.
• sort - Sorts the values of a data set.
• treeify - Computes a tree structure over a flat tabular dataset.

Computes aggregate summary statistics (e.g., median, min, max) over groups of data.

The short format for summarize uses a single object hash that maps from field names to one or more aggregation operations: {"foo": "mean", "bar": ["sum", "median"]}. The aggregation operation can be either a single string or an array of strings, each a valid aggregation operation name.

The long format for summarize uses an array of aggregate specification objects. The previous short format example translates to the following long format: [{"field": "foo", "ops": ["valid"]}, {"field": "bar", "ops": ["sum", "median"]}]

An aggregate specification supports the following properties:

The supported aggregation operations are:

Many of the aggregation functions above are straightforward, but a few deserve additional discussion.

The 'values' and 'count' functions operate directly on the input objects and return the same value regardless of the provided field name. Similar to SQL's count(*), these can be specified with the special name "*", as in "summarize": {"*": "count"}.

The 'argmin' and 'argmax' functions are a bit unusual: instead of returning the minimum or maximum value of a field, they return the original input object that contains the minimum or maximum value. This can be useful for retrieving another field associated with the minimum or maximum value (e.g., for each region, in which year did I have the maximum revenue?). If multiple entries share the minimum or maximum value, the first observed input object will be returned.

Output

The aggregate transform outputs a new array of data objects, one for each group, with the computed aggregate statistics.

Example

For the following input data:

[{"foo": 1, "bar": 1}, {"foo": 1, "bar": 2}, {"foo": null, "bar": 3}]

This short format aggregate transform

{"type": "aggregate", "summarize": {"foo": "valid", "bar": ["sum", "median"]}}

would produce the following output:

[{"valid_foo": 2, "sum_bar": 6, "median_bar": 2}]

Similarly, this long format aggregate transform:

{
  "type": "aggregate",
  "summarize": [
    {"field": "foo", "ops": ["valid"]},
    {"field": "bar", "ops": ["sum", "median"], "as": ["s", "m"]}
  ]
}

would produce the following output:

[{"valid_foo": 2, "s": 6, "m": 2}]

Bins raw data values into quantitative bins (e.g., for a histogram).

Output

The bin transform sets the following values on each node datum:

These properties may be renamed by specifying an output map. For example, "output": {"start": "bs", "end": "be"} will use bs and be as the output properties instead of bin_start and bin_end.

Example

{"type": "bin", "field": "amount", "min": 0, "max": 10, "maxbins": 5}

This example will bin values in the amount field into one of 5 bins between 0 and 10. Given the following input data:

[
  {"amount": 3.7},
  {"amount": 6.2},
  {"amount": 5.9},
  {"amount": 8}
]

The bin transform produces the following output:

[
  {"amount": 3.7, "bin_start": 2, "bin_end": 4},
  {"amount": 6.2, "bin_start": 6, "bin_end": 8},
  {"amount": 5.9, "bin_start": 4, "bin_end": 6},
  {"amount": 8, "bin_start": 8, "bin_end": 10},
]

Counts the number of occurrences of a text pattern, as defined by a regular expression. The countpattern transform will iterate through each data object and count all unique pattern matches found within the designated text field.

Both the pattern and stopwords parameters below are not "raw" regular expression patterns – they are embedded in a string. As a result, take care to make sure you use proper escape characters as needed (e.g., to match digits, use "\\d", not "\d").

Output

The countpattern transform returns a new array of data objects, with two properties: text (a text segment that matches the pattern), and count (the number of observed occurrences of the pattern). The names of these new properties can be changed by setting the transform's output map. For example, the parameter "output": {"text": "t", "count": "c"} causes the properties t and c to be used instead of text and count.

Example

{
  "type": "countpattern",
  "field": "comment",
  "pattern": "\\d+",
  "stopwords": "13"
}

This example counts the occurrences of each digit sequence in the comment field, except for the number 13.

Running the transform on this input data:

[
  {"comment": "between 12 and 12.43"},
  {"comment": "43 minutes past 12 o'clock (and 13 seconds)"}
]

will produce the following output:

[
  {"text": "12", "count": 3},
  {"text": "43", "count": 2},
]

Compute the cross-product of two data sets.

Output

The cross transform outputs an array of data objects with two properties: one containing an item from the primary data set (named a by default), and an item from the secondary data set (named b by default). These names can be changed by setting the transform's output map. For example, the parameter "output": {"left":"thing1", "right":"thing2"}, causes the properties thing1 and thing2 to be used instead of a and b.

Example

{"type": "cross", "diagonal": false}

This example crosses a data set with itself, ignoring entries with the same indices (e.g., along the diagonal). If the input data is [1, 2, 3], then the cross transform will output:

[
  {"a":1, "b":2},
  {"a":1, "b":3},
  {"a":2, "b":1},
  {"a":2, "b":3},
  {"a":3, "b":1},
  {"a":3, "b":2},
]

Similarly, with the same input data, the following

{"type": "cross", "filter": "datum.a <= 2 && datum.b >= 2"}

will produce:

[
  {"a":1, "b":2},
  {"a":1, "b":3},
  {"a":2, "b":2},
  {"a":2, "b":3}
]

Organizes a data set into groups or "facets". The facet transform is useful for creating collections of data that are then passed along to group marks to create hierarchical structure in a visualization. It can also be used (like the aggregate transform) to compute descriptive statistics over subgroups of data. In this sense, it is similar to a "group by" operation in SQL.

Output

The facet transform returns a transformed data set organized into facets. Vega uses a standardized data structure for representing hierarchical or faceted data, which consists of a hierarchy of objects with properties for each key and a values array containing all data objects in the facet.

Example

{"type": "facet", "groupby": ["category"]}

Facets the data according to the values of the category attribute. Given the following input data:

[
  {"category":"A", "value":0},
  {"category":"B", "value":1},
  {"category":"A", "value":2},
  {"category":"B", "value":3}
]

The facet transform produces a hierarchical collection of data arrays:

[
  {
    "category": "A",
    "key": "A",
    "values": [{"category":"A", "value":0}, {"category":"A", "value":2}]
  },
  {
    "category": "B",
    "key": "B",
    "values": [{"category":"B", "value":1}, {"category":"B", "value":3}]
  }
]

Each facet (group) is bundled into a "sentinel" object that includes:

• The original key properties and values (category in this example).
• A string concatenating all key values (key). This can be useful in conjunction with ordinal scales.
• The array of grouped data objects (values).

When a faceted data set is the input data for a group mark, Vega will automatically lookup the values array and pass it down as the data source for any marks contained within each enclosing group mark instance.

Filters elements from a data set to remove unwanted items.

Output

The filter transform returns a new data set containing only elements that match the filter test.

Examples

{"type": "filter", "test": "datum.x > 10"}

This example retains only data elements for which the field x is greater than 10.

{"type": "filter", "test": "log(datum.y)/LN10 > 2"}

This example retains only data elements for which the base-10 logarithm of y is greater than 2.

Collapse ("fold") one or more data properties into two properties: a key property (containing the original data property name) and a value property (containing the data value). The fold transform is useful for mapping matrix or cross-tabulation data into a standardized format.

Output

The fold transform returns a new array of data objects, with two additional properties: key (an extracted property name), and value (an extracted data value). The names of these new properties can be changed by setting the transform's output map. For example, the parameter "output": {"key": "k", "value": "v"} causes the properties k and v to be used instead of key and value.

Example

{"type": "fold", "fields": ["gold", "silver"]}

This example folds the gold and silver properties. Given the following input data:

[
  {"country": "USA", "gold": 10, "silver": 20}, 
  {"country": "Canada", "gold": 7, "silver": 26}
]

this example will produce the following output:

[
  {"key": "gold", "value": 10, "country": "USA", "gold": 10, "silver": 20},
  {"key": "silver", "value": 20, "country": "USA", "gold": 10, "silver": 20},
  {"key": "gold", "value": 7, "country": "Canada", "gold": 7, "silver": 26},
  {"key": "silver", "value": 26, "country": "Canada", "gold": 7, "silver": 26}
]

Extends data elements with new values according to a calculation formula.

Output

The formula transform returns the input data set, with each element extended with the computed formula value on the field property.

Examples

{"type": "formula", "field": "logx", "expr": "log(datum.x)/LN10"}

This example computes the base-10 logarithm of x and stores the result on each datum as the "logx" property.

{"type": "formula", "field": "hr", "expr": "hours(datum.date)"}

This example extracts the hour of the date field, and stores the result on each datum as the hr property.

Performs imputation of missing values.

Output

The impute transform returns the input data set, with additional imputed tuples for missing values.

Examples

{
  "data": [{
    "name": "table",
    "values": [
      {"x": 0, "y": 28, "c":0}, {"x": 0, "y": 55, "c":1},
      {"x": 1, "y": 43, "c":0}, {"x": 1, "y": 91, "c":1},
      {"x": 2, "y": 81, "c":0}, {"x": 2, "y": 53, "c":1},
      {"x": 3, "y": 19, "c":0}
    ],
    "transform": [
      {
        "type": "impute",
        "groupby": ["c"],
        "orderby": ["x"],
        "field": "y",
        "method": "value",
        "value": 500
      }
    ]
  }]
}

In this example, if the transform would impute the following tuple:

{"x": 3, "c": 1, "y": 500}

Extends a primary data set by looking up values on a secondary data set. In other words, performs a join that adds new properties onto the primary data set only. Lookup accepts one or more key values for the primary data set, each of which are then searched for within a single key field of the secondary data set. If a match is found, the full data object in the secondary data set is added as a property of the primary data set.

Output

The lookup transform extends the primary data set with matching records in the secondary (on) data set, and stores the value from the secondary data in a field specified by the as parameter.

The use of lookup often results in the need for nested property accessors. For example, if the primary data object {key: 'a', count: 5} is used to lookup {id: 'a', value: 3.14} and the as parameter is ["obj"], the result will be: {key: 'a', count:5, obj: {id: 'a', value: 3.14}}. To access the extended value property, use obj.value as the field name.

Example

{
  "type": "lookup",
  "on": "unemployment",
  "onKey": "key",
  "keys": ["id"],
  "as": ["value"],
  "default": null
}

This example matches records in the input data with records in the data set named "unemployment", where the values of id (primary data) and key (secondary data) match. Matching records in the secondary data are added to the primary data in the field named "value".

Computes an ascending rank score for data tuples based on their observed order and any key fields. This is particularly useful for sorting ordinal scales by multiple key fields.

Output

The rank transform extends the primary data set with an additional field called rank. This property's name can changed by setting the transform's output map. For example, the parameter "output": {"rank": "r"} causes the name r to be used instead of rank.

Example

With the following snippet of a Vega specification

{
  "data": [{
    "name": "table",
    "values": [
      {"x": "A","y": 12}, {"x": "A","y": 32},
      {"x": "B","y": 6},  {"x": "B","y": 35},
      {"x": "C","y": 19}, {"x": "C","y": 66}
    ],
    "transform": [
      {"type": "sort","by": ["y"]},
      {"type": "rank","field": "x"}
    ]
  }]
}

the table data set will contain the following tuples:

[
  {"x":"B", "y":6,  "rank":1},
  {"x":"A", "y":12, "rank":2},
  {"x":"C", "y":19, "rank":3},
  {"x":"A", "y":32, "rank":2},
  {"x":"B", "y":35, "rank":1},
  {"x":"C", "y":66, "rank":3}
]

Sorts the values of a data set. This is useful if subsequent transforms are order dependent (e.g., with a visual layout) or to determine rendering order of mark items. The sort order of an ordinal scale's domain can be specified on the scale directly.

Output

The sort transform returns the input data set with elements sorted in place. Note: Sorting the elements on a data set will not affect the order of values in an ordinal scale. The latter must be specified on the scale directly.

Example

{"type": "sort", "by": "-_id"}

This example sorts a data set in descending order by the value of the _id field.

Computes a tree structure over a flat tabular dataset.

Output

The treeify transform computes a tree structure by adding children and parent pointers to every tuple. The names of these new properties can be changed by setting the transform's output map. For example, the parameter "output": {"children": "c", "parent": "p"} causes the properties c and p to be used instead of children and parent.

Visual encoding transforms can be used to create more advanced visualizations, including layout algorithms and geographic projections.

Visual Encoding Transforms: force, geo, geopath, hierarchy, linkpath, pie, stack, treemap, voronoi, wordcloud

Performs force-directed layout for network data. Force-directed layouts treat nodes as charged particles and edges (links among nodes) as springs, and uses a physics simulation to determine node positions. The force transform acts on two data sets: one containing nodes and one containing links. Apply the transform to the node data, and include the name of the link data as a transform parameter.

Note that the force transform modifies the nodes data only. It does not modify any properties of the links data. To layout the links, elsewhere you can use a lookup transform to join the node data onto the links data and a linkpath transform to route the edge path.

Output

The force transform sets the following values on each node datum:

These properties may be renamed by specifying an output map. For example, "output": {"x": "xcoord", "y": "ycoord"} will use "xcoord" and "ycoord" as the output properties instead of "layout_x" and "layout_y".

Example

{"type": "force", "links": "edges", "linkDistance": 70, "charge": -100, "iterations": 1000}

This example assumes a data set named "edges" has already been defined, and has appropriate source and target attributes that reference the graph nodes.

Performs a cartographic projection. Given longitude and latitude values, sets corresponding x and y properties for a mark.

Output

The geo transform sets the following values on each datum:

Example

{
  "type": "geo",
  "lat": "latitude",
  "lon": "longitude",
  "projection": "winkel3",
  "scale": 300,
  "translate": [960, 500]
}

This example computes a Winkel3 projection for lat/lon pairs stored in the latitude and longitude attributes.

Creates paths for geographic regions, such as countries, states and counties. Given a GeoJSON Feature data value, produces a corresponding path definition, subject to a specified cartographic projection. The geopath transform is intended for use with the path mark type.

Output

The geopath transform sets the following values on each datum:

Example

{
  "type": "geopath",
  "field": "data",
  "projection": "winkel3",
  "scale": 300,
  "translate": [960, 500]
}

This example creates path definitions using a Winkel3 projection applied to GeoJSON data stored in the data attribute of input data elements.

Computes tidy, cluster, and partition layouts.

Output

The hierarchy transform sets the following values on each datum:

Example

See this link for example specifications of cartesian & radial trees, cluster & radial dendrograms, and a time-slice treemap using treeify.

Computes a path definition for connecting nodes within a node-link network or tree diagram.

Output

The linkpath transform sets the following values on each datum:

Example

{"type": "link", "shape": "line"}

Creates straight-line links.

{"type": "link", "shape": "curve", "tension": 0.15}

Creates curved links with a limited amount of curvature (tension = 0.15).

Computes a pie chart layout. Given a set of data values, sets startAngle and endAngle properties for a mark. The pie encoder is intended for use with the arc mark type.

Output

The pie transform sets the following values on each datum:

Examples

{"type": "pie", "field": "price"}

Computes angular widths for pie slices based on the field price.

{"type": "pie"}

Computes angular widths for equal-width pie slices.

Computes layout values for stacked graphs, as in stacked bar charts or stream graphs.

Output

The stack transform sets the following values on each datum:

Example

{"type": "stack", "groupby": ["x"], "sortby": ["c"], "field": "y"}

Computes a squarified treemap layout. The treemap transform is intended for visualizing hierarchical or faceted data with the rect mark type.

Output

The treemap transform sets the following values on each datum:

Example

{"type": "treemap", "field": "price"}

Computes a treemap layout where elements are sized according to the field price. This example assumes the input data is hierarchical or has already been suitably faceted.

Computes a voronoi diagram for a set of input seed points and returns the computed cell paths.

Output

The voronoi transform sets the following values on each datum:

Example

{"type": "voronoi", "x": "layout_x", "y": "layout_y"}

Adds Voronoi cell paths based on previously computed layout coordinates.

Computes a word cloud layout, similar to Wordle. The wordcloud transform is intended for visualizing words or phrases with the text mark type.

Output

The wordcloud transform sets the following values on each datum:

Example

{
  "type": "wordcloud",
  "text": "word",
  "font": "Helvetica Neue",
  "fontSize": "count",
  "fontScale": [10, 64]
}

Computes a word cloud layout for the word property, sized by the count property. The resulting font sizes will be scaled to the range [10, 64] pixels.