Data Transforms - smclements/vega GitHub Wiki

Wiki ▸ Documentation ▸ Data Transforms

A data transform performs operations on a data set prior to visualization. Common examples include filtering and grouping (e.g., group data points with the same stock ticker for plotting as separate lines). Other examples include layout functions (e.g., stream graphs, treemaps, graph layout) that are run prior to mark encoding.

All transform definitions must include a type parameter, which specifies the transform to apply. Each transform then has a set of transform-specific parameters. Transformation workflows are defined as an array of transforms; each transform is then evaluated in the specified order.

These workflows can be specified as part of either a dataset definition:

  "data":[
    {
      "name": "stats",
      "source": "table",
      "transform": [
        {"type": "facet", "groupby": ["x"]}
      ]
    }
  ]

or added to a mark's from property:

  "marks": [
    {
      "type": "group",
      "from": {
        "data": "fields",
        "transform": [{"type": "cross"}]
      },
      ...
    }
  ],

Many transforms accept data attributes, or fields (denoted below as "Field"), as parameters. Data field parameters are strings that describe either individual attributes (e.g., "price") or access paths (e.g., "price.min"). Data fields can also access array indices: for example "list.0" is equivalent to list[0] in normal JavaScript.

These transforms can be used to manipulate the data: to filter and sort elements, form groups, and merge different data sets together.

aggregate, bin, cross, facet, filter, fold, formula, sort, zip

Computes aggregate summary statistics (e.g., median, min, max) over groups of data.

The short format uses a single object hash that maps from field names to one or more aggregation operations: {"foo": "mean", "bar": ["sum", "median"]}. The aggregation operation can be either a single string or an array of strings, each a valid aggregation operation name.

The long format uses an array of aggregate specification objects. The previous short format example translates to the following long format: [{"field": "foo", "ops": ["valid"]}, {"field": "bar", "ops": ["sum", "median"]}]

An aggregate specification supports the following properties:

The supported aggregation operations are:

Many of the aggregation functions above are straightforward, but a few deserve additional discussion.

The 'values' and 'count' functions operate directly on the input objects and return the same value regardless of the provided field name. Similar to SQL's count(*), these can be specified with the special name "*", as in "summarize": {"*": "count"}.

The 'argmin' and 'argmax' functions are a bit unusual: instead of returning the minimum or maximum value of a field, they return the original input object that contains the minimum or maximum value. This can be useful for retrieving another field associated with the minimum or maximum value (e.g., for each region, in which year did I have the maximum revenue?). If multiple entries share the minimum or maximum value, the first observed input object will be returned.

The aggregate transform outputs a new array of data objects, one for each group, with the computed aggregate statistics.

For the following input data:

[{"foo": 1, "bar": 1}, {"foo": 1, "bar": 2}, {"foo": null, "bar": 3}]

This short format aggregate transform

{"type": "aggregate", "summarize": {"foo": "valid", "bar": ["sum", "median"]}}

would produce the following output:

[{"valid_foo": 2, "sum_bar": 6, "median_bar": 2}]

Similarly, this long format aggregate transform:

{
  "type": "aggregate",
  "summarize": [
    {"field": "foo", "ops": ["valid"]},
    {"field": "bar", "ops": ["sum", "media"], "as": ["s", "m"]}
  ]
}

would produce the following output:

[{"valid_foo": 2, "s": 6, "m": 2}]

Bins raw data values into quantitative bins (e.g., for a histogram).

The bin transform returns the input data set, with an additional property, bin, that contains the binned value for the specified field. This property may be renamed by specifying an output parameter like so: "output": {"bin": "b"}.

{"type": "bin", "field": "amount", "min": 0, "max": 10, "maxbins": 5}

This example will bin values in the amount field into one of 5 bins between 0 and 10. Given the following input data:

[
  {"amount": 3.7},
  {"amount": 6.2},
  {"amount": 5.9},
  {"amount": 8},
]

The bin transform produces the following output:

[
  {"amount": 3.7, "bin": 2},
  {"amount": 6.2, "bin": 6},
  {"amount": 5.9, "bin": 4},
  {"amount": 8, "bin": 8},
]

Compute the cross-product of two data sets.

The cross transform outputs an array of data objects with two properties: one containing an item from the primary data set (named a by default), and an item from the secondary data set (named b by default). These names can be changed by setting the transform's output map. For example, the parameter "output": {"left":"thing1", "right":"thing2"}, causes the properties thing1 and thing2 to be used instead of a and b.

{"type": "cross", "diagonal": false}

This example crosses a data set with itself, ignoring entries with the same indices (e.g., along the diagonal). If the input data is [1, 2, 3], then the cross transform will output:

[
  {"a":1, "b":2},
  {"a":1, "b":3},
  {"a":2, "b":1},
  {"a":2, "b":3},
  {"a":3, "b":1},
  {"a":3, "b":2},
]

Organizes a data set into groups or "facets". The facet transform is useful for creating collections of data that are then passed along to group marks to create hierarchical structure in a visualization. It can also be used (like the aggregate transform) to compute descriptive statistics over subgroups of data. In this sense, it is similar to a "group by" operation in SQL.

The facet transform returns a transformed data set organized into facets. Vega uses a standardized data structure for representing hierarchical or faceted data, which consists of a hierarchy of objects with key and values properties.

{"type": "facet", "keys": ["category"]}

Facets the data according to the values of the category attribute. Given the following input data:

[
  {"category":"A", "value":0},
  {"category":"B", "value":1},
  {"category":"A", "value":2},
  {"category":"B", "value":3}
]

The facet transform produces a hierarchical collection of data arrays:

[
  {
    "category": "A",
    "key": "A",
    "values": [{"category":"A", "value":0}, {"category":"A", "value":2}]
  },
  {
    "category": "B",
    "key": "B",
    "values": [{"category":"B", "value":1}, {"category":"B", "value":3}]
  }
]

Each facet (group) is bundled into a "sentinel" object that includes:

• The original key properties and values (category in this example).
• A string concatenating all key values (key). This can be useful in conjunction with ordinal scales.
• The array of grouped data objects (values).

When a faceted data set is the input data for a group mark, Vega will automatically lookup the values array and pass it down as the data source for any marks contained within each enclosing group mark instance.

Filters elements from a data set to remove unwanted items.

The filter transform returns a new data set containing only elements that match the filter test.

{"type": "filter", "test": "datum.x > 10"}

This example retains only data elements for which the field x is greater than 10.

{"type": "filter", "test": "log(datum.y)/LN10 > 2"}

This example retains only data elements for which the base-10 logarithm of y is greater than 2.

Collapse ("fold") one or more data properties into two properties: a key property (containing the original data property name) and a value property (containing the data value). The fold transform is useful for mapping matrix or cross-tabulation data into a standardized format.

The fold transform returns a new array of data objects, with two additional properties: key (an extracted property name), and value (an extracted data value). The names of these new properties can be changed by setting the transform's output map. For example, the parameter "output": {"key": "k", "value": "v"} causes the properties k and v to be used instead of key and value.

{"type": "fold", "fields": ["gold", "silver"]}

This example folds the gold and silver properties. Given the following input data:

[
  {"country": "USA", "gold": 10, "silver": 20}, 
  {"country": "Canada", "gold": 7, "silver": 26}
]

this example will produce the following output:

[
  {"key": "gold", "value": 10, "country": "USA", "gold": 10, "silver": 20},
  {"key": "silver", "value": 20, "country": "USA", "gold": 10, "silver": 20},
  {"key": "gold", "value": 7, "country": "Canada", "gold": 7, "silver": 26},
  {"key": "silver", "value": 26, "country": "Canada", "gold": 7, "silver": 26}
]

Extends data elements with new values according to a calculation formula.

The formula transform returns the input data set, with each element extended with the computed formula value.

{"type": "formula", "field": "logx", "expr": "log(datum.x)/LN10"}

This example computes the base-10 logarithm of x and stores the result on each datum as the "logx" property.

{"type": "formula", "field": "hr", "expr": "hours(datum.date)"}

This example extracts the hour of the date field, and stores the result on each datum as the hr property.

Sorts the values of a data set (whether it be flat or faceted).

The sort transform returns the input data set with elements sorted in place.

{"type": "sort", "by": "-_id"}

This example sorts a data set in descending order by the value of the _id field.

Merges two data sets together according to a provided join key. If no join key is provided, the data sets are merged by their indices. In essence, zip extends a data set with values from another data set. When matching by index and the secondary (with) data set is shorter than the primary data set, the zip transform will cycle through values.

The zip transform extends the primary data set with values from a matching member of the secondary ("with") data set, and stores the value from the secondary data in the field specified by the as parameter.

{
  "type": "zip",
  "key": "id",
  "with": "unemployment",
  "withKey": "key",
  "as": "value",
  "default": null
}

This example matches records in the input data with records in the data set named "unemployment", where the values of id (primary data) and key (secondary data) match. Matching values in the secondary data are added to the primary data in the field named "value".

Visual encoding transforms can be used to create more advanced visualizations, including layout algorithms and geographic projections.

force, geo, geopath, link, pie, stack, treemap

Performs force-directed layout for network data. Force-directed layouts treat nodes as charged particles and edges (links among nodes) as springs, and uses a physics simulation to determine node positions. The force transform acts on two data sets: one containing nodes and one containing links. Apply the transform to the node data, and include the name of the link data as a transform parameter.

By default, the force transform sets the following values on each node datum:

These properties may be renamed by specifying an output map. For example, "output": {"fixed": "f", weight": "w"}.

{"type": "force", "links": "edges", "linkDistance": 70, "charge": -100, "iterations": 1000}

This example assumes a data set named "edges" has already been defined, and has appropriate source and target attributes that reference the graph nodes.

Performs a cartographic projection. Given longitude and latitude values, sets corresponding x and y properties for a mark.

By default, the geo transform sets the following values on each datum:

{
  "type": "geo",
  "lat": "latitude",
  "lon": "longitude",
  "projection": "winkel3",
  "scale": 300,
  "translate": [960, 500]
}

This example computes a Winkel3 projection for lat/lon pairs stored in the latitude and longitude attributes.

Creates paths for geographic regions, such as countries, states and counties. Given a GeoJSON Feature data value, produces a corresponding path definition, subject to a specified cartographic projection. The geopath transform is intended for use with the path mark type.

By default, the geopath transform sets the following values on each datum:

{
  "type": "geopath",
  "field": "data",
  "projection": "winkel3",
  "scale": 300,
  "translate": [960, 500]
}

This example creates path definitions using a Winkel3 projection applied to GeoJSON data stored in the data attribute of input data elements.

Computes a path definition for connecting nodes within a node-link network or tree diagram.

By default, the link transform sets the following values on each datum:

{"type": "link", "shape": "line"}

Creates straight-line links.

{"type": "link", "shape": "curve", "tension": 0.15}

Creates curved links with a limited amount of curvature (tension = 0.15).

Computes a pie chart layout. Given a set of data values, sets startAngle and endAngle properties for a mark. The pie encoder is intended for use with the arc mark type.

By default, the pie transform sets the following values on each datum:

{"type": "pie", "value": "data.price"}

Computes angular widths for pie slices based on the field data.price.

{"type": "pie"}

Computes angular widths for equal-width pie slices.

Computes layout values for stacked graphs, as in stacked bar charts or stream graphs.

By default, the stack transform sets the following values on each datum:

{"type": "stack", "groupby": ["x"], "sortby": ["c"], "field": "y"}

Computes a squarified treemap layout. The treemap transform is intended for visualizing hierarchical or faceted data with the rect mark type.

By default, the treemap transform sets the following values on each datum:

{"type": "treemap", "field": "price"}

Computes a treemap layout where elements are sized according to the field price. This example assumes the input data is hierarchical or has already been suitably faceted.