UDF - Kotlin/kotlin-spark-api GitHub Wiki

Example

See the examples in code.

UDF API specification

A(BCD)EF:

  • A (xor)
    • udf to just create a (vararg) UDF
    • udaf to just create a UDAF
    • udafUnnamed to create an unnamed UDAF (for when udaf fails to get a name from the Aggregator)
    • udf.register to immediately create and register a (vararg) UDF or UDAF. Requires B or requires C to be a NamedUserDefinedFunction, an Aggregator, or a (vararg) UDF / UDAF function reference.
  • B
    • name. A String. Optional unless registering an unnamed (vararg) UDF or UDAF. Provides (new) name for the (vararg) UDF or UDAF.
  • C (xor with E)
    • namedUdf. A NamedUserDefinedFunction(Vararg), requires A to be udaf.register (will provide name).
    • udf. A UserDefinedFunction(Vararg), requires A to be udaf.register and requires B (will provide name).
    • func. A UDF- or UDAF function reference (will provide name based on func).
    • varargFunc. A vararg UDF function reference. Requires A to be NOT udf (will provide name based on varargFunc).
    • agg. An Aggregator (can be created using aggregatorOf()) to create a UDAF. Requires A to be NOT udf (will provide name based on agg).
    • zero, reduce, merge, finish, Optional bufferEncoder, Optional outputEncoder. Creates UDAF in place instead of using agg (will provide name, "Aggregator" by default).
  • D
    • nondeterministic. A Boolean. Optional, defaults to false. Updates (vararg) UDF or UDAF to non-deterministic. For instance, for randomization.
  • E (xor with C)
    • func. An in-place UDF lambda (won't provide name, unless specified in B or F).
    • varargFunc. An in-place vararg UDF lambda (= takes a single array as parameter) (won't provide name, unless specified in B or F).
  • F (Optional, or)
    • .register(). Will register the (vararg) UDF or UDAF. Requires B or requires C to be a NamedUserDefinedFunction, an Aggregator, or a (vararg) UDF / UDAF function reference.
    • .register(name: String). Will register the (vararg) UDF or UDAF with (new) name.
    • .withName(name: String). Will give the (named) udf the (new) name name.