EvaluatorInternals - shark8me/lenskit GitHub Wiki

Evaluation scripts are Groovy scripts, compiled as subclasses of EvalScript. Using Groovy's meta-programming support for creating domain-specific languages along with builder objects, they are high-level scripts that can rather easily run evaluations.

At the top level, EvalTask classes provide the entry point to configuring a recommender. Commands are registered by name in META-INF/lenskit-eval/methods/<method>.properties files on the classpath. For example, the train-test evaluator is registered with the following content in META-INF/lenskit-eval/methods/trainTest.properties:

task=org.grouplens.lenskit.eval.traintest.TrainTestEvalTask

This tells the configurator that, when looking for a method called trainTest, instantiate the task [TrainTestEvalCommand][], configure it using an appropriate constructor, invoke its execute() method, and return the output. If the last argument to the method is a closure, it invoked to configure the builder prior to calling execute().

Methods can also be registered for builders; in this case, the build() method is called rather than execute(). Also, unlike tasks, builders are always run immediately, even when they are used within a target.

When a command is configured with a closure, the closure is invoked with appropriate bridge object as its delegate. This delegate is responsible for handing individual directives. The default delegate is DefaultConfigDelegate, which resolves directives against the methods of the command using reflection. For a directive foo:

• If the value is a future that has not yet completed, schedule the directive to actually be called when the future is available.
• If there exists a setFoo method, it is used. Arguments are converted using Groovy's standard conversions; in addition, a single String argument can be be converted to a File and classes with public no-arg constructors are automatically instantiated. If no method directly exists, but there is a single-argument setFoo whose type has an available builder, the builder is instantiated with the non-closure arguments to the directive, configured with the closure (and a builder delegate), and the method is used.
• If there's an addFoo, it is used just as setFoo, with one additional rule: if given a single argument which implements List<X>, where X is the parameter type to addFoo, addFoo is called multiple times, once with each element of the list. This is how adding the crossfolder as a dataset works, as it produces multiple data sets.
• Otherwise, look for a named builder or task and instantiate it, looking in META-INF/lenskit-eval/methods as described above.

Therefore, the JavaDoc for the various builder classes serves as a complete reference to the configuration directives available for any particular class.

An alternate delegate can be configured for a builder by annotating the builder class with the ConfigDelegate annotation.

If a type is annotated with the BuiltBy annotation, then the configuration engine will attempt to use the builder specified in the annotation to build it when needed (e.g. to fill the argument of a setFoo or addFoo method).

Builders can also be registered in properties files. The META-INF/lenskit-eval/builders.properties file maps class names to builder class names. All files with that path are read from the classpath to locate builders.