Recording Examples - Interactml/iml-unreal GitHub Wiki

What are Examples?

Examples serve as the teaching material a model uses to learn to associate inputs with outputs. A Training Set is built up by recording a number of examples, each associating one or more sets of input parameters with an expected output value (or values).

Training Set

To record examples into a training set asset you use the Example Recorder node with the target Training Set asset plugged into it (or selected in the pin drop-down UI).


Hooking up a Training Set asset variable

Setting a specific Training Set asset

See Asset Types for how to set up Training Set assets 👉

Parameters

The input parameters are provided in the form of a structure type (called "InteractMLParameters") and can be built using the "Collect All The Things!" Blueprint node from a variety of input data types (int, float, bool, vector, etc.).

See Collecting Parameters to learn about sourcing parameter values 👉

Expected Output

When recording an example parameter set you need to also specify the expected output of the model. This can be either a simple number or a more richly defined structured value. The types of parameter structures are referred to as Labels.

Simple Number

If you only have the simple numeric output then selecting which to record for is fairly easy and can be as simple as a constant you keep setting, or a counter you can tick up and down with a button press.

Composite Type

Composite types are a bit more fiddly to arrange as having several values to manually adjust for each recording is probably a pain to do and fraught with chance for mistakes. However, you can use Label Tables to define preset output values that can be hooked up by indexing the table, then you are only adjusting the table index (e.g. with a couple of buttons) for each recording.

See Labels for more on defining Labels and how to build a Label Table 👉

To switch between the simple and choose the composite output type, select the node and change the Label Type value in the Details panel.

User Interaction

Recording is normally something done separately to actually running a model since it's equivalent to 'configuration' of your application rather then the running of it which is the 'normal operation' phase. Of course, you can build applications that feature recording and training as part of their functionality if you want to as well.

The recording process is a bit more 'hands-on' than just running the model in its final application role and so typically needs a bunch of user input handling script to drive it. There are several functions that will need hooking up to support it and some brief examples of these are given below. It is beyond the scope of this documentation to describe a more sophisticated and flexible input system, but the provided Demo Project uses such systems and you can learn from and reuse those as you see fit.

Triggering Record

The record input can be bound to a controller button or key using standard Blueprint nodes. A simple example might look like this:

Record is used in two different ways depending on whether the model you are using is Single Sampling based or Series Sampling based, that-is, if each example has one snapshot of the input parameters stored in it or if it has a sequence of snapshots. The way these are triggered differs slightly:

  • Single Sampling - A single sample is recorded every time the Record input transitions from false to true ('rising edge' trigger).
  • Series Sampling - A new series of samples starts recording when the Record input transitions from false to true, and keeps recording additional samples every frame tick until the Record signal becomes false. At this point the whole recorded series of parameters is added as a single example to the Training Set. The example is associated with the Expected Output value that was present at the start of recording.

To configure the sampling mode of the node, select it and change the Mode value in the Details panel.

Deleting Examples

Deletion is immediate at the point any of the delete pins transitions from false to true ('rising edge' trigger). There are three different delete operations you can perform.

  • Delete Last - Delete the newest recorded example from the training set (like an Undo for recording)
  • Delete Output - Deletes all examples associated with the current Expected Output value
  • Delete All - Deletes all recorded examples leaving the training set empty (as-if a new training set)

These inputs can be triggered using the same techniques that the Record input is driven with.

💡 Although you don't need to expose these to the user, it can make the recording/training process much nicer to use if you at least support the 'Delete Last' function so they can Undo accidental or bad examples.

💡 Recorded examples can be inspected and deleted using the Training Set Asset window in the editor as well, see Training Set for information 👉

Results

Any changes made by using the node are applied to the Training Set asset and will result in a number of things happening:

Outputs

The Example Recorder node has a Changed output pin that will transition true briefly (one frame tick) each time a new example is recorded (or deleted) and can be used to trigger audio/visual feedback that a recording has taken place.

Data

The Training Set asset will be marked as modified and mean that it shows up as unsaved in the editor. You will need to save the asset to commit the changes to disk. This can either be done using the normal Unreal Editor Save function, or if you are recording in a standalone application you can call the InteractML Save function (see Utility Blueprints).

💡 Running in-editor will modify the asset, but it may not show as modified until you end the Play-in-Editor session.

See Your Data for information how how/where the recorded example data is stored 👉


👈 Collecting Parameters | 🏠 Home | Training Models 👉

⚠️ **GitHub.com Fallback** ⚠️