Neural Networks Add Model - smartarch/ivis-core GitHub Wiki

This page provides a more detailed description of the settings which can be set when adding a new prediction model. Please look at the Quick Start guide first for an overview on how a new prediction is created. The general model settings, prediction parameters and signals are described there. Here, we will follow with the more advanced settings.

Signals

As described in the Quick Start, several properties of the signals can be specified.

Automatic preprocessing of the signals is performed before feeding them into the neural network. For numerical signals, if minimum or maximum is specified, the min-max normalization is used – scaling the data into the interval [0, 1] – otherwise, the mean-std normalization is used – the data are scaled so that their mean is 0 and standard deviation is 1. For categorical signals, one-hot encoding is used.

Tunable settings and parameters

Some settings are tunable by the hyperparameter tuner. That means that their value is automatically optimized for the best model accuracy. For these settings, the radio buttons Fixed and Tuned can be used to either specify a fixed value, or let the tuner search for the best value automatically.

For the tuned numbers, the minimum and maximum are required. The default value can also be specified. The sampling setting acts as a hint for the distribution of the value, e.g. Logarithmic will assign equal sampling probabilities to each order of magnitude of the range.

For lists, the count of elements can be tuned. This can be used to specify a variable number of hidden layers of the network. If the count of the elements is tuned, specifying the minimum and maximum allowed count is required. The items in the list are generated from the item specifications. It is recommended to set as many item specifications as the maximum allowed count, but if less specifications are set, they will be repeated. For example, if only one specification is set, it will be used for all the items in the list (the number of items is set by the minimum and maximum count).

Neural network architecture and hyperparameters

The currently implemented architectures and their hyperparameter can be found in Architectures. At the moment, we recommend using the LSTM architecture.

The default settings of the hyperparameters should be reasonable for most applications. On the Architectures page, recommendations for the hyperparameters settings can be found.

Training parameters

The training parameters set the settings for the Tuning and Training.

Time interval for training data can be set to use only a portion of the data for the model training. By default, all the data are used.

The Start training immediately and Enable automatic predictions check boxes can be used to influence how the prediction model is trained and used. If Start training immediately is checked (recommended), the training job starts immediately after the form is submitted. Otherwise, it must be started by hand from the Model overview page. If Enable automatic predictions is checked (recommended), predictions are generated automatically when new data are added to the signal set. This can be enabled and disabled also from the Model overview page. Regarding the automatic predictions, the Minimal interval between generating new predictions can be set. This option is useful for data with high frequency of newly incoming records as it prevents the prediction job from running too often and consuming too many resources.

Advanced tuning parameters

These are the parameters for the hyperparameter tuner. The tuner is used to search for the best values of the tunable parameters (see [Tunable settings and parameters](#Tunable settings and parameters) above).

The Max tuning trials sets how many configurations of the hyperparameters will be tested before the best hyperparameters are selected. The higher the number, the better the chance of finding the best possible model, but also the longer the tuning process takes. For each configuration, the network is trained and the loss (MSE) is computed Executions per trial times. For higher number of executions,the estimate of the network's accuracy/error is better, but the tuning process also takes longer.

Advanced training parameters

This section specifies the parameters for the optimizer which optimizes the connection weights inside the neural network.

The Learning rate is the main parameter of the optimizer, and it determines the step size in each iteration of the optimizer algorithm. By default, it is set to be tunable and the sampling is set to logarithmic in order to search through the orders of magnitude.

The Training epochs sets the maximum number of epochs of the training. If the Early stopping is enabled, the training is interrupted when the validation loss stops decreasing. This is a measure to prevent overfitting of the network. The Early stopping patience setting determines the number of epochs to wait before interrupting the training.

Saving and cloning the model

The prediction model can be saved using the Save and leave button. After that, the Model overview page is displayed. If the Start training immediately checkbox was checked, the training of the model starts and the progress of the training is displayed on the overview page.

From the overview page, a copy of the model can be created using the New model with same settings button. Before the copy is saved, the same screen as for creating the model is shown, but with all the settings copied from the other prediction. This is useful when one wants to test several sets of hyperparameters and leave all the other settings the same.

When the training of the model finishes, the New model from tuned parameters button is enabled. This is similar to cloning the model's settings, but this time, the tunable variables are replaced by their best found values. The settings can again be changed (often slightly) and the new prediction model created for training.