Analysis of performance - m-marini/wheellyj GitHub Wiki

---

No longer valid since release 0.25.0

---

Weelly application can produce the kpis of training session by setting -k <path> command line option.

java -classpath lib/wheelly-0.6.0.jar org.mmarini.wheelly.app.Wheelly -k <path>

Three main parameters control reinforcement learning.

The trainingAlpha $\alpha$ parameter controls the amount of changes in the neural network.

The critic estimates the agent's behaviors by predicting the residual advantage value of the environmental state. The actor estimates the probability distributions of the actions to be performed in the given environmental state.

A small value of trainingAlpha produces small changes on the neural network and can converge towards the optimal solution, but requires a large number of steps increasing the learning time.

On the other hand, a great value produces great changes by reducing the number of steps required to find the optimal solution but may fail to converge because it cannot modify the network in a fine way.

A wrong prediction causes bad delta correction values and wrong behaviors.

The lambda $TD(\lambda)$ parameter modulates the decay of the eligibility traces. Eligibility traces track network changes over time allowing weighted corrections to past effects. They somehow simulate the Hebb effect of neuronal synapses.

A zero value of lambda generates the effects of the TD(0) algorithm so only immediate effects are used to correct the network. A value of 1 generates the effect of the Montecarlo algorithm with averaged effects over infinite time. In practice, values ​​lower than 1 are used.

Too high values weighs more on past events, obscuring the short-term effects.

The octave program td_report generates the agent performance report. The kpi folder and the report folder should be selected.

The report is a markdown document with linked the charts of performance. It describes the trend of the characteristic parameters of the behavior of the robot and of the neural network.

For each parameter a summary table is shown with the number of measured steps, average value, standard deviation, minimum, median, maximum, the type of trend (linear or exponential) and the start / end values ​​of the trend. As well a table with the distribution of values ​​is shown.

Following are the diagram of values ​​over time and the histogram of values.

The main indicator of an agent's performance is the reward. The agent's goal is to maximize average rewards over the long time.

A decreasing trend indicates that the agent does not find the optimal solution and is behaving worse and worse. Possible causes of this behavior are the incorrect learning rate trainingAlpha parameter or the lambda parameter of the temporal difference.

Training an agent occurs with a cycle of interactions between environment and agent. At first the agent has no knowledge of decision making to learn and thus generates random actions. The rewards received which naturally depend on the actions taken by the agent can be positive, negative or null. The agent changes its behavior by strengthening the actions that produce positive rewards and weaken the actions that produce negative rewards. As the training progresses the agent should then select the actions that produce positive rewards and therefore the discount return should increase.

A first indicator of the dynamics of learning can be the distribution of the discount rewards in the training interval.

The median of the discount return tells us, for example, the threshold for which the agent has received a discount rewards lower and higher than this threshold for the same time.

The time intervals in which the discount return exceeds a certain threshold can also be interesting.

The time elapsed from the start of training and the first time the discount return exceeds a certain threshold can give an indication of the learning rate.

But exceeding the threshold cannot alone be considered an indicator of improvement, it could in fact be a completely random or temporary fact. It is therefore also necessary to analyze the time in which the average rewrds remains above the threshold and the frequency with which it exceeds such threshold.

The report plots the maximum probability of each action during the training session.

• Direction max probability
• Speed max probability
• Sensor direction max probability

If agent discover a deterministic best behavior the max probability should increase to the max value.

The max value depends on the final softmax layer temperature $T$ and the number of possible values $n$.

$P(max) = \frac{e^{\frac{1}{T}}}{e^{\frac{1}{T}}+(n-1)e^{-\frac{1}{T}}}$

$T = \frac{2}{\log(\frac{P(max)(n-1)}{1 - P(max)})}$

The ratio between the maximum and minimun probability is

$\frac{P(max)}{P(min)} = e^{\frac{2}{T}}$

Gradually lower values ​​indicate instead that the agent has not found a deterministic behavior that maximizes the reward and therefore randomly selects the actions.

Note that the speed action controls not only the movement speed but also the halt command. In particular, the first n - 1 values ​​determine the speed of movement while the last value (n-1) determines the halt of the robot.

The other parameter ploted by octave program is the delta $\delta_i$ correction factor for each action.

• Delta direction probability
• Delta speed probability
• Delta sensor direction probability

During training, the agent calculates the delta correction value by analyzing the reward received and the reward expected by the critic or the difference of residual advantage between the state before and after the action performed.

The delta is used to modify the parameters of the network (the weights of the connections), reinforcing or weakening the action performed if the reward is greater or less than expected.

The delta factor indicates how much the agent is correcting the neural networks, so a large delta value indicates that the agent is still looking for the best behavior, while a small value indicates that the agent has found a solution and is refining it.

The report plots the estimation of advantage residual value by the critic agent.

The critic's goal is to find the behavior that maximizes this value.

A decreasing trend in value indicates that none of the behaviors implemented resulted in improvements in rewards.

As well as for each action, the delta parameter of advantage residual estimation (critic) is also shown by the report.

Another useful parameter for evaluating the agent's performance and shown in the report is the average rewards received used by critic to evaluate the advantage residual .

Action gradients are shown last.

• Direction gradient
• Speed gradient
• Sensor direction gradient

The gradients allow to understand the state of optimization of the network.

Normally the gradients should have a descending trend which indicates a convergence of the neural network towards a local optimal point.

An increasing trend, on the other hand, could be caused by high values ​​of the learning rates which lead to too large deviations in the state space of the neural network.

Values ​​that are too low may show a vanishing gradient problem.

In this case the network fails to further optimize the behavior.

In these cases, if the behavior is not optimal enough, it is necessary to review the architecture of the neural network.

Release 0.14.0