14. Reward Functions - m-marini/wheellyj GitHub Wiki

Objective Composition

Overview

The composeObjective function defines a composite reward function that combines multiple RewardFunction objectives into a single evaluation strategy.

The composed objective evaluates the provided reward functions sequentially and returns the first non-zero reward produced by the configured objectives.

This mechanism allows multiple objectives to be combined while preserving a priority order among them.

Purpose

The primary purpose of this function is to create a hierarchical reward evaluation system in which multiple behaviors can be defined and evaluated using a single RewardFunction interface.

The composition strategy allows the reinforcement learning environment to:

  • Combine multiple task objectives.
  • Prioritize more important objectives over secondary ones.
  • Return the first meaningful reward generated during evaluation.
  • Avoid accumulating rewards from multiple objectives.

This approach is useful when different behaviors should be considered alternatives rather than additive contributions.

Reward Evaluation Logic

The returned reward function evaluates an environment transition:

(state0, action, state1)

The evaluation process is:

  1. Initialize the reward value to 0.
  2. Evaluate the first objective in the list.
  3. Store the returned reward.
  4. If the reward is not equal to 0, stop the evaluation.
  5. Otherwise continue with the next objective.
  6. Return the resulting reward value.

The evaluation rule is:

for each objective:
    reward = objective(state0, action, state1)

    if reward != 0:
        return reward

return 0

Decision Table

Objective Result Action
Objective returns non-zero reward Return reward immediately
Objective returns 0 Evaluate next objective
All objectives return 0 Return 0

Priority and Reward Suppression

The composition strategy is based on priority evaluation rather than reward aggregation.

When an objective returns a non-zero value, the evaluation stops immediately and all subsequent objectives are ignored for the current transition.

This means that both positive and negative rewards can suppress lower-priority objectives.

A negative reward does not represent a failure of the composition mechanism; it is a valid outcome that takes precedence according to the configured objective priority.

Example:

[
    AvoidContact,
    Label,
    Explore
]

If AvoidContact returns -1, the composed objective returns -1 and Label and Explore are not evaluated.

Therefore, the order of objectives must be carefully defined according to their importance and expected behavior.

Behavioral Summary

The composeObjective function implements a priority-based reward composition strategy.

Unlike additive reward aggregation:

$$ R = R_1 + R_2 + ... + R_n $$

this implementation follows a first-non-zero strategy:

$$ R = \begin{cases} R_i & \text{where } i \text{ is the first objective producing } R_i \neq 0 \ 0 & \text{if all objectives return } 0 \end{cases} $$

The order of objectives in the input list determines their priority.

Example:

[
    AvoidContact,
    Label,
    Explore
]

Evaluation order:

  1. Check AvoidContact.
  2. If no reward is generated, check Label.
  3. If no reward is generated, check Explore.

The first non-zero reward terminates the evaluation.

General Reward Evaluation Assumptions

All reward functions are evaluated after the environment transition has been applied.

The reward value is computed using the previous state (s0), the executed action, and the resulting state (s1).

Reward functions may return positive values, negative values, or zero depending on the objective definition.

AvoidContact Interface

Overview

The AvoidContact interface defines a reward function used in a reinforcement learning environment to encourage a robot to maintain mobility and avoid becoming constrained by obstacles. It assigns a configurable reward (typically a penalty) whenever movement is blocked in either the forward or backward direction.

Purpose

The primary objective of this component is to discourage situations in which the robot loses mobility due to obstacles or environmental constraints.

The reward function is intended to be integrated into the reinforcement learning reward system, allowing the learning algorithm to associate blocked movement with an undesirable state.

Reward Evaluation Logic

The reward function evaluates the robot status after an action has been executed.

The logic is:

  1. Retrieve the robot status from the resulting state (s1)
  2. Check whether the robot can move forward.
  3. Check whether the robot can move backward.
  4. If movement is blocked in at least one direction, return the configured reward.
  5. Otherwise, return 0

The decision can be summarized as follows:

Robot State Returned Reward
Can move both forward and backward 0
Cannot move forward or backward reward
Cannot move in at least one direction reward

The reward value is typically negative, acting as a penalty, although positive values are technically supported.

Behavioral Summary

The AvoidContact objective promotes obstacle avoidance by penalizing states in which the robot loses mobility in either the forward or backward direction.

The implementation detects a loss of mobility rather than an explicit physical collision. A blocked movement condition may also be caused by factors such as controller limitations, motor constraints, or other environmental conditions.

For applications requiring accurate collision detection, this objective should be combined with dedicated contact sensors or collision detection mechanisms.

This approach keeps the reward function lightweight and easy to integrate into reinforcement learning environments, where preserving mobility is considered a desirable behavior.

SensorLabel Interface

Overview

The SensorLabel interface defines a reward function for reinforcement learning tasks in which the robot must stop in front of a detected labeled target while correctly orienting its sensor toward it.

The objective combines multiple conditions related to the robot's position, motion, sensor orientation, and the presence of a detected label marker. A reward is assigned only when all these conditions are simultaneously satisfied.

Purpose

The objective encourages the robot to:

  • Approach a detected labeled target.
  • Stop within a specified distance from the target.
  • Orient its sensor toward the target.
  • Remain stationary while observing the target.

This reward is particularly useful for training behaviors involving target recognition, inspection, docking, or object interaction.

Configuration Parameters

The reward function is configured through the following parameters.

Parameter Description
minDistance Minimum acceptable front distance from the target.
maxDistance Maximum acceptable front distance from the target.
velocityThreshold Maximum wheel speed considered as "stopped".
sensorRange Maximum angular deviation allowed between the sensor heading and the target direction.
reward Reward returned when all conditions are satisfied.

Default values are provided for optional parameters:

Constant Default Value
DEFAULT_VELOCITY_THRESHOLD 0.01
DEFAULT_SENSOR_RANGE 0°
DEFAULT_REWARD 1.0

Reward Evaluation Logic

The reward function is evaluated after each environment transition and performs the following operations:

  1. Retrieve the current RobotStatus.
  2. Read the measured front distance.
  3. Determine the robot position.
  4. Search for the nearest detected label marker.
  5. Verify that:
    • the front distance is between minDistance and maxDistance;
    • both wheel velocities are below the configured threshold;
    • the sensor heading is within the configured angular range;
    • at least one labeled marker exists.

If every condition is satisfied, the configured reward is returned; otherwise, the reward is 0.

Decision Table

Condition Required
Front distance within range ✓
Left wheel stopped ✓
Right wheel stopped ✓
Sensor correctly oriented ✓
At least one labeled marker exists ✓

The reward is assigned only if all conditions are simultaneously satisfied.

Behavioral Summary

The SensorLabel objective rewards the robot for reaching a stable observation position in front of a labeled target. The robot must remain within a configurable distance, be effectively stationary, orient its sensor toward the target, and operate in an environment containing at least one labeled marker.

By combining distance, motion, orientation, and target availability constraints, the objective promotes precise positioning and reliable target acquisition. It is well suited for reinforcement learning tasks involving inspection, object recognition, docking, and other sensor-guided interactions.

Label Interface

Overview

The Label interface defines a reward function for reinforcement learning tasks where the robot must stop in front of a specific labeled target while satisfying positional and orientation constraints.

Unlike the SensorLabel objective, which rewards the presence of any labeled marker, this objective is associated with a particular label specified in the configuration. The reward is granted only when the robot reaches the desired target position, remains stationary, faces it within a configurable angular tolerance, and keeps its sensor correctly aligned.

Purpose

The objective encourages the robot to:

  • Navigate toward a specific labeled target.
  • Stop within a configurable distance from the target.
  • Face the target within a specified angular tolerance.
  • Keep the sensor oriented in the desired direction.
  • Remain stationary while observing the target.

This objective is suitable for tasks requiring interaction with predefined landmarks, docking stations, or mission-specific objects identified by unique labels.

Configuration Parameters

The reward function is configured using the following parameters.

Parameter Description
label Identifier of the target marker.
minDistance Minimum accepted distance from the target.
maxDistance Maximum accepted distance from the target.
velocityThreshold Maximum wheel speed considered stationary.
directionRange Maximum angular deviation between the robot heading and the target direction.
sensorRange Maximum angular deviation allowed for the sensor orientation.
reward Reward returned when all conditions are satisfied.

Default values are provided for optional parameters.

Constant Default Value
DEFAULT_VELOCITY_THRESHOLD 0.01
DEFAULT_DIRECTION_RANGE 180°
DEFAULT_SENSOR_RANGE 0°
DEFAULT_REWARD 1.0

Reward Evaluation Logic

After each environment transition, the reward function performs the following checks:

  1. Retrieve the current robot status.
  2. Search for the configured labeled marker.
  3. Verify that the marker exists.
  4. Verify that both wheel speeds are below the velocity threshold.
  5. Compute the distance between the robot and the marker.
  6. Compute the direction from the robot to the marker.
  7. Verify that:
    • the distance is within the configured interval;
    • the robot is facing the target within the configured angular tolerance;
    • the sensor is aligned within the configured sensor tolerance.

If every condition is satisfied, the configured reward is returned; otherwise, the reward is 0.

Decision Table

Condition Required
Target label exists ✓
Robot stationary ✓
Distance within range ✓
Robot facing target ✓
Sensor correctly aligned ✓

The reward is assigned only if all conditions are simultaneously satisfied.

Behavioral Summary

The Label objective rewards the robot for accurately approaching a specific labeled target. Unlike more generic labeling objectives, this implementation requires the presence of a predefined marker identified by its label. The robot must stop within a configurable distance, face the target, and keep its sensor correctly aligned before the reward is granted.

By combining target identification, positioning, orientation, and motion constraints, this objective supports reinforcement learning tasks involving navigation to known landmarks, docking stations, inspection points, or mission-specific destinations.

MoveToLabel Interface

Overview

The MoveToLabel interface defines a reward function that encourages the robot to move closer to the nearest labeled target.

Unlike objectives that reward reaching or observing a target, this objective provides a progress-based reward whenever the robot reduces its distance to the closest labeled marker after executing an action.

Purpose

The objective encourages the robot to continuously reduce its distance from the nearest labeled target.

Rather than rewarding the completion of a task, it rewards progress toward the goal. This makes the objective particularly useful during the early stages of reinforcement learning, where dense rewards help the agent learn effective navigation behaviors.

Configuration Parameters

The objective requires only one configurable parameter.

Parameter Description
reward Reward returned when the robot moves closer to the nearest labeled marker.

Default values are provided for optional parameters.

Constant Default Value
DEFAULT_REWARD 1.0

Reward Evaluation Logic

The reward function compares the robot's distance from the nearest labeled marker before and after an action.

The evaluation proceeds as follows:

  1. Compute the initial distance (d0) from the robot to the nearest marker.
  2. Compute the final distance (d1) after the action.
  3. Compare the two distances.
  4. Return the configured reward if the robot has moved closer to the target.

The decision rule is:

if d1 < d0:
    return reward
else:
    return 0

Decision Table

Condition Returned Reward
Robot moved closer to nearest marker reward
Distance unchanged 0
Robot moved farther away 0
No labeled markers available 0

Behavioral Summary

The MoveToLabel objective provides a dense shaping reward that encourages the robot to make continuous progress toward the nearest labeled target. Instead of rewarding only goal completion, it reinforces every action that reduces the distance to the closest marker.

Because it depends solely on the change in distance between consecutive states, the objective is computationally lightweight and complements more specific objectives such as reaching, observing, or docking with labeled targets. It is particularly effective for accelerating the learning of navigation behaviors in reinforcement learning environments.

Explore Interface

Overview

The Explore interface defines a reward function for reinforcement learning tasks where the robot is encouraged to discover new areas of the environment.

The objective compares the robot's radar-based map representation before and after an action and provides a reward when the number of known map cells increases. This encourages exploration behaviors by rewarding actions that expand the robot's knowledge of the surrounding environment.

Purpose

The objective promotes autonomous exploration by rewarding the robot when it obtains new environmental information.

The robot receives a positive reward when:

  • The radar map contains more known cells after an action.
  • The robot has successfully explored previously unknown areas.

This objective is designed to support exploration strategies in unknown environments and can be combined with navigation or target-oriented objectives.

Configuration Parameters

The objective supports the following configuration parameter.

Parameter Description
reward Reward returned when the robot discovers new map cells.

Default values are applied when optional parameters are omitted.

Constant Default Value
DEFAULT_REWARD 1.0

Reward Evaluation Logic

The reward function evaluates the difference between two consecutive radar maps.

The evaluation process is:

  1. Retrieve the radar map from the previous state (s0).
  2. Retrieve the radar map from the current state (s1).
  3. Count the number of known cells in both maps.
  4. Compare the two values.
  5. Return the configured reward if the number of known cells has increased.

The decision rule is:

if knownCells(s1) > knownCells(s0):
    return reward
else:
    return 0

Decision Table

Condition Returned Reward
New radar cells discovered reward
No new cells discovered 0
Exploration does not increase map knowledge 0

Behavioral Summary

The Explore objective encourages the robot to discover unknown areas by rewarding increases in radar map knowledge. Instead of relying on predefined targets, it evaluates the amount of new environmental information obtained after each action.

This objective is particularly useful in autonomous exploration scenarios, where the robot must build knowledge of an unknown environment. It can be combined with other objectives to balance exploration, navigation, and task-specific behaviors.

Release

Current version: 0.26.0