RL Objectives - m-marini/wheellyj GitHub Wiki

Objective defines the rewards generated by the environment based on the agent's behavior.

The function consists of a prioritized list of basic objectives. The final reward is determined by the first objective that generates a non-zero value.

Basic objectives

ActionSet

The objective rewards the value when the robot performs certain actions. It can be defined as the movement action index or the rotation action index of the sensor or a combination of the two actions.

Parameters:

  • move: the move action index
  • sensor: the sensoraction index
  • reward: the returned reward default = 1
# Rewars for move action = 0 and sensor action = 0
$schema: https://mmarini.org/wheelly/objective-action-set-schema-1.0
class: org.mmarini.wheelly.objectives.ActionSet
move: 0
sensor: 0

AvoidContact

The objective rewards the value when the robot comes into contacts with obstacles.

Parameters:

  • reward: the returned reward default = -1

E.g.

$schema: https://mmarini.org/wheelly/objective-avoid-contact-schema-0.1
class: org.mmarini.wheelly.objectives.AvoidContact

Cautious

The cautious objective rewards the behavior that brings the robot to the safest position, i.e. the centroid of the polar map

Returns a negative reward (-0.5) if the proxy sensor is not frontal directed or returns a positive reward (0 ... 1) proportional to the minimum distance of the nearest obstacle compared to the maximum distance of the polar map.

Parameters:

  • maxDistance: maximum distance of polar map (m)

E.g.

$schema: https://mmarini.org/wheelly/objective-cautious-schema-0.1
class: org.mmarini.wheelly.objectives.Cautious
maxDistance: 0.5

ConstantReward

The target always returns a fixed value. It can be used as the last target to determine the default value

Parameters:

  • reward: the returned reward

E.g.

$schema: https://mmarini.org/wheelly/objective-constant-schema-0.1
class: org.mmarini.wheelly.objectives.ConstantReward
reward: -1

Explore

The Explore objective returns the reward if the number of known sectors in the radar view is increased.

Parameters:

  • reward: the returned reward default = 1

E.g.

$schema: https://mmarini.org/wheelly/objective-explore-schema-0.2
class: org.mmarini.wheelly.objectives.Explore

Label

The label objective returns the reward if the robot is stopped within wheels speed range with the a labeled target in front within a orientation range at a distance within a distance range and the sensor directed to the front within a direction range.

Parameters:

  • minDistance: minimum distance of obstacle (m)
  • maxDistance: maximum distance of obstacle (m)
  • velocityThreshold: the maximum velocity of wheels (pps) default = 0.01
  • directionRange: The range of valid robot orientation related to the goal (DEG) default = 180
  • sensorRange: The range of valid sensor direction (DEG) default = 0
  • reward: the reward in case of matched objective default = 1

E.g.

$schema: https://mmarini.org/wheelly/objective-label-schema-0.1
class: org.mmarini.wheelly.objectives.Label
minDistance: 0.5
maxDistance: 1
velocityThreshold: 5
directionRange: 15
sensorRange: 0
reward: 1

No Move

The no move objective rewards the behaviour that keeps the robot quite stationary

Returns the reward if the robot speed is less than threshold speed with sensor pointed in front.

Parameters:

  • velocityThreshold:the maximum velocity of robot (pps)
  • reward: the returned reward default = 1
  • sensorRange: The range of valid sensor direction (DEG) default = 0

E.g.

$schema: https://mmarini.org/wheelly/objective-nomove-schema-0.1
class: org.mmarini.wheelly.objectives.NoMove
velocityThreshold: 5

Stuck

The stuck objective rewards the behaviour that keeps the robot stationary at a given distance of a frontal obstacle.

Returns a fuzzy reward dependent on the distance from the front obstacle.

Parameters:

  • distance0: minimum distance for 0 reward (m)
  • distance1: minimum distance for 1 reward (m)
  • distance2: maximum distance for 1 reward (m)
  • distance3: maximum distance for 0 reward (m)
  • sensorRange: the range of valid frontal direction of proximity sensor (DEG)

E.g.

$schema: https://mmarini.org/wheelly/objective-stuck-schema-0.1
class: org.mmarini.wheelly.objectives.Stuck
distance0: 0.2
distance1: 0.5
distance2: 1
distance3: 1.5
sensorRange: 15

Release 0.20.1