NeuralDualDice

NeuralDualDice estimates the policy value $\rho^\pi$ by first approximating the stationary distribution correction $w_{\pi/D}: S \times A \to R_{\geq 0}$ in the tabular setting. It is a gradient-based estimator that parameterizes the function arguments of the dual DualDICE objective with neural networks and performs stochastic gradient descent-ascent based on logged data. It inherits from NeuralDice, but overrides all the necessary base methods.

Unlike NeuralGenDice and NeuralGradientDice, NeuralDualDice only supports the discounted case, i.e., $0 < \gamma < 1$.

🧮 Mathematical Formulation

Fenchel-Rockefeller duality is applied to the primal DualDICE objective from TabularDualDice, to yield the dual DualDICE objective:

$$ J(v, w) \doteq E_{ s_0 \sim d_0, ~ (s, a) \sim d^D, ~ s' \sim T(s, a) } \left [ L(v, w; s_0, s, a, s') + \phi(w(s, a)) \right ], \quad \phi(x) \doteq |x|^p, \quad p > 1. $$

The loss term $L(v, w; s_0, s, a, s')$ is given by:

$$ L(v, w; s_0, s, a, s') \doteq (1 - \gamma) E_{ a_0 \sim \pi(s_0) } [ v(s_0, a_0) ] + w(s, a) \left ( \gamma E_{ a' \sim \pi(s') } [ v(s', a') ] - v(s, a) \right ). $$

The optimization problem in DualDICE alternates between maximizing $J(v, w)$ with respect to $v$ and minimizing it with respect to $w$. Hence, stationary distribution correction $w_{\pi / D} = w^\ast$ is estimated based on the optimized $v^\ast$.

For further details, refer to the original paper: DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections

🏗️ Constructor

def __init__(
        self,
        gamma, p, seed, batch_size,
        learning_rate, hidden_dimensions,
        obs_min, obs_max, n_act, obs_shape,
        dataset, preprocess_obs=None, preprocess_act=None, preprocess_rew=None,
        dir=None, get_recordings=None, other_hyperparameters=None, save_interval=100):

Args:

• All the arguments of NeuralDice.
• p (float): Regularization function exponent $p$.

💵 Loss

def get_loss(self, v_init, v, v_next, w):

Overrides the base class get_loss to compute the dual DualDICE objective.

🧪 Example

from some_module import NeuralDualDice

estimator = NeuralDualDice(
    gamma=0.99,
    p=2.0,
    seed=0,
    batch_size=64,
    learning_rate=1e-3,
    hidden_dimensions=(64, 64),
    obs_min=obs_min,
    obs_max=obs_max,
    n_act=4,
    obs_shape=(8,),
    dataset=df,
    dir="./logs"
)

estimator.evaluate_loop(n_steps=10_000)

rho_hat = estimator.solve_pv(weighted=True)