A Survey on Policy Gradient Methods for Robotics (2005)

Authors: Jan Peters, Stefan Schaal Area: Reinforcement Learning, Robotics Link: Springer

Why robotics is a hard RL setting

Standard RL benchmarks run in simulation at millions of steps per second. Physical robots cannot. Each interaction with hardware is slow, potentially damaging, and irreversible. This makes sample efficiency a hard constraint rather than a performance metric: a method that requires a million environment interactions to converge is simply not usable on a robot arm. Beyond sample count, robotics introduces continuous high-dimensional action spaces (joint torques, velocities), strict real-time control requirements, and dynamics that are difficult to model accurately. These constraints make it inadequate to import algorithms designed for Atari games directly. The survey reviews which policy gradient variants are actually viable under these constraints and why.

Likelihood ratio vs. perturbation vs. natural gradient methods

Likelihood ratio methods, including REINFORCE, estimate the gradient by differentiating the log probability of the trajectory, avoiding any model of dynamics. They are unbiased but high variance. Perturbation methods estimate the gradient by directly measuring how small parameter changes affect return, which works well in low-dimensional parameter spaces but scales poorly. Natural gradient methods are the paper’s primary contribution to practice. Standard gradient ascent treats all parameter directions as equivalent, but equal steps in parameter space can produce very different changes in the policy distribution. The Fisher information matrix F captures this local geometry of the distribution manifold. The natural gradient direction is F^{-1} · ∇J, which moves in the steepest ascent direction in distribution space rather than parameter space, producing updates that are invariant to the parameterization of the policy. This leads to more consistent improvements per sample, which is the central requirement for hardware RL.

Dynamic Movement Primitives as robot policy representations

Rather than parameterizing policies as neural networks, the survey advocates Dynamic Movement Primitives (DMPs): movements encoded as learned differential equations that produce smooth, stable trajectories suitable for motor control. DMPs can be initialized from human demonstrations, greatly reducing the number of RL iterations needed to reach a good solution. This combination of imitation-based initialization with natural gradient refinement defines the practical methodology the paper recommends. The compatible function approximation framework is also covered, which ensures the critic’s gradient estimate is aligned with the actor’s parameterization so that the critic does not introduce systematic bias into the policy update.

Sample efficiency and the case for natural gradients

The Fisher matrix is n × n where n is the number of policy parameters, making direct inversion expensive for large networks. The survey identifies this as a practical bottleneck and discusses approximate methods. The insight that natural gradients outperform vanilla gradients in terms of samples-per-improvement directly motivated the design of TRPO, which found a way to approximately apply the natural gradient direction using conjugate gradient methods without ever constructing F explicitly.

Results and impact

The survey established natural policy gradients as the appropriate class of methods for physical robot learning and gave practitioners a structured comparison of the trade-offs between likelihood ratio, perturbation, and natural gradient approaches. It remains a standard reference for applying RL to physical systems and is directly credited as an influence on TRPO’s design. The identification of sample efficiency as the primary axis for evaluating robotics RL algorithms shaped how subsequent hardware-learning papers framed their contributions.