Overview

  • many humanoids split control into an upper body stack (Cartesian trajectory, IK, joint PD for the arms) and a lower body stack (an RL balance or walking policy for the legs)
  • the split is a software choice only: physically the arms and legs share one floating base, so every arm motion pushes on the pelvis and moves the CoM
  • the lower body policy must therefore see the arm state and be trained with realistic arm motion, or it falls when the arms move fast or carry a load

Mathematical Formulation

Generic humanoid

GroupSymbolDimExample
floating base pose7 (position + quaternion)pelvis in world
legs (+ waist if owned by the policy)
arms (+ waist if owned by IK)

Floating base dynamics

SymbolMeaning
mass matrix, size
Coriolis, centrifugal and gravity terms
selection matrix, the base is not actuated
contact Jacobian and contact forces at the feet

Where the coupling lives

Take the first 6 rows (the unactuated base rows):

The base has no motor. The only things that can balance the term are the leg motion and the foot forces. So every arm acceleration is a disturbance the legs must absorb.

Wrench from one arm on the torso

Treat one arm as a body with mass , CoM , angular momentum about its CoM and shoulder point . With :

The torso feels the reaction . A fast forward reach pushes the torso backward and pitches it.

CoM excursion

Example: a 3 kg arm on a 35 kg robot whose arm CoM moves 0.4 m forward shifts the whole body CoM by about cm, which is a large fraction of a small foot.

ZMP with angular momentum

With the centroidal angular momentum and ground at :

Arm motion changes both and . Balance needs to stay inside the support polygon, with a margin.

Lower body policy observation

BlockSymbolDimSource
base angular velocity3IMU gyro
projected gravity3IMU orientation
locomotion command3planner or joystick
leg joint positionsencoders
leg joint velocitiesencoders
previous actionpolicy
arm joint positionsencoders
arm joint velocitiesencoders
arm targets (feedforward)upper body IK
gait phase2clock

A history of the last frames is often stacked, giving .

Critic only (privileged) inputs in asymmetric actor critic: base linear velocity (3), foot contact forces (), friction (1), payload mass at each hand (2), arm link mass and CoM offsets, external push force (3).

Action and low level control

The arms get from IK and use their own PD loop.


Algorithmic / Architectural Flow

flowchart LR
    subgraph Upper["Upper body, 100 to 500 Hz"]
        T["Cartesian target, world frame"] --> RA["Re-anchor with T_WB"]
        RA --> TG["S-curve or quintic + SLERP"]
        TG --> IK["IK / QP"]
        IK --> PDU["Arm PD"]
    end
    subgraph Lower["Lower body, policy 50 Hz"]
        OB["Observation builder"] --> PI["RL policy"]
        PI --> PDL["Leg PD, 1 kHz"]
    end
    EST["State estimator"] --> RA
    EST --> OB
    IK -- "q_U des (feedforward)" --> OB
    PDU -- "q_U, qdot_U" --> OB
    PDU -. "reaction wrench on pelvis" .-> PDL

Training recipe for the lower body policy:

  • drive the arms in simulation with a random reach generator (random targets, S-curve or quintic timing), not with fixed arms
  • use a curriculum on arm speed and reach distance, so the policy first learns to stand and then to absorb larger disturbances
  • randomise hand payload, arm link masses and arm PD gains
  • add reward terms for torso orientation, CoM over the support polygon and small action rate, with a lower weight on exact foot placement while the arms move
  • decide who owns the waist: the policy (better balance) or IK (bigger workspace), and never both

Sim-to-Real & Hardware Failure Modes

Things that break on hardware:

  • out of distribution arm motion: real reaches that are faster or heavier than any reach seen in training make the policy fall, match the real arm speed range during training
  • arm state missing from the observation: the policy then only notices the disturbance through the IMU after the torso has already tipped, so it always reacts late
  • unknown payload: a grasped object moves the CoM and the policy has no input for it, randomise payload in training or estimate it from arm joint torques
  • commanded vs measured arm state: in simulation the arm tracks almost perfectly, while on hardware it lags, so a policy that relies only on is wrong about where the arm really is, feed both
  • timing between two stacks: the upper and lower stacks often run in separate processes at different rates, messages over DDS arrive with jitter, so stamp every message and use the latest arm state at policy time
  • ankle torque saturation: when the CoM moves near the edge of the foot, the ankle saturates before the policy can recover, limit the reach so the static CoM stays inside a shrunk support polygon
  • coupled oscillation: the pelvis sways, the arm corrects in world frame, the arm reaction sways the pelvis again, see the re-anchoring note for bandwidth separation
  • arm motor heating: holding an extended pose for long uses large shoulder torque, thermal derating then lowers the real arm stiffness