Energy-Based Policy Orchestration

Abstract

Hierarchical control scales robot behavior by separating task-level planning from low-level control. In dynamic, high-dimensional tasks, however, a fixed planner-to-policy interface can fail when contacts, balance, or disturbances evolve faster than high-level policy updates. Execution therefore requires choosing not only which low-level policy to run, but also which policy-specific goal should be given to it. We formulate this problem as policy orchestration: runtime selection of a policy–goal pair that is likely to remain feasible while tracking the planner target. We develop EPO, an energy-based policy orchestrator for modular goal-conditioned policy families. For each policy, EPO learns a composite energy over low-level goals, where lower energy corresponds to higher feasibility and more accurate target realization. At runtime, EPO refines each policy's goal and executes the lowest-energy policy–goal pair. Because energies are policy-specific, new policies can be incorporated by learning their own energy models, without retraining the existing policies or a centralized scheduler. In simulation, EPO improves the empirical precision–robustness trade-off by refining policy-specific goals before selectively switching among policies based on energy scores. We further demonstrate sim-to-real deployment of the same frozen policies and energy models on real humanoid loco-manipulation tasks.

From a fixed target-to-goal map to a choice of policy and goal

In a standard hierarchy the planner's target z* is passed to one low-level policy through a fixed map. EPO sits between the planner and a library of frozen goal-conditioned policies and turns each target into a policy–goal pair (k, z): which policy runs, and which goal it is given. The planner and the policies are unchanged; with one policy and a fixed map, EPO reduces to the standard interface.

A shared 16-D command

Targets and goals live in one command space. Each policy reads the subspace it was trained on; the energy models receive the padded 16-D command and a mask of the active coordinates.

Goal refinement moves only the active base-velocity and torso-height coordinates; hand targets are passed through unchanged.

Four policies, two trade-offs

The library is built along two axes that trade against each other. Robustness vs. precision: training with stronger perturbations and wrist payloads buys disturbance tolerance at the cost of nominal tracking precision. Generalist vs. specialist: a policy that only stands gives up walking but tracks hand and height targets more precisely. All four policies share one reward function and differ only in command ranges, perturbation magnitudes and wrist payloads; they are trained with reinforcement learning and frozen before EPO is trained. No single policy is best everywhere, which is what EPO exploits.

Numbers: each policy executed alone on the evaluation commands (128 seeds × 3 terrains); error is the tolerance-normalised tracking error on normal commands, push survival is after one 3.0 m/s push. Modular by construction: EPO-M2 uses the two generalists; EPO-M3 adds π_stand and EPO-M4 adds π_stand^R. Each addition trains only the new policy's energy models (about 30 GPU-minutes); existing policies, energy models and the orchestration rule stay unchanged, whereas a learned MoE router is retrained on every library change (40–60 GPU-minutes).

The same frozen policies and energy models on hardware

EPO deployed on the humanoid for whole-body loco-manipulation, with the policies and energy models trained in simulation.

A single policy against the library

Each clip replays two recorded evaluation episodes that share the seed, the command sequence and the disturbance. Left: one policy with goal refinement. Right: EPO choosing among four policies. The chip in the corner and the strip below show which policy is executing and the predicted feasibility F_k of every policy.

Rendering: every clip replays the recorded base pose and joint angles of an Isaac Lab evaluation episode in Isaac Sim; nothing is re-simulated. Clips are examples chosen by a fixed rule (for the comparisons: the lowest seed where the single policy is terminated and EPO is not); each caption gives the counts over all episodes of that test, and the numbers over all 768 pushes are in Results.

More terrains and commands

The same evaluation on uneven ground, random blocks, waves and stairs: pushes compared against a single policy, and EPO under command sequences that jump between the extremes of the training range every 2 s. Yellow spheres mark the planner's hand targets. These clips are rendered on the terrain mesh of the evaluation, rebuilt from the same generator settings and seed.

One energy per policy, used twice

Every policy π_k in the library comes with two small models trained on its own rollouts. The feasibility model predicts whether the robot survives the next second if π_k is given goal z. The consistency model predicts the outcome π_k will reach, as a Gaussian, and scores how much of it lands inside the tolerance box around the command.

E_k(s,z;z^\star) = -\log F_k(s,z) - \log C_k(s,z;z^\star)

The same energy drives both decisions the controller makes.

Learning the energy models

For each policy, both models are trained offline from H = 50-step rollouts (one second at 50 Hz) of that policy alone, with goals sampled in the shared command space. Every policy uses the same sampling protocol and the same definition of success, so energies of different policies sit on one likelihood scale and can be compared directly.

Energy landscapes of π_gen for two refinement calls recorded during evaluation, over forward velocity and torso height. The path is the commanded goal, three refinement steps and the returned goal. Top: a typical call under normal commands. Bottom: the call with the largest energy decrease under extreme commands.

Near the general policy's precision, near the robust policy's survival

Walking: EPO-M4 tracks with error 0.548, close to the general policy with refinement (0.541) and below the robust one (0.557), while raising survival from 93.6% to 96.9%. Standing: error 0.431, below both the general (0.457) and the robust policy (0.466), at 97.7% survival against 87.9% for the general policy. Against the learned MoE router over the same four policies, EPO-M4 has lower error in both modes (0.548 vs 0.568 walking, 0.431 vs 0.461 standing).

Error is the tolerance-normalised tracking error on normal commands. Survival is the percentage of episodes without termination, averaged over normal commands, extreme commands and one 3.0 m/s push. +Opt is a single policy with goal refinement. MoE is a learned mixture-of-experts router over the same library.

Few switches in normal tracking, fast switches after a push

Averaged over normal and push episodes, EPO-M4 switches 1.38 times per episode against 15.46 for MoE-M4, about 11× fewer. The switches also go where they are needed: under normal commands EPO switches 0.8–1.8 times per episode while MoE switches 15–31 times, and in push episodes EPO switches 1.0–1.7 times while MoE almost never does (0.01–0.12). The accumulated-advantage budget B suppresses switching when energies are close, and the feasibility threshold τ forces a switch when the executing policy is in trouble.

Mean policy switches per episode, 768 episodes per bar (EPO with B = 40, τ = 0.9). Normal: episodes under normal walking and standing commands. Push: episodes with one 3.0 m/s push; MoE switches almost never there, so its push bars are barely visible.

What happens after a push

Walking (EPO-M4, 384 pushes): the first switch comes a median 2 ticks after the push. 318 episodes switch away and finish on another policy, mostly the robust π_gen^R; 10 switch away and end back on the pre-push policy; 39 never leave it; 17 fall. Standing (384 pushes): EPO-M4 already stands on π_stand^R before the push in most episodes, so the first switch comes later (median 11 ticks) and 146 episodes never switch; 14 fall. Returning to the pre-push policy after recovery is the exception, not the rule.

Safety comes from the threshold τ, precision from goal refinement

In brackets: paired difference to full EPO-M2 over the same 128 seeds × 3 terrains (768 push episodes); * marks a 95% whole-seed bootstrap interval that excludes zero. Removing the feasibility threshold τ raises push falls from 4.3% to 25.3%; selecting by consistency alone (no feasibility, hence no τ) raises them to 28.5%; removing refinement raises normal error by +0.0071.

BibTeX

@inproceedings{epo2026,
  title     = {Energy-Based Policy Orchestration},
  author    = {Ye, Weirui and Wang, Yen-Jen and Hsieh, Elvis and Dou, Zhiyang and Zhao, Shibo and Abbeel, Pieter and Du, Yilun and Kaelbling, Leslie Pack and Lozano-P{\'e}rez, Tom{\'a}s},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}