MODEL-EVOLVING REINFORCEMENT LEARNING
MERL.CoRL 2026
Trust-Calibrated VLA
Policy Post-Training
with Evolving Imagination
A world model can imagine.
MERL learns how much to trust it.
“Put the block into the box.”2× playback
Ground Calibrate Improve
1 Sun Yat-sen University2 Shenzhen Loop Area Institute3 The Chinese University of Hong Kong, Shenzhen4 Shanghai AI Laboratory5 University of Oxford6 University of Science and Technology of China
79.7%
Average success on LIBERO-PROFour suites · three training seeds
10.9%
Fewer high-threshold refinement stagesCompared with the matched model-free control
80.0%
Observed physical-robot success32 / 40 trials · two tasks · fixed-data adaptation
01 / THE INSIGHTBEYOND SIMULATOR UPDATES
Better imagination.
More deliberate learning.
Deployment shifts beyond the demonstrations. A world model offers imagined experience, but even an updated simulator produces uneven predictions. Reliability should decide how imagination enters policy learning.
01
Model-free RL
Environment→Policy
Grounded feedback.
Interaction intensive.
02
Static model-based RL
Frozen simulator→Policy
Additional imagination.
The simulator can become stale.
03
Online-updated MBRL
Updated simulator→Policy
Adapting predictions.
Fixed use and uniform replay.
04 / MERL
Trust-calibrated RL
Evolve→Calibrate→Learn
Adapt the simulator.
Calibrate its learning influence.
Matched refinement controls on LIBERO-PRO (paper, Table 1). Model-based controls also use grounded data. Values are reported point estimates.
02 / THE METHODEVOLVE → CALIBRATE → REFINE
One loop.
Four trust decisions.
Model-Evolving Reinforcement Learning couples an evolving simulator with two levels of reliability control. Explore a refinement stage below.
01 / GROUNDED EXPERIENCE
Start with what the environment tells you.
The VLA policy executes action chunks and records observations, commands and sparse terminal success. These grounded trajectories support both policy learning and simulator adaptation.
Each policy call decodes categorical tokens into up to 8 steps of 7D continuous actions.
Stage-level ρ How much?Stage-level h How far?Chunk-level p Which chunks?Chunk-level w How much influence?
See the complete method figure+
03 / THE TRUST LABAN INTERACTIVE EXPLANATION
Give imagination
the influence it earns.
Stage-level trust sets the global imagination budget. Chunk-level trust controls local selection and influence. Change the inputs to see the two mechanisms at work.
GLOBAL SCHEDULINGHow much?
How much?
How far?
Higher estimated simulator error gives imagined experience less influence and a shorter rollout.
Lower errorHigher error
Illustrative input to the paper’s schedule; κ = 1. This is a mechanism explanation, not a measured training trace.
POLICY-LOSS MIXTUREρs =
ℒMERL = (1 − ρs) ℒgrounded + ρs ℒimagined
IMAGINATION HORIZON environment steps
08162432
2.5 action-chunk equivalents · C = 8
ρ is a loss coefficient; it does not specify the fraction of collected samples.
LOCAL REPLAY CONTROLWhich chunks?
Which chunks?
How much influence?
Sampling probability p decides how often a chunk is selected. Optimization weight w scales its influence after selection.
Four illustrative chunk errors. Priorities use (ε + 0.001)−1; weights use max(0.05, exp(−ε)), matching the public configuration.
CANDIDATE / ERRORSAMPLING pWEIGHT w
Lower error earns more replay probability and greater loss weight. These example errors are explanatory inputs, not experimental observations.
Estimated reliability guides learning; it does not certify a physically correct future.Implementation
04 / LOOK INSIDE IMAGINATIONSIMULATOR EVOLUTION
Same rollout condition.
Different predicted futures.
Online updates bring predictions closer to the observed trajectory. Average quality improves, while reliability can still vary from chunk to chunk.
Ground truthObserved future
Frozen simulatorStatic-MBRL
Evolving simulatorOnline updated
Selected original frames from the paper’s qualitative simulator comparison (Appendix D). Frame numbers indicate positions in the displayed sequence, not elapsed environment steps.
Held-out simulator diagnostics
PSNR ↑22.4 → 25.9
LPIPS ↓0.238 → 0.171
Proxy MAE ↓0.183 → 0.126
05 / THE EVIDENCELIBERO-PRO · FOUR SUITES
Trust makes
imagination more useful.
Higher final success and faster high-threshold refinement under shared grounded budgets. Tasks preserve their semantics while deployment conditions change.
Average success rate
Mean ± standard deviation · 3 training seeds† WMPO and WoVR are adapted to the same training and evaluation protocol; each WoVR seed uses an independent simulator. Paper, Table 2.
All reported metrics +
| Method | Avg. SR (%) ↑ | AUC ↑ | S2T-H ↓ |
|---|
Original learning curves. AUC is normalized learning-curve area. S2T-H averages stages to 80% success on Spatial/Object/Goal and 65% on Long. Each stage permits six grounded trajectories and at most 3,072 transitions. Simulator updates and imagination add model computation; these stage counts are not wall-clock speedups.
SCORING RECURSIVE IMAGINATIONUseful trust estimates.
Useful trust estimates.
No future-state query.
A frozen residual predictor scores unpaired recursive chunks. At depths 2–4, held-out validation reports Pearson r = 0.68 and Rank-AUC = 0.80 over 1,500 chunks. Accuracy weakens at deeper horizons.
Read the validation06 / ON THE PHYSICAL ROBOTAGILEX PiPER X
Imagination in training.
Actions in the real world.
Fixed-data refinement from 50 demonstrations per task, followed by closed-loop evaluation on held-out placements and colors. No additional physical training rollouts.
TASK 01
Block-to-Box
“Put the block into the box.”
85%MERL-Full · 17 / 20 trials
Blue block
TASK 02
Lid-to-Pot
“Put the lid onto the pot.”
75%MERL-Full · 15 / 20 trials
Placement 01
Placement 02
All eight clips use 2× equivalent physical-time playback: 3× source clips play at ⅔ media rate; 2× clips play unchanged. Baseline: Online-Updated MBRL, as labeled in the original presentation. These are separate qualitative trials, not a controlled execution-time comparison.
80.0%MERL-Full · 32 / 40 successes
95% CI [65.2, 89.5]
95% CI [65.2, 89.5]
Block-to-Box: 17 / 20Lid-to-Pot: 15 / 20MERL-Visual achieves 77.5% (31 / 40). Overlapping confidence intervals do not resolve a proxy-only gain.
Evaluation protocol07 / EXPLORE THE WORKREAD · REPRODUCE · CITE
Take a closer look.
Research materials and the implementation, in one place.
Read the paperarXiv:2610.16517Explore the codeTraining, evaluation & documentationView the posterEnglish research overview
Abstract +
Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability–efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.
Scope & limitations +
MERL is evaluated with tokenized VLA policies on structured tabletop tasks and fixed RGB sensing. LIBERO-PRO perturbations preserve task semantics; these results do not establish broad new-task learning. Predictive trust depends on calibration coverage and becomes less accurate at deeper horizons. Simulator updates add computation. Physical experiments cover two tasks using offline refinement and do not establish online physical RL.
BIBTEX
Build on this work.
@misc{li2026merl,
title = {Trust-Calibrated VLA Policy Post-Training
with Evolving Imagination},
author = {Wenzhan Li and Yiran Qin and Heng Zhou and
Huirui Wang and Yulan Guo and Ruimao Zhang},
year = {2026},
eprint = {2610.16517},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.16517}
}