PonderPounce-WM: When Does Predicting the Future Help Context-Dependent Robot Control?

Jaeyoon Jung¹, Suhwan Choi^1,2, Yunsung Lee^†1, Youngjae Yu^†2

¹ MAUM.AI ² Seoul National University
^† Co-corresponding authors

Under Review

Future prediction as auxiliary supervision for an episode context engine. Learning to anticipate visual changes improves control, even when the policy receives only cognition.

PonderPounce

67.42%RoboMME success
60.83% baseline

67.44%Real-robot success
60.98% baseline

1 + 3One cognition token
Three future tokens during training

Overview

Predicting the future while remembering the past

Robot control often depends on evidence that is no longer visible. PonderPounce retains demonstrations and execution history in a pretrained multimodal language model, Ponder, and passes continuous cognition to a fast action model, Pounce.

PonderPounce-WM adds three learned future tokens, supervised to predict changes in frozen visual features. We study two independent choices: whether cognition comes before or after future tokens, and whether the controller receives cognition alone or all four tokens. This is auxiliary feature prediction from episode context, rather than action-conditioned rollouts for planning.

Architecture

Two design axes, one shared context engine

Token order inside Ponder

Cognition-first: C, F₁, F₂, F₃.
Future-first: F₁, F₂, F₃, C.

With causal attention, only future-first lets cognition read the same query’s future tokens. Both layouts retain tokens from previous queries.

Delivery to Pounce

The policy receives cognition only or cognition plus all future tokens, alongside its current sensory inputs and cognition age.

Future-prediction gradients shape the shared context engine during training. The prediction head is not needed to generate actions.

RoboMME results

Future supervision helps. Direct delivery need not.

On 16 memory-dependent tasks, cognition-first ordering with cognition-only delivery reaches 67.42%, up 6.59 percentage points over PonderPounce. Directly delivering future slots under the same ordering lowers success to 64.00%.

PonderPounce

60.83%

4 slots, no future loss

60.00%

WM · cog-first · cog + future

64.00%

WM · cog-first · cognition only

67.42%

Average success (%) · 0–100 scale

RoboMME success rates by task family
Method	Order	Policy reads	Counting	Permanence	Reference	Imitation	Mean
PonderPounce (1 slot)	—	Cognition	74.67	62.83	72.17	33.67	60.83
PonderPounce (4 slots, no future loss)	—	Cognition	74.33	61.33	66.50	37.83	60.00
PonderPounce-WM	Future-first	Cog + future	77.67	60.33	69.83	44.83	63.17
PonderPounce-WM	Future-first	Cognition	79.00	61.00	75.50	41.17	64.17
PonderPounce-WM	Cog-first	Cog + future	77.50	59.33	73.00	46.17	64.00
PonderPounce-WM	Cog-first	Cognition	84.83	68.83	72.00	44.00	67.42

Success rates (%), averaged over three evaluations of the same checkpoint. Each repeat contains 50 episodes per task; each family averages four tasks. The 1-slot baseline comes from the original PonderPounce paper. These are evaluation repeats, not independent training seeds.

The strongest configuration benefits from learning to predict while keeping the policy input limited to cognition. Gains are not uniform: MoveCube and PickHighlight decline by 10 and 9 percentage points, respectively.

Representation analysis

What changes inside cognition?

More future information is linearly decodable

On 192 held-out episodes, the 1-second future-feature probe improves from 0.253 explained variance for the baseline to 0.336 for the strongest controller. Future slots themselves carry still more decodable future information, but this does not translate into better control when they are directly delivered.

Tested past-event attributes remain largely decodable

Hidden-container location, repetition ordinal, and counts of previously binned cubes remain readable from full-context cognition. Some location read-outs decline after future supervision. Near-ceiling scores on several probes limit conclusions about broader memory preservation.

Future-token removal depends on training order

Without retraining, removing all future tokens from Ponder at inference changes cog-first success from 67.42% to 66.17%, retaining most of the gain. The same intervention drops future-first from 64.17% to 8.75%. This finding applies to the tested checkpoints; removing tokens also changes context and positions.

Real-robot results

67.44% mean success across four tasks

We evaluate on the same 59 held-out scenes used by PonderPounce, using the right arm of a Trossen ALOHA Stationary AI Kit with external and wrist cameras. All methods use the same 710 teleoperated training episodes. PonderPounce-WM improves PutFruits, TrackCube, and DrawPattern, while RepickBlock drops by one successful trial.

Real-robot successes and trials, with unweighted mean success rate
Method	PutFruits	TrackCube	RepickBlock	DrawPattern	Mean
π₀.₅	9/14	4/16	0/14	1/15	23.99%
FrameSamp+Modul	7/14	5/16	3/14	9/15	40.67%
PonderPounce	11/14	10/16	6/14	9/15	60.98%
PonderPounce-WM	12/14	12/16	5/14	11/15	67.44%

Entries are successes/trials. Mean weights the four task rates equally, rather than pooling all trials. Prior-method results are reproduced from PonderPounce. With 14–16 trials per task, one trial changes a task’s rate by roughly 6–7 percentage points.

Scope

What these experiments establish

Future-feature prediction can improve context-dependent control without directly conditioning the policy on future slots. The controlled study uses one base architecture and RoboMME, with prediction targets from one frozen visual encoder and one camera view. The probes do not establish why direct future-token delivery reduces performance.