Venue: ICML 2026 (Oral) · arXiv: 2603.04639 · Code: RoboMME/robomme_benchmark Affiliations: University of Michigan · Stanford · Figure AI Traction (2026-06-09): ★111 GitHub · 6 citations (arXiv preprint) One-page version: ICML-2026-RoboMME · See also VLA Memory review

RoboMME is the first benchmark that turns "which memory mechanism should a VLA use?" into a controlled, apples-to-apples comparison. It contributes (1) a cognitively-motivated task taxonomy (temporal / spatial / object / procedural memory) over 16 manipulation tasks, and (2) the MME-VLA suite — 14 memory-augmented policies all built on the same π0.5 backbone, factored as 3 memory representations × instantiations × 3 integration mechanisms. The headline empirical result: no single memory design wins everywhere; the best overall non-oracle policy, FrameSamp + Modulator (perceptual memory, 44.51 % avg), beats all symbolic and recurrent variants and the prior SOTA, but symbolic memory still wins on counting / event-salient tasks. Humans score 90.5 %, so headroom is large.

Generalist VLAs (π0.5, OpenVLA, etc.) are largely Markovian — they condition on the current observation only. Real manipulation is history-dependent: counting repeated actions, re-finding an object that was occluded, imitating a demonstrated motion strategy. Memory-augmented VLAs have appeared (MemoryVLA, HAMLET, SAM2Act, MemER…), but each is evaluated on its own narrow, non-standardized setup, so the community cannot say which memory representation helps which kind of task. RoboMME fixes the evaluation substrate and then sweeps the design space on a single backbone.

RoboMME is the only suite that is simultaneously non-Markovian, partially observable, dynamic, video-conditioned, and covers all four memory types with subgoal + keyframe annotations.

16 tasks in 4 suites, each targeting one primary memory faculty (T = temporal, S = spatial, O = object, P = procedural):

Tasks range from 208 to 1,134 average timesteps (VideoPlaceOrder is the longest, a long-video ordinal-reference task), so the suite spans short reactive tasks to very long-horizon video-conditioned ones.

This is the core technical contribution: a factored design space on a frozen-recipe π0.5 backbone, so differences are attributable to the memory design rather than the base model.

(A) Symbolic memory — history is summarized as interpretable language subgoals. At each step an auxiliary VLM conditions on the current image + prior subgoals and emits the next subgoal. Two instantiations:

• SimpleSG — plain instructions (pick up the green cube).
• GroundSG — grounded subgoals that append image coordinates (pick up the green cube at [63, 152]), which helps spatial reasoning.
• Subgoal generators: Gemini-2.5-Pro (prompt-only), a fine-tuned Qwen3-VL-4B (QwenVL) trained on the dataset's subgoal annotations, or Oracle (simulator ground truth, used as an upper bound). Symbolic memory requires extra subgoal annotations to train the VLM.

(B) Perceptual memory — history is a sequence of visual tokens from past images, extracted by the π0.5 vision encoder (no proprioception needed). Two token-selection strategies:

• TokenDrop — token dropping (à la TimeChat): removes temporally redundant patches by RGB difference.
• FrameSamp — uniform frame sampling (à la VideoLLM): evenly downsamples frames and concatenates their tokens.

(C) Recurrent memory — compress the visual-token sequence into a fixed-size latent state via recurrence. Two models:

• TTT — Test-Time Training: updates fast weights online with a self-supervised loss, then reads out features.
• RMT — Recurrent Memory Transformer: processes the stream in segments and recurrently updates learnable memory slots with a transformer.

Symbolic memory integrates trivially (concatenate subgoals onto the task instruction). The neural memory tokens (perceptual / recurrent) need architectural plumbing, so the paper studies three ways to inject them into π0.5's VLM + action-expert structure:

Prior methods compared (4): (1) π0.5 no-memory; (2) π0.5 w/ past actions (concatenates past actions as explicit symbolic memory, UniVLA-style); (3) SAM2Act+ (SAM2 memory bank + discrete keyframe actions executed by an external motion planner); (4) MemER (VLM infers symbolic subgoals from a buffer of selected keyframes — a hybrid of perceptual + symbolic memory).

Evaluation protocol: fixed memory budget of 512 tokens (matching the current-observation image-token count) for fair comparison; 50 episodes/task (800 total) with held-out seeds; single multi-task model, ≤1,300 steps/episode; scores averaged over the last 3 checkpoints × 3 seeds (9 runs).

Grounded in the paper's Appendix A (math) and the official code in RoboMME/robomme_policy_learning — the model suite lives there (an openpi/π0.5 fork), while the benchmark env is in robomme_benchmark. Weights: Yinpei/mme_vla_suite.

π0.5 is a Mixture-of-Transformers (MoT) dual-expert model. In code, HistoryPi0 subclasses openpi's Pi0 (src/mme_vla_suite/models/integration/history_pi0.py) and runs two Gemma experts under one blockwise-causal attention:

• VLM expert (PaliGemma = SigLIP + Gemma-2B) — a SigLIP image encoder (PaliGemma.img) turns each camera view into vision tokens; these plus the tokenized language prompt form the prefix. This is the "understand the scene" expert (paligemma_variant = "gemma_2b").
• Action expert (Gemma-300M) — takes the noised action chunk + a flow-matching timestep and forms the suffix; it predicts the flow-matching velocity that denoises the action chunk. In π0.5 it is conditioned on the diffusion timestep via AdaRMSNorm (use_adarms=[False, True], adarms_cond = time_emb).

Attention is blockwise-causal: action (suffix) tokens attend to the VLM prefix; the mask is built so vision tokens never attend to memory tokens, but action tokens can (make_attn_mask, mask_na). The training loss is the standard flow-matching objective on the action expert's predicted velocity.

From a streaming horizon of 16 past frames (streaming_obs_horizon: 16, single view), FrameSamp evenly samples frames (S_t = EvenSample({0…t}; N), Appendix Eq. 14) and mean-pools each to 16 tokens (token_per_image: 16, pool_type: mean), under a 512-token budget. The mem_encoder builds each memory token from the SigLIP image embedding (2048-d) + a positional embedding (768-d) — and in the best config no proprioceptive state (use_state_emb: false, confirming the paper's "visual tokens alone suffice") — then projects to memory_token_dim: 1024, yielding the memory tokens M_t (obs.static_image_emb / static_pos_emb in embed_memory). (Config: models/config/robomme/perceptual-framesamp-modul.yaml. TokenDrop instead keeps the least-redundant patches by RGB/Jaccard similarity; recurrent variants feed recur_* streams through a TTT or RMT encoder. memory_token_dim is 2048 for context-integration but 1024 for modulation/expert.)

Crucially, in modulation mode the memory tokens do not enter the token sequence (that is the context variant, which prepends M_t to the prefix). Instead memory conditions the AE through feature-wise AdaLN, implemented in history_gemma.py and gated by mem_mods=[False, True] so that only the action expert is modulated — the VLM expert is left untouched, preserving its pretrained behaviour. The injection happens inside HistoryBlock — i.e. in every transformer layer — right before the FFN sublayer, and only on the action-expert stream (the code guard is if i == len(xs)-1 and integration_type=="modulation", where the loop index i runs over experts, so len(xs)-1 selects the action expert, not a particular layer). At each such layer (Appendix Eqs. 21–23):

1. Cross-attention, action → memory (MemoryAttention): query = current action features s̃ₜ, key/value = memory tokens Mₜ, with a softmax masked over valid memory slots → rₜ = Attn_mod(Q = s̃ₜ, K = Mₜ, V = Mₜ).
2. Produce scale & shift (MemoryRMSNorm → a single Dense(2·width) on rₜ): (γₜ, βₜ) = MLP_mod(rₜ).
3. AdaLN-modulate the action features: ŝₜ = γₜ ⊙ Norm(s̃ₜ) + βₜ (code: normed * (1 + scale) + shift, with near-zero init so the block starts as identity).

So the action expert is conditioned on both signals simultaneously: π0.5's existing time-AdaRMSNorm (diffusion timestep) and the new memory-AdaLN (history). Because the conditioning is a lightweight per-feature scale/shift rather than extra sequence tokens, it barely perturbs the base policy — which is exactly why the paper finds modulation the most robust integration and why TokenDrop's lost global context hurts less here than under context integration.

flowchart LR
  subgraph IN[Inputs]
    OBS["current obs<br/>front + wrist RGB"]
    LANG["language goal"]
    HIST["frame history"]
  end
  HIST -->|"FrameSamp: 16-frame horizon,<br/>mean-pool 16 tok/img, ≤512 budget"| MENC["mem_encoder<br/>(SigLIP 2048-d + pos 768-d,<br/>no state) → 1024-d"]
  MENC --> MT["memory tokens Mₜ"]
  OBS --> SIG["SigLIP encoder"]
  SIG --> VLM["VLM expert<br/>(PaliGemma: SigLIP + Gemma-2B)"]
  LANG --> VLM
  VLM -->|"prefix KV (blockwise-causal)"| AE
  NOISE["noised action chunk + flow time τ"] --> AE["Action expert (Gemma-300M)"]
  TIME["time_emb"] -->|"AdaRMSNorm"| AE
  MT -->|"cross-attn → γ,β (AdaLN)"| AE
  AE --> V["flow velocity vθ → denoise → action chunk aₜ"]

Key code pointers: models/integration/history_pi0.py (HistoryPi0, embed_memory, embed_prefix/suffix, compute_loss routing on integration_type ∈ {context, modulation, expert}) · models/integration/history_gemma.py (HistoryBlock, MemoryAttention, MemoryRMSNorm, applying mem_attn + AdaLN before the FFN of every layer on the action-expert stream) · src/openpi/models/pi0.py (base MoT π0.5). The expert variant instead adds a third Gemma-150M memory expert with the blockwise rule "action expert attends to VLM + memory experts; VLM and memory do not attend each other".

(Oracle rows are upper bounds — they use simulator-ground-truth subgoals. 🏆 = best non-oracle.)

Reading the table:

• Memory helps a lot: the best learned policy (44.51 %) is ~2.5× the no-memory π0.5 baseline (17.93 %).
• Perceptual > Symbolic > Recurrent on average (for learned, non-oracle settings). FrameSamp + Modul (44.51 %) is the best learned policy and edges out the strongest prior method MemER (42.38 %).
• Recurrent memory underperforms — TTT/RMT sit at 18–22 %, barely above no-memory, suggesting fixed-size latent compression loses too much for these tasks.
• The oracle gap is huge: GroundSG + Oracle hits 84.08 %, so if perfect subgoals were available, symbolic memory would dominate — the bottleneck is the subgoal-predicting VLM, not the representation.
• TokenDrop < FrameSamp because aggressive patch pruning removes global spatial context (hurts e.g. StopCube, which needs object-distance awareness).
• Memory-as-Modulator is the best integration for both perceptual variants — its lightweight AdaLN conditioning preserves the base policy.

Regrouping the 16 tasks by functional demand exposes that different memory designs are complementary:

• Perceptual (FrameSamp+Modul) dominates motion-centric, time-sensitive, long-horizon tasks — where extended visual history and low-level motion cues matter.
• Symbolic wins short-horizon and event-salient tasks — where a clear subgoal is enough (note time-sensitive ≈ 0 % for symbolic: language can't express "press exactly now").
• MemER wins dynamic scene-change tasks, because it keeps raw keyframe images during online execution.

Four history-dependent real-robot tasks (success out of 40):

The simulation ordering holds on real hardware: perceptual memory generalizes best overall (25/40), symbolic memory is strongest on the language-friendly Put Fruits task, and no-memory π0.5 essentially fails (4/40).

Reformulating each task as an online VideoQA problem (participants pick the next high-level action; an oracle planner guarantees flawless low-level control) yields 90.5 % human success — but humans still fail consistently on long-horizon (PatternLock) and time-critical (StopCube) tasks. RoboMME is therefore hard even with perfect motor control: the memory/reasoning demand alone is non-trivial.

• Q1 — best design? Perceptual memory overall; FrameSamp + Modul = 44.51 %; Memory-as-Modulator is the best integration; TokenDrop < FrameSamp.
• Q2 — is symbolic reasoning alone enough? No — even oracle subgoals degrade on manipulation-intensive (StopCube, InsertPeg) and cluttered scenes (BinFill, PickHighlight); precise visuomotor control becomes the bottleneck.
• Q3 — human ceiling? 90.5 %, still imperfect → rigorous testbed.
• Q4 — task-dependence? No single representation dominates; strengths are complementary (symbolic → counting/short-horizon/event-salient; perceptual → imitation/motion/time-sensitive/long-horizon; MemER → scene-change).
• Q5 — efficiency? Memory adds compute; the paper sweeps a 64–512-token memory budget and reports the efficiency–performance (TFLOPs vs success) trade-off.

RoboMME is an insight/benchmark paper, not a new architecture — and that is the point. By holding the π0.5 backbone, the 512-token budget, and the task taxonomy constant, it converts an ad-hoc design folklore ("add a memory bank") into a measurable science. Three messages land:

1. Match memory to task structure. There is no universal best; the practical recommendation is perceptual memory via AdaLN modulation as the strongest single default, with symbolic memory for counting/reference and a future path toward combining representations.
2. The subgoal predictor, not the symbolic representation, is the ceiling. GroundSG+Oracle (84 %) vs GroundSG+QwenVL (33 %) shows most of the loss is in VLM subgoal prediction — a concrete target for improvement.
3. Recurrent compression is not yet competitive for manipulation history at this budget.

In the 2026 VLA landscape — where long-horizon and occlusion-heavy manipulation is a frontier and memory modules are proliferating (MemoryVLA, HAMLET, VLA Memory review) — RoboMME provides the standardized yardstick the subfield was missing, which is why it landed as an ICML 2026 Oral.

• arXiv: 2603.04639
• ICML 2026: https://icml.cc/virtual/2026/poster/65933
• Code / benchmark: RoboMME/robomme_benchmark
• Figures 1–3 reproduced from the RoboMME paper (Univ. of Michigan / Stanford / Figure AI, 2026) for scholarly review; © the authors.

• One-page summary: ICML-2026-RoboMME
• VLA Memory review · MemoryVLA · HAMLET
• ICML 2026 index

← Back to ICML-2026 · Home