🤖 Robotics arXiv Digest

Monday, October 5, 2026

📄 30 papers 📂 6 research areas Generated by Claude

🔭 Research Landscape

The loudest signal in today's 30 papers is a wave of VLA skepticism and stress-testing. Three independent diagnostic papers converge on the same uncomfortable finding: high benchmark success hides shallow grounding. 'Encoded but Not in Control' (#18) shows VLAs and world-action models grab the scene-default object even when told to pick a different one — the instruction is encoded in their activations but barely reaches the action head. 'Do VLAs Understand…' (#27) finds physical properties like mass are less decodable than price, and that robot pretraining weakens them relative to the base VLM. OGAM (#29) reports that none of four popular VLAs completes any of 24 feasible out-of-benchmark instructions. TAPDreamer (#8) adds a security dimension: a single 6.5%-area patch, built from a public encoder alone, takes FastWAM from 97.7% to 0% — the shared visual encoder is a single point of failure. ACG-Bench (#24) completes the picture for bimanual skill composition, where the best architecture reaches only 21.5%.

The constructive response comes in two flavors. One is cheap, targeted post-hoc fixes that leave the backbone intact: augment only the critic during RL post-training (#2, +7.8/+10.0 pts), adapt only the action side with forward-only zeroth-order updates (VLA-ZO, #22), retune impedance gains offline from a port-Hamiltonian energy budget (PHRetune, #26), monitor object-grounded attention with conformal thresholds (#29), or spend operator time where the policy fails (Mulligan, #28). The other is a JEPA world-model renaissance: H-JEPA (#15, with LeCun) stacks latent world models across timescales (AntMaze 18%→73%), while EpicWorldModel (#1, the day's top-ranked paper) makes JEPAs stochastic so variance can drive exploration under occlusion. Their pairing is notable — hierarchy and stochasticity are precisely the two capabilities critics have said deterministic, single-scale JEPAs lack.

A third thread is latency as a first-class design objective for generative policies: RealtimeWAM (#23) gets ~25× speedups via one-step distillation and cross-expert pipelining, Adaptive Mean Flow (#4) makes flow policies smooth enough for every-step closed-loop control, and Bilinear Flow Policy (#14) tackles the orthogonal problem of extrapolating to unseen goals with a provable error bound. Outside manipulation, safety evaluation is becoming generative too — ControlPed (#5) renders 3DGS pedestrian conflicts that drop seven E2E driving models from 88.8 to 47.4 HDScore — while GAMBIT (#6) shows learned coordination with a verified safety fallback scaling to 1,000+ robots. Read together, the batch suggests the field is moving from 'does the foundation model succeed?' to 'why, under what shifts, and how fast' — with diagnostics, monitors and verified fallbacks becoming as important as the policies themselves.

VLA Robustness, Grounding & Safety

Stress-testing and hardening vision-language-action policies: augmentation in RL post-training, adversarial patches, instruction-grounding and physical-property probes, runtime monitoring, and cheap test-time adaptation.

#2 How (and How Not) to Use Data Augmentation in VLA Po...
#8 TAPDreamer: Transferable Adversarial Patches for Wor...
#18 Encoded but Not in Control: Revealing the Grounding...
#22 VLA-ZO: Fast Zeroth-Order Adaptation for Vision-Lang...
#24 Arm-wise Compositional Generalization in Dual-Arm Vi...
#27 Do VLAs Understand and Adapt to the Objects They Han...
#29 OGAM: Connecting Systematic Testing to Runtime Assur...

World Models & Generative Policies

Latent JEPA world models that plan under uncertainty and across timescales, plus faster and more extrapolative flow-based policies.

#1 EpicWorldModel: Exploration-driven Planning with Lat...
#4 Adaptive Mean Flow for Responsive Closed-Loop Robot...
#14 Bilinear Flow Policy: Distributional Extrapolation f...
#15 H-JEPA: End-to-End Learning of Hierarchical World Mo...
#23 RealtimeWAM: One-Step Asynchronous World Action Mode...

Data Engines, Deployment & Evaluation

Where the training signal and the deployment settings come from: self-evolving humanoid motion data, failure-targeted data collection, offline controller-gain retuning, quality-diversity grasps, and a physical-design arena for frontier models.

#12 InterMimicGen: Scaling Humanoid Loco-Manipulation th...
#16 ArtifactArena: Evaluating Models by What They Build...
#21 Towards Robust Prehensile Manipulation in Open-Ended...
#26 Demonstration-Calibrated Port-Hamiltonian Retuning f...
#28 Mulligan: Performance-Guided Data Collection for Eff...

Medical & Magnetic Servoing

Precision servoing from unconventional feedback: learned latent magnetic features, B-mode ultrasound images, and force-image coupling with analytic rate limits.

#9 MagServo: Uncertainty-Resilient Hierarchical Magneti...
#10 Recon2Servo: Robotic Ultrasound Visual Servoing via...
#30 Dual-Rate Force-Image Control with Model-Based Orien...

Navigation, Driving & Locomotion

Getting from A to B safely: pedestrian-risk scenario synthesis for E2E driving, 4D-radar planning, LiDAR-to-control social navigation, biped obstacle avoidance, and drift-aware V-SLAM.

#5 Controllable and Photorealistic Pedestrian Risky Mot...
#7 End-to-End Safe Social Navigation via Multi-Task Rei...
#13 Hierarchical Reinforcement Learning for Collision-Fr...
#19 Radar2Plan: Benchmarking 4D Radar for End-to-End Ope...
#25 Human-in-the-Loop Neuro-Symbolic Drift Anticipation...

Multi-Robot & Human-Robot Collaboration

Teams of robots and robot-human pairs: learned thousand-robot trajectory coordination, ground-provided drone localization, collective-awareness taxonomy, cloth co-transport, and an agentic social humanoid.

#3 A Taxonomy on Collective Awareness
#6 GAMBIT: Learning to Plan Continuous Multi-Robot Traj...
#11 From Social Reasoning to Embodied Interaction: An Ag...
#17 Towards Quadruped-Provided Localization and Active T...
#20 Physics Residual Dynamics and Reduced Order Whole-Bo...

VLA Robustness, Grounding & Safety

#2 h=41

How (and How Not) to Use Data Augmentation in VLA Post-Training

2026-10-05 cs.ROcs.LG 👤 J. Vanschoren (h=41)

Bram Grooten, Joaquin Vanschoren

Core Contributions

A clean, actionable negative result: augmenting the actor's images during RL post-training of a VLA collapses training entirely, while augmenting only the critic helps.
Explains a common practitioner failure — 'just add DrQ-style augmentation' — and gives a rule: keep actor inputs clean in both rollouts and updates, push invariance through the value function.
+7.8 (π0.5) and +10.0 (GR00T N1.5) points out-of-distribution success on LIBERO-Plus, consistent across two very different VLA backbones.
Sweeps augmentation types and strengths to produce practical recipes, making it a reference for anyone doing RL fine-tuning of large VLAs.

Show abstract

Vision-language-action (VLA) models currently demonstrate strong performance in a wide range of real-world robotics tasks. However, they often still lack the generalization ability to handle large visual out-of-distribution shifts. Post-training of VLAs with reinforcement learning (RL) has been shown to benefit robustness, but significant room for improvement remains. In this work, we systematically study the effect of image augmentation on VLA post-training. We find that it is crucial to augment only the critic module during RL updates, while leaving the actor's input clean during both rollouts and updates. For $π_{0.5}$ and GR00T N1.5 this raises out-of-distribution success on LIBERO-Plus by $7.8$ and $10.0$ points respectively, while augmenting the actor collapses training entirely. We investigate a range of augmentation types and strengths, and provide practical recommendations for improving generalization in VLA post-training.

#8 h=21

TAPDreamer: Transferable Adversarial Patches for World Action Models

2026-10-05 cs.CVcs.AIcs.RO 👤 Basel Alomair (h=21)

Xuanyu Lu, Fengqing Jiang, Kaiyuan Zheng, Yichen Feng, Yaorui Ding

Core Contributions

A black-box, query-free attack: uses only a public visual encoder to craft one fixed patch that transfers across tasks and world-action-model architectures.
Key mechanism: patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint — a ViT-level vulnerability, not a policy-level one.
One frozen patch covering ~6.5% of the image drops FastWAM from 97.7% → 0.0% (40 LIBERO tasks) and 90.8% → 0.0% (50 RoboTwin tasks); random patches of equal size leave ~80% success.
Implication: defending the action head is insufficient — shared encoders in WAMs/VLAs are a single point of failure that needs its own robustness work.

Show abstract

World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet world action models depend on camera inputs whose manipulation can corrupt the visual representations used across tasks and action policies. Existing attacks on these models optimize against the victim's actions or predicted futures and therefore require access to target-model outputs. In this paper, we propose an attack, TAPDreamer, against world action models that instead uses a public encoder alone to construct a fixed local perturbation that transfers across tasks and action architectures. TAPDreamer requires no target-policy queries. Our key insight is that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint, and this shift remains stable across task observations. Guided by this insight, TAPDreamer uses six frames from one source task to maximize the global L1 distance between clean and patched encoder representations. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.8% to 0.0% across 50 RoboTwin tasks; matched random patches retain 81.5% and 79.2% success. The same patches reduce success to 2.1% and 0.8% on two DreamWAM configurations and to 10.0% on Motus. These results show that protecting downstream action generation alone is insufficient: defenses for world action models must also secure shared visual encoders against persistent local perturbations.

#18 h=15

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

2026-10-05 cs.ROcs.LG 👤 Kun Wu (h=15)

Shaohan Jiang, Jiahang Cao, Qiduo He, Fengting Deng, Kun Wu

Core Contributions

Shows that high task success masks a grounding gap: when the instruction names a different visible object, all tested VLAs and WAMs still grab the scene-default target.
The language is not ignored — linear probes recover the instructed target and intermediate action predictions shift — but attention from instruction tokens to action generation is weak.
UMAP and shared NMF reveal representations increasingly organized by scene identity, so the scene, not the instruction, effectively selects the task.
Proposes a concrete progress criterion: policies must follow valid intent changes that conflict with scene-favored behavior. Pairs naturally with #27 and #29.

Show abstract

Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.

#22 h=12

VLA-ZO: Fast Zeroth-Order Adaptation for Vision-Language-Action Models

2026-10-05 cs.ROcs.AI 👤 Taesik Gong (h=12)

Jaemin Kim, Jiahn Kim, Taesik Gong

Core Contributions

Makes test-time VLA adaptation feasible on inference-only hardware via zeroth-order (forward-only) optimization, which needs no backprop memory.
Key systems insight: confine adaptation to the action side, freeze the vision-language prefix, and reuse its conditioning states across all perturbation queries and steps.
Schedule-aware prefetching hides state-transfer costs, giving 25.6× (q=16) and 32.5× (q=64) end-to-end speedups over naive ZO.
On LIBERO camera-viewpoint shifts, success climbs from 48.3% to 58.2% / 63.6% — showing that faster ZO converts directly into larger, more useful query budgets.

Show abstract

Adapting vision-language-action (VLA) models to deployment-time distribution shifts is important for reliable robotic operation, but conventional first-order adaptation can exceed the memory budget of inference-oriented deployment platforms. Zeroth-order (ZO) optimization offers a forward-only alternative with inference-level memory, but accurate gradient estimation requires many perturbation queries, making naive ZO prohibitively slow for large VLA models. We present VLA-ZO, a framework for fast ZO adaptation that exploits the structure of VLA computation. By confining adaptation to the action side, VLA-ZO keeps the expensive vision-language prefix frozen and reuses its conditioning states across perturbation queries and optimizer steps, while schedule-aware prefetching hides state-transfer overhead. On LIBERO camera-viewpoint shifts, VLA-ZO reduces end-to-end adaptation time by 25.59$\times$ at $q=16$ and 32.54$\times$ at $q=64$ relative to baseline ZO, while improving average task success from 48.27% without adaptation to 58.17% and 63.58%, respectively. These results show that making ZO faster can make larger query budgets practical, providing a promising path toward resource-efficient VLA adaptation on deployment platforms.

#24 h=11

Arm-wise Compositional Generalization in Dual-Arm Vision-Language-Action Models

2026-10-05 cs.RO 👤 Zhen-Fei Yin (h=11)

Zaibin Zhang, Binghao Ran, Yuhan Wu, Zhongbo Zhang, Yifan Wang

Core Contributions

Frames dual-arm generalization as recomposing known per-arm atomic skills (reordering, synchronization, cross-task) and builds ACG-Bench: 23 task-condition pairs, 17 unseen compositions.
Controlled study on a shared π0.5 backbone comparing arm-token grouping, skill-specific LoRA (SkillLoRA), and arm-wise attention.
The combination, AE-VLA, reaches 21.5% unseen-composition success vs 2.9% (single π0.5) and 5.5% (two independent π0.5s) in sim; 39% vs 10% on real SO101 arms.
Low absolute numbers are the headline: today's bimanual VLAs barely compose skills, and architecture — not just data — matters.

Show abstract

Generalization in multi-arm collaboration can be studied as composing familiar atomic skills in new ways across arms. However, existing evaluations offer limited insight into which training and architectural choices support this ability under different coordination requirements. We introduce \textbf{ACG-Bench}, a benchmark for \emph{Arm-wise Compositional Generalization} that provides a common testbed for studying skill recomposition in dual-arm policies. It contains 23 task--condition pairs across 8 task families, with 6 in-domain conditions and 17 unseen compositions covering reordering, synchronization, their combination, and cross-task composition. All methods receive the same per-arm atomic prompts, and success requires achieving the task goal while satisfying physical milestones and specified order or timing constraints. Using $π_{0.5}$ as a common vision-language-action backbone, we compare representative data-augmentation and architectural strategies with shared source data and a common evaluation protocol. Our architectural study examines arm-token grouping, skill-specific LoRA adapters (SkillLoRA), and arm-wise attention (AWA), highlighting the complementarity of skill-conditioned parameters and attention structure. Combining these choices yields \textbf{AE-VLA}, which achieves 21.53\% generalization success in simulation, compared with 2.94\% for Single $π_{0.5}$, 3.06\% for MA-VLA, and 5.53\% for two independently controlled $π_{0.5}$ policies. On physical SO101 robots, AE-VLA reaches 39.00\% mean success across five unseen conditions, compared with 10.00\% for the strongest baseline. These findings provide empirical guidance for designing dual-arm policies that generalize beyond fixed training routines.

#27 h=9

Do VLAs Understand and Adapt to the Objects They Handle, or Simply Replay Learned Behaviors?

2026-10-05 cs.ROcs.AI 👤 Xin-Nuo Xu (h=9)

Xinnuo Xu

Core Contributions

Asks whether VLA generalization reflects physical understanding or incidental replay — probing seven VLAs with linear probes and RSA.
Physical properties (mass, fragility, friction, deformability, size) are less decodable than non-physical ones (category, material, even price), and robot pretraining weakens them versus the base VLM.
In a controlled LIBERO test, heavier objects get the same lifting motion and success drops; the few models that adapt respond to lexical/visual cues, not mass.
Together with #18, a sobering signal that VLA 'generalization' often means matching scenes to memorized motions.

Show abstract

This paper asks whether VLA generalization is grounded in a global understanding of objects' physical properties that enables policies to adapt their motion to unseen setups, or if policies simply replay the motions they've learnt that happen to succeed in new setups. The former reflects genuine generalization; the latter reflects incidental robustness. We first examine awareness of physical properties in seven VLAs by applying linear probing and representational similarity analysis (RSA) to their activations. We find that physical properties, including mass, fragility, deformability, friction and size are less decodable than non-physical properties such as semantic category, material, sound and price in nearly every modality stream. Compared with their base VLMs, robot pre-training weakens the linear encoding of physical properties in the language stream. Neither pre-training nor downstream fine-tuning strengthens the alignment between physical-property differences and activation distances. We then ask whether the weak physical information present in these activations shapes the actions a VLA generates. In a controlled LIBERO case study, we increase the mass of an in-domain object and signal the change through language or vision. Most VLAs use similar lifting behaviour for the heavier and original-mass objects, leading to task success declines. The few exceptions change their behaviour in response to lexical or visual cues rather than to mass itself. These results suggest that VLAs encode physical properties weakly and do not reliably use them to adapt their motion.

#29 h=9

OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies

2026-10-05 cs.LGcs.AIcs.RO 👤 F. D. de Oliveira (h=9)

Haki Darwish, Xiangyu Yin, Changwen Li, Rongjie Yan, Francisco Gomes de Oliveira Neto

Core Contributions

Links finite systematic testing to runtime assurance: attention divergence observed in testing becomes an online stop signal for failures beyond the test suite.
Pairwise action-template × object instructions expose that none of OpenVLA, OpenVLA-OFT, UniVLA, or π0.5 completes any of 24 feasible out-of-benchmark instructions, and hazardous requests trigger behavior substitution.
Monitors gradient-weighted attention projected through object masks, aligned to a success reference via DTW, with conformal thresholds (α = 0.05) — no failure labels needed.
Stops 87–100% of failing episodes at median 5–12 s of a 20 s budget with 3–5% false stops.

Show abstract

Benchmarks expose vision-language-action (VLA) policies to few canonical instructions, while exhaustive deployment testing is impossible. We introduce Object-Grounded Attention Monitoring (OGAM), connecting systematic testing to runtime assurance: testing reveals attention divergence between successful and failed executions, and OGAM uses this signal to stop failures beyond the finite suite. We generate scene-grounded instructions through pairwise combinations of action templates and objects, and separately test meaning-preserving paraphrases. All 87 out-of-benchmark cases reveal problematic behavior across OpenVLA, OpenVLA-OFT, UniVLA, and $π_{0.5}$: none completes any of the 24 feasible instructions, while infeasible or hazardous requests also trigger behavior substitution. At each action query, we project gradient-weighted visual attention through object masks and group it by instruction role for comparison across tasks and policies. Dynamic time warping aligns this course with a successful reference despite speed differences; conformal calibration on successful episodes sets the early-stopping threshold for sustained deviations, with a nominal false-stop target of $α=0.05$. Across four policies, OGAM stops 87-100% of failed episodes at median times of 5-12s within a 20s budget, with observed false-stop rates of 3-5%, without failure-labeled training. Finite testing thus identifies attention patterns that support online intervention before failure fully unfolds.

World Models & Generative Policies

#1 h=52

EpicWorldModel: Exploration-driven Planning with Latent World Models

2026-10-05 cs.LGcs.AIcs.RO 👤 Felix Heide (h=52)

Bowen Feng, Julian Ost, May Mei, Anirudha Majumdar, Felix Heide

Core Contributions

Identifies a structural blind spot in JEPA world models: being deterministic, they average over futures and so cannot represent 'the goal might be behind that occluder' — exactly the case where planning should explore.
Replaces the deterministic predictor with a flow-matching predictor trained jointly with the latent space, so the model samples multiple plausible futures only when goal-relevant content is missing from history.
Turns flow predictive variance (linked to an upper bound on predictive entropy) into an exploration bonus inside CEM planning — uncertainty becomes an actionable signal rather than a nuisance.
Up to 22% higher success than LeWorldModel on partially observable planning tasks, suggesting stochastic latents are a prerequisite for JEPA planning outside fully observed lab scenes.

Show abstract

Latent world models based on Joint-Embedding Predictive Architecture (JEPA) are deterministic by design. While successful in fully observable scenarios, this paradigm breaks down when past observations and actions lead to multiple plausible future possibilities, e.g., due to occlusion. We introduce EpicWorldModel, a framework to train stochastic JEPAs for environments and tasks with inherent uncertainty under partially observability. We jointly train the EpicWorldModel predictor with its latent representation space to directly predict multiple potential future states using a flow-matching objective, when the goal-relevant scene content is absent from the conditioning history. We show that flow predictive variance, motivated by its relation to an upper bound on predictive entropy, serves as a useful exploration guidance for planning. By incorporating this uncertainty signal into Cross-Entropy Method (CEM)-based planning, our approach balances goal-reaching with exploration of uncertain regions where occluded goals are most likely to be located. We demonstrate the effectiveness of EpicWorldModel through a series of latent planning experiments with the best or on-par performance across tasks, showing up to 22% empirical improvement in success rate over LeWorldModel.

#4 h=39

Adaptive Mean Flow for Responsive Closed-Loop Robot Control

2026-10-05 cs.ROcs.AI 👤 O. Egeland (h=39)

Aksel Vaaler, Marco Job, Christian Holden, Olav Egeland

Core Contributions

Attacks the latency-vs-responsiveness trade-off that pushes diffusion/flow policies toward open-loop action chunking, which ignores the freshest state.
Uses Mean Flow (few-step flow matching) for low latency, and conditions each new prediction on a noise-corrupted copy of the previous trajectory whose SNR rises with lookahead time.
That schedule keeps near-term actions consistent with what was just executed while leaving far-future actions free to re-plan — a principled alternative to temporal ensembling or bidirectional decoding.
Enables fully closed-loop, every-step replanning without jitter across simulated and real tasks; code is released.

Show abstract

Diffusion- and flow-based robot policies have recently become widespread in robotic Imitation Learning (IL) due to their high performance and ability to model continuous and multimodal distributions. However, the iterative denoising procedure used by these models introduces significant prediction latency, hindering high-frequency closed-loop robot control and leading to jittery, unstable motion when frequent updates to the robot's action predictions are used. Therefore, it is common practice to train models to predict chunks of actions that can be executed sequentially without feedback, even when this reduces responsiveness and may mean the most recent state information is not used. In this article, we present Adaptive Mean Flow (AMF), a flow-based IL method that enables smooth and responsive, fully closed-loop robot control. AMF uses Mean Flow, which is an accelerated form of Flow Matching (FM), to minimize prediction latency. To ensure smoothness and consistency across predictions, AMF uses a corrupted version of the trajectory from the previous step when predicting new robot actions, with the signal-to-noise ratio increasing over the time parameter of the trajectory. This discourages large changes in the prediction from one step to the next, while allowing freedom to adapt the predictions for future steps. We evaluate AMF across a wide range of simulated and real robot tasks and demonstrate significantly improved performance compared with baselines. Code: https://github.com/akselva/Adaptive-mean-flow-RoboticIL.

#14 h=18

Bilinear Flow Policy: Distributional Extrapolation for Goal-Conditioned Visuomotor Imitation

2026-10-05 cs.RO 👤 M. Itkina (h=18)

Wonsuhk Jung, Sundhar Vinodh Sangeetha, Chen Xu, Abhishek Gupta, Masha Itkina

Core Contributions

Names and formalizes 'distributional extrapolation': generalizing to goals outside demonstration support without collapsing to a single mode.
Retrieves a learned 'anchor' demo and expresses the novel observation-goal pair as anchor + residual; a bilinear flow models how both shape the action distribution.
Proves out-of-distribution action error is bounded by in-distribution flow-matching error (under assumptions), yielding pre-deployment diagnostics for which goals will work.
2.63× OOD success vs a standard GCIL flow policy and 1.36× vs the best extrapolation baseline in sim; +32% on two real tasks.

Show abstract

Goal-conditioned imitation learning (GCIL) with flow matching is a promising framework that can represent multimodal behaviors while adapting to diverse, user-specified goals, yet often fails when goals lie outside the demonstration support. To extrapolate to such unseen goals without collapsing multimodality - a problem we call distributional extrapolation - we introduce Bilinear Flow Policy (BFP), a generative visuomotor policy that combines transductive retrieval with a bilinear conditional flow. Given an unseen observation-goal pair, BFP retrieves an "anchor" training example and transductively reformulates the unseen pair as this familiar anchor plus a residual term. For this decomposition to guide action prediction, the residual must compactly encode how the current observation-goal pair differs from the anchor, and the anchor must be chosen so that this difference is predictive of the corresponding action distribution. BFP achieves this with pretrained visual features and a novel learned anchor-selection algorithm. The novel bilinear flow then models how the anchor and the residual jointly determine the multimodal action distribution. We prove that, for bilinear flow under suitable assumptions, action distribution error at unseen goals is bounded by the in-distribution flow-matching error up to problem-dependent factors. Across five manipulation tasks in simulation, BFP achieves 2.63x the out-of distribution success rate of a GCIL policy and 1.36x that of the strongest extrapolation-targeted baseline. On two real-world tasks, BFP improves over GCIL by 32%. Finally, our theory yields practical, pre deployment diagnostics for predicting which trained policies will extrapolate well and to which unseen goal.

#15 h=16

H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning

2026-10-05 cs.LGcs.RO 👤 Yann LeCun (h=16)

Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun, Randall Balestriero

Core Contributions

Trains a hierarchy of action-conditioned JEPAs end-to-end, each level in its own latent space and predicting farther ahead — unlike multi-horizon models sharing one latent.
Planning is top-down: higher-level predictions become subgoals for the level below, letting upper levels discard fast unpredictable detail.
Visual AntMaze success rises from 18% to 73% with a three-level hierarchy while using less planner compute; ablations credit both temporal decomposition and learned goal representations.
With inverse-dynamics supervision it extends to DROID real-robot video — a step toward LeCun's long-advocated hierarchical JEPA planning on real data.

Show abstract

Long-horizon planning with latent world models requires reasoning across timescales and levels of abstraction. Existing task-agnostic JEPA world models predict and plan at a single timescale or with multiple horizons in one shared latent space. We introduce H-JEPA, an end-to-end recipe for training a hierarchy of action-conditioned JEPAs in which each level predicts farther ahead in its own learned latent space. Planning proceeds top-down: the top level optimizes progress toward the goal, and each level's predictions become subgoals for the planner below it. When factors in the data evolve at separated timescales, higher levels discard fast, unpredictable detail and retain slower task-relevant state. Across four simulated navigation and manipulation environments, hierarchical planning improves over a flat JEPA; on Visual AntMaze, a three-level hierarchy raises success from 18% to 73% using less planner compute. Ablations attribute these gains to both temporal decomposition and higher-level goal representations. With inverse-dynamics supervision, the approach extends to diverse real-robot videos from DROID, where hierarchy improves offline planning fidelity at lower planner compute.

#23 h=11

RealtimeWAM: One-Step Asynchronous World Action Models

2026-10-05 cs.CVcs.LGcs.RO 👤 Rui-Hao Gong (h=11)

Chengtao Lv, Jinyang Du, Shuyi Feng, Yang Yong, Shiqiao Gu

Core Contributions

Identifies two latency sources in efficient World Action Models: multi-step action denoising and the video expert blocking the action expert.
Teacher-Anchored Consistency Distillation adds supervision from the teacher's multi-step endpoint, fixing a 'local-global gap' where low per-step consistency error still yields wrong final actions.
Cross-Expert Wavefront Pipelining overlaps the two experts by block-wise sharing of the video KV cache, synchronizing only when action attention needs it.
~25× end-to-end speedup on H100 with <1% performance drop on LIBERO, LIBERO-Plus, and RoboTwin; code and checkpoints released.

Show abstract

World Action Models (WAMs) incorporate visual representations from video generation backbones to guide action prediction. Recent efficient WAMs adopt Mixture-of-Transformers (MoT) architectures and compute video representations once for reuse by the action expert. However, intra-expert iteration (\ie, multi-step action denoising) and inter-expert waiting (\ie, sequential execution of the video and action experts) still limit inference efficiency. To this end, we present RealtimeWAM, an extremely efficient WAM variant with one-step action generation and asynchronous inference, addressing these two bottlenecks. To reduce intra-expert iteration, we propose Teacher-Anchored Consistency Distillation (TACD) to address a local-global error gap: low local consistency error alone does not guarantee accurate final actions. TACD supplements local consistency with explicit supervision from the frozen teacher's multi-step rollout endpoint, enabling accurate one-step action generation. Additionally, we propose Cross-Expert Wavefront Pipelining (CEWP) to eliminate unnecessary expert-level waiting. It overlaps the two experts through block-wise sharing of the video KV cache, synchronizing only immediately before the corresponding action attention consumes it. Extensive experiments across diverse benchmarks (\eg, LIBERO, LIBERO-Plus and RoboTwin) and model variants (\eg, Fast-WAM and Faster-WAM) demonstrate the superiority of RealtimeWAM. Notably, RealtimeWAM maintains near-lossless performance (\ie, $<1\%$ drop) across these benchmarks while delivering significant end-to-end speedup (\eg, $\sim25\times$ on H100). Our code and checkpoints are available via this \href{https://github.com/ModelTC/LightX2V/tree/main/examples/realtimewam}{link}.

Data Engines, Deployment & Evaluation

#12 h=18

InterMimicGen: Scaling Humanoid Loco-Manipulation through Self-Evolving Motion Imitation

2026-10-05 cs.ROcs.CVcs.GR 👤 Xue-Ting Li (h=18)

Yucheng Zhang, Sirui Xu, Jinhong Li, Liuyu Bian, Anatulya Nandi

Core Contributions

Turns sparse, heterogeneous human-object mocap into a growing, robot-executable dataset via a data flywheel where data and tracker improve each other.
Contact-preserving retargeting keeps whole-body coordination and dexterous hand-object relationships across robot morphologies — a common failure point for prior retargeting.
Each round applies small task-preserving edits (where and how an interaction happens), fine-tunes the generalist tracker, and keeps only variants that succeed in physics simulation.
Edits compound into broad coverage around the original demos while preserving semantics; one generalist tracker handles the set and transfers to real humanoids.

Show abstract

Captured human-object interactions provide rich supervision for humanoid loco-manipulation, but they are sparse, heterogeneous, and not directly executable by robots. We introduce InterMimicGen, a self-evolving motion-imitation framework in which robot motion data and a tracking policy improve each other. First, we consolidate motion-captured human-object interaction datasets and retarget them into humanoid robot references while preserving whole-body coordination and dexterous hand-object relationships. This produces a large and diverse humanoid robot reference collection for dexterous whole-body loco-manipulation. Second, we train a physics-based generalist tracker that executes these references in simulation on a humanoid with dexterous hands, covering a scale and diversity beyond prior humanoid tracking systems for loco-manipulation. Third, we close a data flywheel: each round makes small, task-preserving changes to where an interaction takes place and how the body performs it, fine-tunes the tracker on them, and keeps only the variants whose simulated execution completes the task, which seed the next round. With more iterations, these small edits compound into broader coverage around the sparse original demonstrations while preserving task semantics and motion quality. Experiments show contact-preserving retargeting across robot configurations, broad tracking with a single generalist policy, executable motions that keep growing over augmentation rounds, and transfer to real robots. InterMimicGen provides a unified path from heterogeneous human demonstrations to a continually expanding motion resource for humanoid robot learning.

#16 h=16

ArtifactArena: Evaluating Models by What They Build in the Physical World

2026-10-05 cs.ROcs.AI 👤 B. Katz (h=16)

Kushagra Tiwary*, David Mayo*, Nikhil Behari, Xiangzhou Sun, Abdulrahman Alabdulkareem

Core Contributions

Evaluates frontier models by what they build: models must co-design robot hardware and software that compete head-to-head in a physics arena.
Three harnesses separate zero-shot design, verifier-guided refinement with simulator feedback, and open-ended improvement from gameplay data.
Elo rankings from artifact-vs-artifact tournaments create a non-saturating benchmark — unlike static QA suites, the bar rises as entrants improve.
Open submission infrastructure positions it as a living testbed for physical-world engineering ability of LLM agents.

Show abstract

To evaluate the frontier, we must measure models not by what they say, but by what they can engineer and build in grounded physical environments. We introduce \textsc{ArtifactArena}, an open-ended platform where models face a physically grounded hardware-software co-design challenge: engineering fully functional robots to compete in a simulated arena. We evaluate a frontier model's zero-shot, verifier guided refinement, and open-ended physical design capabilities through three harnesses that refine their bot artifacts based on text descriptions, physics simulator feedback, and gameplay data. We benchmark these capabilities with an Elo ranking of frontier models derived from head-to-head tournaments between their artifacts. By releasing this framework and tournament infrastructure for ongoing community submissions, we establish a living, non-saturating testbed to continuously measure the expanding limits of open-ended intelligence in the physical world. Please visit \href{https://artifactarena.ai}{https://artifactarena.ai} for more information.

#21 h=13

Towards Robust Prehensile Manipulation in Open-Ended Environments

2026-10-05 cs.RO 👤 F. Ben Amar (h=13)

Mathilde Kappel, Mahdi Khoramshahi, Louis Annabi, Faïz Ben Amar, Stéphane Doncieux

Core Contributions

Applies Quality-Diversity search to prehensile manipulation, seeking many diverse robust grasps rather than a single optimum.
Uses those grasps as starting points to generate diverse manipulation trajectories on articulated objects.
The bet: behavioral diversity, not a single best policy, is what lets a robot keep working as open-ended environments change. (Short paper; no quantitative results reported.)

Show abstract

We propose to use Quality-Diversity (QD) algorithms to solve robotic prehensile manipulation tasks in open-ended environments. Our approach enables the efficient discovery of a wide range robust grasp configurations, which serve as reliable starting points for generating diverse prehensile manipulation trajectories on articulated objects. The resulting diversity in manipulation behaviors enhances generalization and adaptability, enabling effective deployment continuously evolving open-world settings.

#26 h=10

Demonstration-Calibrated Port-Hamiltonian Retuning for Manipulation Policies

2026-10-05 cs.ROeess.SY 👤 Amit Chakraborty (h=10)

Yulong Yang, Fan Wu, Christine Allen-Blanchette, Amit Chakraborty

Core Contributions

Points out an overlooked knob: VLA/diffusion policies run through impedance controllers whose gains are inherited from data collection, not chosen for the deployed policy.
Learns a port-Hamiltonian model from demonstrations to estimate the effort and energy implied by the policy's actions, then derives a single gain scale in closed form.
Fully offline — no rollouts, gain sweeps, manipulator model, rewards or retraining; gains match the best found in empirical sweeps.
Up to +9.4 pts for Diffusion Policy on LIBERO, improvements on all four real tasks, and gains for SmolVLA and OpenVLA-OFT with reduced jerk.

Show abstract

Diffusion and VLA policies for manipulation are often deployed through downstream impedance controllers. The stiffness and damping gains of these controllers affect task success, yet are commonly inherited from data collection rather than selected for the deployed policy. Although empirical gain sweeps can improve performance, they require repeated evaluation rollouts. We introduce PHRetune, an offline method that derives controller gains for a frozen policy without evaluation rollouts or gain search. Our approach learns a port-Hamiltonian model from demonstrations to estimate the effort and energy associated with the policy's predicted actions. The policy is applied to recorded demonstration observations, and its predictions are assessed against demonstration-derived effort and energy budgets. From this comparison, we derive a single gain scale in closed form, adjusting the downstream controller while preserving the policy and its action representation. The gains are fixed before evaluation, without requiring a prior manipulator model, task rewards, policy retraining, or additional runtime computation. Across LIBERO suites, PHRetune improves Diffusion Policy success by up to 9.4 percentage points, with the derived gains achieving the highest observed success rates in empirical gain sweeps. On all four real-world manipulation tasks, PHRetuned Diffusion Policy outperforms the nominal policy, alternative gain-tuning methods, and a policy-retraining baseline. The same procedure improves success with SmolVLA and OpenVLA-OFT on every task, while reducing acceleration and jerk for both VLA backbones.

#28 h=9

Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning

2026-10-05 cs.ROcs.LG 👤 Chelsea Finn (h=9)

Lars Ankile, Perry Dong, Rohan Bhowmik, Aneesh Muppidi, David D. Yuan

Core Contributions

Treats the initial-state distribution of supervised deployment as a decision: start episodes at observed failures and untried states instead of sampling uniformly.
Motivated by the observation that failures cluster in a small subset of placements, so uniform collection wastes operator time on solved cases.
Adds a value function trained on all data, including failures that imitation discards (HiL-IDQL), for value-based action selection.
+10–34 points real-task success at matched budgets, evaluated on 2,550 blinded held-out episodes; human-robot team completes 98% of collection episodes.

Show abstract

Learning from human demonstrations is a reliable way to teach robots new tasks, but the gains from each additional demonstration shrink as the policy improves. Continued improvement can instead come from supervised deployment, where an operator places the objects and intervenes when the policy fails. We ask how to maximize improvement from a fixed budget of supervised episodes on high-precision manipulation tasks with wide ranges of object placements. We observe that failures can concentrate in a small subset of initial states, so uniform collection spends much of the operator's time on states the policy already handles. Mulligan makes the initial-state distribution a decision, starting each round's episodes at observed failures and untried states. To further improve data efficiency, we augment interactive imitation learning with a value function trained on all data, including failures that imitation discards. Across three real-world tasks evaluated on 2,550 held-out, blinded episodes and two simulated tasks, Mulligan outperforms uniform initial-state sampling at matched collection budgets, and combined with value-based action selection, HiL-IDQL+Mulligan, improves final real-task success by 10-34 percentage points. With operator interventions, the human-robot team completes 98% of collection episodes, remaining productive while the policy learns. Videos, code, and data are available at https://mulligan.page/.

Medical & Magnetic Servoing

#9 h=21

MagServo: Uncertainty-Resilient Hierarchical Magnetic Servoing via Learned Latent Representations

2026-10-05 cs.RO 👤 Zhong-Liang Jiang (h=21)

Yuhan Tan, Yameng Zhang, Pei Liu, Yao Zhong, Zhongliang Jiang

Core Contributions

Servos 6-DoF directly on learned latent magnetic features, sidestepping explicit pose estimation and analytical field models that propagate modeling error.
Masked-reconstruction pretraining makes the latent robust to noise and disturbances; a learned latent dynamics model replaces hand-derived Jacobians.
Hierarchical control — nonlinear MPC for coarse approach, local Jacobian inversion for fine regulation — reaches 0.386 mm / 0.479° mean terminal error.
Generalizes to unseen magnet-source configurations without retraining, relevant for capsule endoscopy and occluded medical navigation.

Show abstract

Magnetic navigation provides contact-free and line-of-sight-independent feedback for robotic systems, yet existing approaches typically rely on explicit pose estimation or direct use of raw magnetic measurements, making accurate control susceptible to modeling errors, measurement noise, and disturbances. This work presents MagServo, a hierarchical learning-based framework for robust 6-DoF magnetic servoing directly using the learned latent magnetic feature. MagServo learns uncertainty-resilient magnetic representations through masked reconstruction and captures state-dependent interaction dynamics between robot motion and latent magnetic transitions without analytical magnetic models or explicit Jacobian supervision. Based on the learned dynamics, a hierarchical controller combines nonlinear model predictive control for coarse approach with local Jacobian inversion for precise fine regulation. Extensive physical experiments demonstrate submillimeter and subdegree accuracy, achieving mean terminal errors of 0.386 mm and 0.479 degree for 6-DoF pose reaching. MagServo further outperforms a localization-based control baseline in complex trajectory tracking and maintains robust performance under unseen magnetic-source configurations without retraining. A supplementary video of the real-robot experiments is available at https://youtu.be/rZt1NUP1Mr0.

#10 h=21

Recon2Servo: Robotic Ultrasound Visual Servoing via Learned Image-to-Motion Inference

2026-10-05 cs.RO 👤 Zhong-Liang Jiang (h=21)

Yameng Zhang, Pei Liu, Dianye Huang, Yizhao Qian, Zhongyu Chen

Core Contributions

Recasts 2D-ultrasound visual servoing as relative-pose regression borrowed from trackerless 3D reconstruction, avoiding anatomy-specific priors and handcrafted features.
LoRA-adapted DINOv3 encoder plus a bidirectional relation module infer out-of-plane motion — the hard, ambiguous axis in B-mode imaging.
Combines supervised relative-pose learning with reconstruction-guided closed-loop adaptation and bounded residual correction to stay stable during iterative servoing.
Validated across ultrasound systems on a public dataset and 12-volunteer in-house data, plus real-robot forearm alignment and tracking.

Show abstract

Ultrasound visual servoing is essential for autonomous robotic ultrasound, yet 6-DoF probe control from 2D B-mode images remains challenging due to limited and ambiguous out-of-plane motion cues. Existing methods typically rely on anatomical priors or handcrafted visual features, limiting their generalizability across imaging targets. Inspired by trackerless 3D ultrasound reconstruction, we propose Recon2Servo, a visual servoing framework that learns image-to-motion inference directly from B-mode images for 6-DoF probe control. A DINOv3 encoder with low-rank adaptation and a bidirectional relation module estimate the relative probe pose between current and target images to guide iterative closed-loop target-view alignment. The framework combines supervised relative-pose learning, reconstruction-guided closed-loop adaptation, and bounded residual pose correction to improve motion inference during servoing. Evaluations on a public dataset and an in-house dataset collected from 12 healthy volunteers using different ultrasound systems demonstrate its effectiveness in reconstructed-volume servoing. Additional real-robot demonstrations of target-view alignment and dynamic tracking on a human forearm are provided in the supplementary video: https://youtu.be/qwsOdI-GMYk.

#30 h=9

Dual-Rate Force-Image Control with Model-Based Orientation Limits for Robotic Ultrasound

2026-10-05 cs.ROcs.CVeess.SY 👤 A B M Tahidul Haque (h=9)

Tyler Foster, Qiang Zhang, A B M Tahidul Haque, Anh Thu Nguyen

Core Contributions

Addresses a timing mismatch in robotic ultrasound: fast force loops vs slow, delayed image feedback, so image-driven probe rotation disturbs contact force before the image reacts.
Derives a closed-form orientation-rate limit from contact stiffness, force-loop gains, a rotation-to-force bound, and the excursion budget.
Rate limiting cuts RMS force error by 0.40 N at a 0.94 s convergence cost; candidly, a fixed cap near the predicted ceiling performs as well — the value is the design rule, not online prediction.
An honest ablation-driven paper (delay reconstruction had no resolvable effect) with single-subject in-vivo feasibility.

Show abstract

Robotic ultrasound couples a high-rate contact-force loop with slower, delayed image feedback, so image-guided ultrasound probe rotation can perturb contact force before the resulting image response is observed. We derive a closed-form orientation-rate limit that bounds the modeled rotation-induced estimated-force excursion over a finite horizon while accounting for disturbance rejection by the fast force loop. The limit depends on local contact stiffness, force-loop gains, a conservative rotation-to-force gain bound, the excursion budget, and the prediction horizon. We implement this model in a dual-rate controller with timestamp-based delay reconstruction and joint-torque-based force estimation, and evaluate it on a curved gelatin phantom using paired controller comparisons and component ablations. Relative to unconstrained image guidance, the proposed rate-limited controller reduced first-second root-mean-square (RMS) estimated-force error by 0.40 N while increasing cue-convergence time by 0.94 s. A fixed rate cap near the analytically predicted ceiling produced no resolvable difference in force error and converged 0.32 s faster, indicating that the principal practical value of the model is the rate-design rule rather than online prediction. Delay reconstruction had no resolvable effect at the tested latency. A single-subject popliteal scan demonstrated feasibility, although the image cue was noise-limited on heterogeneous tissue.

Navigation, Driving & Locomotion

#5 h=37

Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation

2026-10-05 cs.RO 👤 Ding Zhao (h=37)

Siyuan Liu, Miao Li, Haibao Yu, Haohong Lin, Qing Zhou

Core Contributions

Bridges two gaps in AV safety testing: trajectory-level generators are controllable but produce no pixels, while video generators produce pixels but are not controllable.
Pipeline: conflict trajectories → text-conditioned motion diffusion for 3D human motion → animatable 3DGS avatars rendered into multi-view sensor streams.
Built on HazardPed (10,352 traffic videos → 422 conflict trajectories, 857 annotated 3D motions), grounding risky pedestrian behaviors in real data.
Seven leading end-to-end driving models drop from mean HDScore 88.8 to 47.4 on 88 scenarios — strong evidence that current E2E benchmarks under-sample dangerous pedestrian behavior.

Show abstract

Evaluating end-to-end autonomous driving under rare, safety-critical vehicle-pedestrian interactions requires photorealistic, sensor-level scenarios. However, trajectory-based scenario generators cannot synthesize raw visual observations, whereas video-based approaches lack controllability. To bridge this gap, we present ControlPed, a novel framework that combines trajectory-level conflict synthesis with 3D Gaussian Splatting (3DGS) to generate photorealistic, motion-controllable safety-critical scenarios. Built upon HazardPed, a dataset derived from 10,352 traffic videos comprising 422 conflict trajectories, HD maps, and 857 annotated 3D human motions, ControlPed first generates conflict trajectories, lifts them into 3D human motion sequences via text-conditioned motion diffusion, and finally renders multi-view sensor observations using animatable 3DGS avatars. Safety evaluation in 88 rendered photorealistic scenarios reveals that seven leading end-to-end driving models suffer a severe performance drop, with their mean HDScore plunging from 88.8 to 47.4, exposing major failure modes under dangerous pedestrian behaviors. The dataset and testing benchmarks will be released to facilitate safety assessment of vehicle-pedestrian interactions.

#7 h=23

End-to-End Safe Social Navigation via Multi-Task Reinforcement Learning and Probabilistic Perception

2026-10-05 cs.ROcs.HC 👤 Antonio Giannitrapani (h=23)

Tommaso Van Der Meer Andrea Garulli, Antonio Giannitrapani, Renato Quartullo, Alberto Vaglio, Alexandre Alahi

Core Contributions

Removes a common cheat in RL social navigation: assuming ground-truth human positions and velocities. JESSI maps raw LiDAR straight to control.
Dirichlet-parameterized continuous actions plus deterministic bounding keep commands kinematically feasible and safety-bounded by construction.
An attention-based perception head is trained with a supervised human-state loss alongside RL; this multi-task signal measurably improves social behavior and makes the policy interpretable.
JAX-based and lightweight; deployed on a real differential-drive robot with better success/social-compliance balance than SOTA baselines.

Show abstract

Autonomous social navigation requires balancing efficiency, physical safety, and social compliance. Reinforcement Learning (RL) methods provide a viable and effective solution but often rely on unrealistic assumptions, such as the knowledge of humans' position and velocity. In this paper, we introduce JESSI (JAX-based E2E Safe Social Interpretable navigation), a lightweight end-to-end RL framework that maps raw LiDAR scans directly to kinematically feasible control commands. JESSI enhances safety via Dirichlet-parameterized continuous action spaces and deterministic bounding, while an integrated attention-based perception module extracts probabilistic human states for interpretable, socially aware decision-making. Through extensive simulations and real-world deployment on a differential-drive robot, we demonstrate that jointly optimizing the RL policy with a supervised perception signal in a multi-task paradigm enhances social behavior. Ultimately, JESSI is able to balance high navigation success rates and superior social behaviors compared to state-of-the-art baselines.

#13 h=18

Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped

2026-10-05 cs.ROcs.AIcs.HCeess.SY 👤 J. Sahoo (h=18)

Jagannath Prasad Sahoo, Saurabh Kumar, Surya Prakash S. K., Samiran Datta, Abhay Dwivedi

Core Contributions

Tackles obstacle avoidance on a strongly underactuated biped (no hip/ankle roll), where steering and balance are tightly coupled.
Jointly trains a high-level velocity-command policy (raycasts, obstacle states, receding-horizon goal) and a velocity-conditioned gait policy with SAC.
Fair comparison design: the frozen low-level gait is reused under A*, RRT*, and APF to isolate the benefit of learned high-level planning.
98% / 88% success in static / dynamic environments vs ≤78% / 68% for planner hybrids, with paths within 4% of A* length.

Show abstract

A bipedal robot cannot deviate from its path to avoid an obstacle without disturbing its balance, and this coupling is most severe on underactuated platforms such as the biped considered here, which has four actuated joints per leg and no hip or ankle roll. This paper presents a Hierarchical Reinforcement Learning (HRL) framework in which a High-Level (HL) policy observes the robot pose, 36 raycast proximity measurements, moving-obstacle states, and a receding-horizon local goal, and outputs a body-velocity command $(v_x, v_y, ω_{yaw})$ every ten control steps, while a velocity-conditioned Low-Level (LL) policy tracks each command through PD-controlled joint targets. Both policies are trained jointly with Soft Actor-Critic (SAC). Because the converged gait is task-agnostic, it is frozen and driven by classical planners over the same command interface, yielding three controlled baselines: SAC+A*, SAC+RRT*, and SAC+APF. Across 100 evaluation trials per method in randomized PyBullet environments, the proposed method reaches the goal in 98.0% of static and 88.0% of dynamic trials, against at most 78.0% and 68.0% for the planner hybrids, with path lengths within 4% of the A* reference, and ablations confirm that each observation channel and reward term contributes materially to this performance.

#19 h=14

Radar2Plan: Benchmarking 4D Radar for End-to-End Open-Loop Ego-Trajectory Planning

2026-10-05 cs.RO 👤 Fang-Qiang Ding (h=14)

Ling Yao, Yichun Xiao, Jin Jin, Yihan Zhang, Fangqiang Ding

Core Contributions

First benchmark dedicated to real-world 4D radar for end-to-end ego-trajectory planning, rather than radar perception alone.
Modular interfaces between encoders, scene representations and planning heads enable controlled sensor-vs-planner comparisons.
Seven sensor combinations × four planners × 12 weather/illumination conditions on DSERT-RoLL and MAN TruckScenes under one protocol.
Radar alone supports competitive planning and is robust in adverse conditions, but its fusion benefit depends on planner architecture — a caution against one-size-fits-all fusion claims.

Show abstract

Adverse weather and poor illumination remain major challenges for robust ego-trajectory planning in mobile autonomy. 4D radar offers reliable sensing under adverse conditions and direct radial-velocity measurements. However, sensing robustness does not necessarily translate into robust downstream planning, while existing 4D radar benchmarks focus primarily on perception rather than trajectory planning. We present Radar2Plan, a modular benchmark for open-loop ego-trajectory planning using real-world 4D radar data. Radar2Plan connects sensor encoders, scene representations, and planning heads through common interfaces, enabling controlled comparisons between different sensor and planner configurations. Using DSERT-RoLL and MAN TruckScenes, we evaluated seven combinations of camera, 4D radar, and LiDAR in four representative planning baselines and 12 weather and illumination conditions under a unified protocol. To our knowledge, Radar2Plan is the first benchmark dedicated to evaluating real-world 4D radars for autonomous driving planning. Experiments show that 4D radar alone supports competitive ego-trajectory planning and robust performance under adverse conditions. Sensor-configuration comparisons further demonstrate its complementary value to other modalities, while revealing dependencies on sensor combination and planning architecture. The modular design also supports additional datasets, sensing modalities, and planners, providing a flexible foundation for future research on radar-based autonomous driving planning.

#25 h=11

Human-in-the-Loop Neuro-Symbolic Drift Anticipation for Reliable Visual SLAM

2026-10-05 cs.RO 👤 Wonse Jo (h=11)

Junhyun Nam, Wonse Jo

Core Contributions

Moves V-SLAM from reacting to drift toward anticipating it, by predicting drift risk before tracking fails.
Pairs neural drift-risk estimation with symbolic constraints so predictions stay physically consistent out-of-distribution.
Uses an LLM to translate qualitative human context (e.g., 'corridor ahead is featureless') into interpretable symbolic constraints — a human-in-the-loop channel rarely seen in SLAM. (Preliminary; limited quantitative detail.)

Show abstract

This paper introduces Hybrid DeepSEE (HDS), a Human-in-the-Loop (HITL) neuro-symbolic framework for proactive drift anticipation in Visual SLAM (V-SLAM). While data-driven models offer predictive power, their "black-box" nature often yields physically inconsistent outputs in out-of-distribution (OOD) environments. To address this, HDS integrates neural drift risk estimation with symbolic constraint reasoning. By utilizing a Large Language Model (LLM) as a reasoning bridge, the framework translates qualitative human context into interpretable symbolic constraints. Building upon this architecture, we propose a superior drift anticipation framework that ensures enhanced reliability and consistency in Visual SLAM

Multi-Robot & Human-Robot Collaboration

#3 h=39

A Taxonomy on Collective Awareness

2026-10-05 cs.RO 👤 P. Campoy (h=39)

Guillermo GP-Lenza, Miguel Fernandez-Cortizas, Martín Molina, Ricardo Sanz, Pascual Campoy

Core Contributions

Argues that Mutual, Shared and Team Situational Awareness — used interchangeably in multi-robot papers — form a strict containment hierarchy with different coordination demands.
Organizes them on two axes (what is being known vs. how uniformly that knowledge is distributed), giving system designers a vocabulary to specify what a team must actually share.
A heterogeneous aerial-ground search-and-rescue case study maps each taxonomy cell to concrete communication and coordination requirements.
Matters because mislabeled awareness requirements lead to over- or under-engineered communication stacks; this is a specification tool rather than an algorithm.

Show abstract

As robotic teams tackle increasingly complex tasks in dynamic and unstructured environments, effective coordination requires agents to maintain accurate, aligned representations of their environment, teammates, and mission state. We argue that Mutual Awareness, Shared Situational Awareness, and Team Situational Awareness --- three concepts widely invoked in the multi-robot systems literature --- are not interchangeable: they form a containment hierarchy in which each type subsumes the previous in scope, and span a distribution spectrum from fully individualized understanding (MA) to fully uniform understanding (SSA), with TSA occupying a mixed position. We establish this through a two-dimensional taxonomy organized along the object of awareness and awareness distribution axes. Treating these terms as synonyms obscures the precise coordination requirements each imposes on a robotic system. A Search and Rescue case study with heterogeneous aerial and ground robots grounds each taxonomy position in concrete coordination requirements. These findings provide a conceptual foundation for principled specification, design, and comparison of collective awareness in multi-robot systems.

#6 h=23

GAMBIT: Learning to Plan Continuous Multi-Robot Trajectories

2026-10-05 cs.ROcs.AIcs.LGcs.MA 👤 Wolfgang Hönig (h=23)

Rishabh Jain, Akmaral Moldagalieva, Lorenzo Magnino, Michael Amir, Keisuke Okumura

Core Contributions

Learns the 'sacrifice' behaviors — a robot yielding locally so the team moves faster — that hand-designed multi-robot heuristics struggle to encode in dense traffic.
Operates over motion primitives for double-integrator dynamics: imitation learning from planner demonstrations, then RL fine-tuning beyond the teacher.
A safeguarded rollout with backup trajectories gives hard collision-freedom guarantees regardless of what the learned policy proposes — learning for efficiency, verification for safety.
Coordinates 1,000+ robots in continuous space with planning latency of a few hundred ms, outperforming both centralized planners and decentralized reactive methods.

Show abstract

GAMBIT is an opening chess move in which a player sacrifices a piece, typically a pawn, to gain a positional advantage later in the game. Analogously, in multi-robot coordination, individual robots may need to forgo locally reward-maximising behaviours to improve overall team performance. Such self-sacrificial behaviours are difficult to capture with manually designed heuristics, particularly in dense, interaction-rich environments. Focusing on double-integrator continuous dynamics, this work studies how to learn such coordinated heuristics over motion primitives for multi-robot trajectory execution. Our framework, GAMBIT, first learns coordinated motion-primitive selection through imitation learning and subsequently fine-tunes the policy through reinforcement learning. We further introduce a safeguarded rollout mechanism with backup trajectories that guarantees collision-free execution at all times. Experiments demonstrate that GAMBIT substantially outperforms a range of baselines, including centralised motion planners and decentralised reactive planners, while exhibiting strong scalability. In particular, it coordinates over a thousand robots with planning latency below a few hundred milliseconds in continuous domains.

#11 h=20

From Social Reasoning to Embodied Interaction: An Agentic Framework for Social Robots

2026-10-05 cs.RO 👤 Ye Shi (h=20)

Ziyu Cheng, Yuewen Guo, Zhirui Liu, Dong Zhang, Haotao Lu

Core Contributions

Closes the loop most social-robot work leaves open: when to engage, what to say, and how to physically express it on a humanoid face and body (Sophia).
Couples a foundation-model agent with long-term memory and proactive engagement to a robot-native expression layer (gestures, speech, mechanical facial expressions).
Streaming execution starts physical expression before full generation completes, substantially cutting response latency — a practical bottleneck for face-to-face HRI.
Shows that disembodied LLM agents plus a puppeted robot are not enough; reasoning must be co-designed with physical expression constraints.

Show abstract

Natural face-to-face human--robot interaction requires a robot to understand an evolving social situation, decide when to engage, and express its intent through coordinated physical behavior. Yet existing approaches rarely close this loop: foundation-model agents provide increasingly capable multimodal reasoning and memory but remain largely disembodied, while expressive virtual agents do not face the physical constraints of real robots, and physical social robots typically address social reasoning and embodied expression only partially. We present ARISE, a unified framework that bridges Agentic Reasoning and Interactive Social Embodiment on the Sophia humanoid robot. ARISE integrates multimodal context understanding, long-term memory, and reactive and proactive interaction to determine when and what to communicate, and translates social intent into robot-native gestures coordinated with speech and mechanical facial expressions through streaming execution. Extensive evaluations on Sophia demonstrate strong perceived interaction quality, expressive and well-coordinated embodied behavior, and substantial latency reductions through streaming execution. These results highlight the importance of jointly reasoning about what to communicate, when to engage, and how to physically express social intent for natural interaction with humanoid robots. Project Page: https://robosocial.github.io/

#17 h=16

Towards Quadruped-Provided Localization and Active Tracking for Micro-UAVs

2026-10-05 cs.RO 👤 A. Faíña (h=16)

Alejandro Lorite Mora, Andrés Faíña

Core Contributions

Offloads the full localization stack from a 27 g micro-UAV to a quadruped with a 7-DoF arm, letting tiny drones fly autonomously where they cannot carry sensors.
Composes arm-camera AprilTag detections with the quadruped's own self-localization, placing the drone in a shared map frame rather than just relative to a camera.
The arm acts as an active observer that repositions to keep the drone close and in view; this cuts tracking error from 11.0 cm to 6.9 cm.
External pose accuracy of 12–16 mm supports flight within 2–5 cm of mocap-fed control — a practical heterogeneous team for confined-space inspection.

Show abstract

Micro unmanned aerial vehicles (micro-UAVs) are small enough to reach confined spaces that larger robots cannot access, but too small to carry the sensing and computing power required for autonomous flight. We move the localization stack entirely off the aerial platform onto a quadruped robot with a 7-degree-of-freedom (DOF) arm, which supplies the micro-UAV (27 g bare, 42 g with fiducial markers) its full 6-DOF pose. A camera at the arm's end-effector detects AprilTag fiducial markers on the drone and composes that observation with the quadruped's own self-localization to place the drone in a shared map frame, so the ground robot localizes its partner, rather than only tracking it relative to the camera. The arm acts as an actively-controlled observer, repositioning to keep the drone in view as both robots move; the drone carries only an inertial measurement unit and fuses the external pose to fly commanded setpoints. In lab flights the external pose is accurate to 12-16 mm, enough to fly the drone autonomously within 2-5 cm of motion-capture-fed control. Having the quadruped actively follow the drone reduces the tracking error from 11.0 cm to 6.9 cm by holding the camera in the close range, where the markers are most accurate.

#20 h=13

Physics Residual Dynamics and Reduced Order Whole-Body Planning for Obstacle Aware Human Robot Cloth CoTransportation

2026-10-05 cs.RO 👤 Gokhan Alcan (h=13)

Moein Forouhar, Kosar Behnia, Anirvan Dutta, Hamid Sadeghian, Ville Kyrki

Core Contributions

For human-robot cloth co-transport, obstacle clearance depends on the unactuated interior of the cloth, not just grasp points — so deformation must be predicted.
A physics-residual conditional recurrent VAE learns corrections to a cheap linearized cloth model, limiting error drift over a 40-step horizon.
Cloth prediction is embedded in MPPI over a reduced-order model of a dual-arm mobile manipulator (non-holonomic base preserved), cutting compute ~80% vs the full 17-DoF model.
Maintains clearance where a corner-following baseline collides; whole-body MPC refinement reduces final cloth deformation from 0.93 m to 0.28 m.

Show abstract

Human--robot co-transportation of deformable objects requires predicting object deformation during motion, since obstacle clearance depends on both the grasp points and the unactuated interior. We present a hierarchical planning framework that combines a learned cloth model with a reduced-order whole-body model of a dual-arm mobile manipulator. A physics-residual conditional recurrent variational autoencoder (p-cRVAE) predicts the full cloth configuration from grasp-point observations by learning a residual correction to a computationally efficient linearized physics model, limiting error accumulation over 40-step planning horizon. The predicted cloth dynamics are embedded in a model predictive path integral (MPPI) planner using a reduced-order representation of a dual-arm mobile manipulator that preserves the non-holonomic base constraint and arm workspace limits. An MPC layer subsequently refines the sampled motion into smooth, executable references for whole-body control. The reduced-order formulation achieves tracking performance comparable to the full 17-DoF model while reducing computation time by approximately 80%. Across four co-transportation scenarios and two carrying speeds, the proposed framework maintains cloth-obstacle clearance where a corner-following baseline results in collisions, while whole-body refinement reduces final cloth deformation from 0.93\,m to 0.28\,m.