🤖 Robotics arXiv Digest

Tuesday, September 29, 2026

📄 30 papers 📂 7 research areas Generated by Claude

🔭 Research Landscape

Today's batch has an unusually sharp message: the field's evaluation tools are catching up with its models, and the news is sobering. Four papers build diagnostics that separate failure modes aggregate success rates blur together. LIBERO-MAX (#2) holds seed, initial state and pre-event actions fixed and shows that a single mid-task change (target moves, camera shifts) costs fourteen current VLA and world-action policies 11–26 points. EgoGears (#1) finds that every MLLM tested loses on average 22.5 points when evidence must be bound across encounters rather than within one video, and RoboChrono (#19) finds a model scoring 95.4% on frame matching yet 32.9% on frame ordering. Cross-skeleton retargeting (#23) goes further and proves that standard objectives cannot even identify the map they claim to learn. The shared lesson: temporal state — what happened, in what order, to which object — is the weak link in today's embodied models.

A second cluster responds directly to that weakness by wrapping frozen policies with lightweight runtime modules instead of retraining them. SimpleARM (#8, sharing authors with LIBERO-MAX) keeps compact typed state rather than raw frames and lifts memory-dependent success from 44.5% to 67.2%; Spotter (#26) lets a VLM watch in parallel and only intervene on detected errors, raising π_0.5 from 53% to 83% on a real robot; RawVLA (#30) inserts a neural ISP ahead of the frozen VLA; and both Rho (#28) and the multi-agent RFT pipeline (#27) adapt by doing RL in the latent noise space of a frozen flow-matching action expert — a recipe that is quietly becoming the standard way to fine-tune π-style models (Rho needs only 15 corrected episodes). Even BlenDAgger (#12) fits this mould, blending rather than replacing policy actions during human corrections.

The third theme is data amplification for humanoids and manipulation from cheap human sources, with physics used as the filter. PRISM (#3) multiplies a few real videos into hundreds of counterfactual ones via video generation and then projects them onto physically plausible trajectories; EgoAlign (#9) puts the target controller in the loop while humans collect egocentric demos; VidAct (#14) and Video2STL (#10) extract object-centric 3D motion or formal temporal-logic specs from single videos; CrossBFM (#29) makes the behavior latent itself portable across humanoids. On the classical side, the batch shows that the right structure can beat brute fidelity: FORM (#6) turns material identification into a linear least-squares solve (minutes → seconds), and the robotic-fish paper (#13) transfers zero-shot from a deliberately crude fluid model because the controller is constrained to the identified band. Across driving, PhysWAM (#22) and V2X-WAM (#4) similarly show world-action models improving when the predicted world is forced to be geometrically or causally consistent with the plan.

VLA Models & Runtime Adaptation

Open VLA foundations plus the memory, reflection, RL and imaging layers wrapped around frozen policies at deployment.

#28 Rho: A Foundation for Efficiently Adaptable VLA Models
#26 Spotter: Let the Embodied Model Lead, and the VLM Refle...
#8 Simple Agentic Memory for Generalist Robot Policies
#27 Cooperative Multi-Agent Vision-Language-Action Models v...
#30 RawVLA: Embodied Neural Image Signal Processor For Robo...

Benchmarks & Diagnostic Evaluation

Controlled protocols that separate failure types aggregate scores hide — mid-task change, cross-video binding, temporal ordering, identifiability.

#1 Does Local Video Understanding Transfer Across Encounte...
#2 LIBERO-MAX: Do Robot Policies Adapt When the World Chan...
#19 RoboChrono: A Real Robot Benchmark for Streaming Task U...
#23 Why Cross-Skeleton Retargeting Is Non-Identifiable: Str...

Humanoid, Legged & Bio-inspired Control

Whole-body humanoid skills from human data, shared cross-robot behavior latents, safety curricula, and a soft robotic fish.

#3 Counterfactual Video Generation Enables Scalable Humano...
#9 EgoAlign: Bridging the Human-Humanoid Gap for Long-Rang...
#29 CrossBFM: Distilling a Shared Latent Behavior Space Acr...
#17 Predictive Safety Curricula for Robust Legged Locomotio...
#13 All You Need Is Low Fidelity: Zero-Shot Sim-to-Real of...

Learning from Videos, Demos & Specifications

Turning videos, human corrections and a few demos into rewards, trajectories or skill libraries.

#10 Video2STL: Grounding VLM-Generated Temporal Specificati...
#14 VidAct: Learning Manipulation from In-the-Wild Videos w...
#12 BlenDAgger: Blended Shared Control for Interactive Imit...
#15 DROM: A Language-Guided Diffusion Framework for Multi-S...

Contact-Rich & Physics-Aware Manipulation

Identifying material laws and hidden kinematics, and regulating force through haptics.

#6 FORM: Robot Manipulation through Direct Material Law Id...
#16 Learning to Explore Hidden Kinematics for Articulated O...
#24 HACo: Learning Haptic Active Compliance for Force-Aware...

Perception, SLAM & Field Robotics

Robust perception for long deployments: glass, domain shift, dense SLAM, and underground mines.

#5 When to Adapt: Multi-Signal Domain Shift Detection for...
#7 Towards Spatial Perception for Heterogeneous Robot Coll...
#11 GlassFormer: Learning Real-time Glass Segmentation usin...
#18 Pow3R-SLAM: Real-Time RGB-D SLAM with 3D Reconstruction...

Autonomous Driving & Multi-Agent Planning

World-action models for driving, personalized ACC, contingency games and diffusion multi-robot planning.

#4 V2X-WAM: A Cooperative World Action Model for End-to-En...
#22 PhysWAM: Physically Consistent World Action Model for A...
#20 Learning from Shared-Control Overrides: Context-Driven...
#21 Cascaded consensus splitting for multi-branch contingen...
#25 Trajectory-Level Mode Guidance for Controllable Diffusi...

VLA Models & Runtime Adaptation

#28

h=18

Rho: A Foundation for Efficiently Adaptable VLA Models

📅 2026-09-29cs.RO👤 T. Hellebrekers (h=18)

Rho Team, Simran Bagaria, Daphne Chen, Dean Fortier, Jianlong Fu

Core Contributions

An open-weights VLA family targeted at three widely used dual-arm platforms (YAM Box, UR AI Trainer, FR3 Duo) with systematic ablations of action-expert architecture and training recipe.
Shows that embodiment-specific midtraining improves downstream data-light adaptation, matching or beating existing open VLAs across evaluated tasks.
Built-in online adaptation: a small latent policy learns to choose noise inputs for the frozen flow-matching expert, handling edge-of-distribution situations with just 15 corrected episodes.
Release of base and embodiment checkpoints lowers the barrier for labs using common commercial bimanual hardware.

Show abstract

General-purpose physical AI models must combine broad visual and linguistic capabilities with precise control across robot embodiments and efficient adaptation to downstream tasks. We introduce Rho, a family of open-weights VLA models for bimanual manipulation designed for data-light task adaptation on 3 embodiments representative of dual-arm robots across research labs and the industry -- YAM Box, UR AI Trainer, and FR3 Duo. We systematically ablate Rho's action-expert architecture and training recipe, and show in controlled simulation and physical-robot experiments that embodiment midtraining improves downstream adaptation. The resulting Rho variants for YAM Box, UR AI Trainer, and FR3 Duo match or outperform existing open-weights VLAs and achieve the strongest overall performance across the tasks, embodiments, and baselines evaluated in this report. We further demonstrate the Rho model family's built-in capacity for online adaptation: an internal latent policy learns from corrective feedback to select observation-conditioned noise inputs for the frozen flow-matching action expert. With as few as 15 corrected episodes, adapting this lightweight module enables Rho to handle task situations at the fringe of its offline finetuning distribution. Together, these results position Rho as both a strong general-purpose robotic manipulation model and a practical foundation for adaptation. We release the base Rho model and the embodiment-specific checkpoints to facilitate Rho's deployment in research experiments and practical industrial use cases.

#26

h=19

Spotter: Let the Embodied Model Lead, and the VLM Reflect for It

📅 2026-09-29cs.ROcs.AI👤 Yue Yang (h=19)

Long Li, Qichao Zhao, Yue Yang, Fan Xu, Zhe Wang

Core Contributions

Shows that current embodied policies rarely repair a known failure on retry — neither by sampling, a language hint, nor best-of-N — because they are trained only on successes.
Reverses the usual VLM-planner / policy-tool hierarchy: the embodied model leads, while a VLM runs in parallel behind a lightweight screener and intervenes only on confirmed errors.
With GPT: +5.6 / +7.5 points for Cosmos Policy and π_0.5 on RoboCasa, π_0.5 47.2%→57.0% on RoboTwin 2.0 Hard, and 53%→83% on a real robot.
Keeps the VLM off the critical path: successful episodes take only 13–16 s longer and ~70% less time than a VLM-led baseline.

Show abstract

Current embodied models do not respond to their own failures, although what just went wrong could inform a small adjustment on the next attempt, the kind of reflection behind the gains of thinking in language models. We test whether they can repair a known error, which requires producing a correction and judging whether it is right. Stopped at a failure and allowed to retry, they seldom repair it through their own randomness or from a language description of the error, and best-of-N selection cannot pick the successful candidate after a failure. We attribute this to training only on successful demonstrations and to inputs too narrow to show what went wrong, and conclude that reflection must come from a vision-language model (VLM), which takes in far more information, such as the episode history and text, and is more general. Prior VLM-led work has the VLM plan every step and invoke the embodied model as a tool, placing the VLM on the critical path. We propose Spotter, which reverses the roles: the embodied model leads and executes continuously, while the VLM runs in parallel, monitors through a lightweight local screener, intervenes only when an error is detected, reflects on and corrects it, and returns control. We run Spotter with Qwen and with GPT as the VLM, and both improve the embodied models; with GPT, Spotter improves Cosmos Policy and $π_{0.5}$ by 5.6 and 7.5 percentage points on RoboCasa, and raises $π_{0.5}$ from 47.2% to 57.0% on the Hard setting of RoboTwin 2.0 and from 53% to 83% on a real robot. Because the VLM steps in only when an error is confirmed, a successful episode with Qwen takes only 13 to 16 s longer than with the embodied model alone and about 70% less time than with a VLM-led baseline using the same model. Our code is available at https://github.com/zqc3117/Spotter.

h=37

Simple Agentic Memory for Generalist Robot Policies

📅 2026-09-29cs.ROcs.AI👤 Ding Zhao (h=37)

Yuyou Zhang, Yunbei Zhang, Miao Li, Janet Wang, Zijian Jin

Core Contributions

Argues robot memory should store interaction-derived state (identities, progress, procedure order) rather than retained or compressed frames.
Training-free layer for frozen policies: the instruction specifies what to monitor, frozen perception tools maintain typed state, and state is retrieved only when a subgoal depends on history.
On all 16 RoboMME tasks, 67.17% mean success vs 44.51% for the best non-oracle baseline.
Ablations are mechanism-specific — removing relation, progress or route state hurts exactly the tasks that need it — strong evidence the gains come from the state design, not extra compute.

Show abstract

Visual-memory systems commonly retain or compress past observations. Robot control additionally requires interaction-derived state that no individual frame may explicitly represent, such as persistent identity relations, accumulated progress, or ordered procedures. We introduce Simple Agentic Robot Memory (SimpleARM), a training-free memory layer for frozen generalist robot policies. From the task instruction, SimpleARM specifies what to monitor; frozen perceptual tools maintain compact typed state online; structured access retrieves that state only when a proposed subgoal depends on history; and current-view grounding resolves recalled entities before execution. We evaluate SimpleARM on RoboMME, a benchmark of memory-dependent robot manipulation tasks that require history information no longer available in the current observation. Across all 16 tasks and three policy seeds, SimpleARM achieves 67.17% mean success, compared with 44.51% for the strongest non-oracle baseline. Matched ablations show mechanism specificity: removing relation, reference, progress, or route state produces large losses where the affected state is retrieved for control, while largely sparing other tasks. These results support a state-based view of robot memory: effective memory for control is not simply retained visual history, but compact task-relevant state derived from the interaction history.

#27

h=19

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning

📅 2026-09-29cs.ROcs.AIcs.MA👤 Yuqing Ma (h=19)

Ruixiao Xu, Wong Lik Hang Kenny, Zhiqian Liu, Jianing Guo, Hanxiao Li

Core Contributions

Single-agent VLA pretraining lacks coordination skills and SFT is capped by demo quality; this proposes a three-stage RL fine-tuning pipeline for multiple cooperating VLAs.
Human demos are requested only where the pretrained VLA repeatedly fails across initial configurations, and offline tuning filters per-agent (not joint) trajectories with positive advantage.
Online RL is done in the frozen VLA's latent noise space, sidestepping unstable co-exploration of full-parameter multi-agent RL.
+23.1%, +16.4% and +44% average success on RoboTwin, RoboFactory and real dual-Franka tasks with π₀/π_0.5 backbones.

Show abstract

We study reinforcement learning (RL) methods for cooperative multi-agent Vision-Language-Action (VLA) models. This problem is challenging because VLAs are pretrained on large-scale single-agent data and therefore lack the fine-grained coordination skills required for inter-robot collaboration. Supervised fine-tuning (SFT) on multi-robot demonstrations partially bridges this gap, but its performance is bounded by the demonstration data and cannot improve from its own experience. We present a three-stage reinforced fine-tuning (RFT) pipeline for multi-agent VLAs. First, initialization-aware data collection sweeps over initial configurations and invokes human demonstrations only when the pretrained VLA repeatedly fails, yielding robustness to initialization shift with reduced human cost. Second, offline credit-filtered tuning assigns credit to individual agents and fine-tunes on per-agent trajectories with positive advantage rather than on entire joint rollouts. Third, we find existing online RL for VLAs are less effective for hard multi-agent tasks, which we attribute to noisy co-exploration and unstable updates. We instead use online latent-space fine tuning, which freeze the VLA and perform RL in its latent noise space. We evaluate our multi-agent VLA with both $π_0$ and $π_{0.5}$ backbones across 11 tasks in RoboTwin, RoboFactory and real-world manipulation with two Franka robots. Our multi-agent VLA improves the average success rate by $+23.1\%$, $+16.4\%$, and $+44\%$ on RoboTwin, RoboFactory, and real-world tasks, respectively. Code available at https://anonymous.4open.science/r/mavla_rft-2BC0/.

#30

h=16

RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulation

📅 2026-09-29cs.ROcs.CV👤 Ziteng Cui (h=16)

Shuhong Liu, Heng Zhou, Lingfeng Qian, Yuhao Fang, Xianbao Hou

Core Contributions

Points out that VLAs inherit whatever a fixed camera ISP produces, and systematically measures how gain, noise, color, tone and bit depth change actions and success.
RawVLA is a streaming neural ISP that renders RAW sensor data for a frozen VLA, allocating capacity to the imaging factors that matter for behavior rather than human-pleasing images.
RawVLA-Bench makes image processing an explicit evaluation variable; the method keeps clean-condition performance while substantially improving robustness under degraded capture.

Show abstract

Vision-language-action (VLA) models typically operate on RGB images produced by a fixed camera image signal processor (ISP), leaving the imaging pipeline outside the learning and evaluation loop. We systematically examine the consequences of this overlooked design choice across five fundamental ISP dimensions: gain, sensor noise, chromatic response, tonal response, and bit depth. Our analysis reveals that RAW-to-RGB processing materially shapes both action prediction and manipulation success, with different ISP dimensions exerting substantially different effects. Guided by these findings, we introduce RawVLA, a streaming neural ISP that adaptively renders RAW observations for frozen VLA policies while concentrating its capacity on the imaging factors relevant to embodied behavior. We further present RawVLA-Bench, a RAW-domain manipulation benchmark to expose image processing as an explicit evaluation variable across clean and adverse acquisition conditions. Experiments on RawVLA-Bench show that RawVLA preserves performance under standard conditions while substantially improving robustness under degraded imaging, establishing adaptive RAW processing as an effective interface between physical cameras and embodied policies.

Benchmarks & Diagnostic Evaluation

h=106

Does Local Video Understanding Transfer Across Encounters? The EgoGears Benchmark

📅 2026-09-29cs.CVcs.AIcs.RO👤 R. Timofte (h=106)

Yuedong Tan, Lei Qi, Yu Liu, Di Wen, Ruiping Liu

Core Contributions

Unlike cross-video leaderboards that report one accuracy, EgoGears pairs single- and multi-video questions over the same 39 outdoor routes, so a drop can be attributed to cross-encounter reasoning rather than weaker local perception.
Every one of 20 comparably evaluated MLLM configurations is worse on multi-video questions, by a mean 22.5 points — and the gap survives fixing answer format and scoring.
Pins the bottleneck on observation–evidence binding and ordered route-state tracking, not simply 'more frames' — a direct warning for robots that must reuse knowledge from past visits.
Repeated traversals at different speeds and lighting (126 recordings, 531 alignment questions) make it a rare controlled testbed for place re-identification with language models.

Show abstract

Embodied systems must make knowledge acquired during one encounter usable in another despite changes in viewpoint, motion, and illumination. Yet aggregate cross-video accuracy conflates failures of local perception with failures to preserve observation identity, establish correspondence, and compose evidence, obscuring whether local video understanding actually transfers. We introduce EgoGears, a complementary single- and multi-video benchmark designed to diagnose this transition. It contains 567 single-video and 1,487 multi-video questions derived from 126 human-collected egocentric recordings covering 39 outdoor routes. Repeated traversals across movement speeds and lighting conditions ground comparisons in shared physical environments; 531 questions require alignment across independent recordings. Single-video questions measure the local visual, spatial, and motion evidence available to a model, while multi-video questions test whether evidence remains bound to the correct observation and can be composed into consistent route relationships. We report 29 single-video and 31 multi-video MLLM configurations across six model families in the main leaderboard. Among the 20 configurations evaluated comparably on both splits, every model performs worse on multi-video questions, with a mean decrease of 22.5 percentage points, and the gap persists when answer format and scoring are held fixed. The gap is not explained simply by additional videos or recording boundaries. The central bottlenecks are observation--evidence binding and ordered route-state tracking. The code and benchmark are publicly available at https://github.com/lei-qi-233/EgoGears.

h=97

LIBERO-MAX: Do Robot Policies Adapt When the World Changes?

📅 2026-09-29cs.ROcs.AI👤 P. Stone (h=97)

Yunbei Zhang, Zijian Jin, Yuanzhe Liu, Janet Wang, Xilun Zhang

Core Contributions

Most robustness benchmarks perturb the scene at reset; LIBERO-MAX perturbs it mid-execution, when the policy's history and committed actions now describe the wrong world.
Paired design (8,000 pairs, same seed, initial state and pre-event actions) isolates regressions caused by the event from failures that would have happened anyway.
Across 14 VLA, hybrid and world-action policies, a single event costs 11.0–25.7 success points; geometry and observation changes hurt everyone, while family rankings interleave — no architecture class is clearly safer.
Increasing query cadence does not close the gap, suggesting that faster replanning alone is not the fix for dynamic scenes.

Show abstract

Robots must often continue a task after a target moves, the viewpoint shifts, or an obstacle appears, even though their earlier observations and committed actions reflect the previous scene. Many simulation robustness benchmarks fix external conditions at reset, leaving this temporal challenge underexamined. We introduce LIBERO-MAX, a benchmark of 8,000 paired cases spanning eight types of changes to geometry, observations, appearance, clutter, and paths. Each pair compares task execution with and without a mid-task event, holding the task, initial state, policy seed, and pre-event action sequence fixed. This controlled comparison distinguishes event-associated regressions from failures already present without the change. Across fourteen current VLA, hybrid, and world-action policies, events reduce success by 11.0-25.7 percentage points. Event profiles reveal shared vulnerabilities to geometry and observation changes, while policy-family rankings interleave. Camera controls show that robustness reflects both competence under the changed conditions and the trajectory from which they are encountered; varying query cadence does not eliminate the gap. Together, the paired protocol and temporal diagnostics establish LIBERO-MAX as a reproducible testbed for diagnosing failures under mid-execution changes and measuring progress toward robot policies that remain effective as the world changes.

#19

h=23

RoboChrono: A Real Robot Benchmark for Streaming Task Understanding

📅 2026-09-29cs.RO👤 Siteng Huang (h=23)

Yuzhou Wu, Longteng Fan, Zimeng Li, Yu Wanchan, Ting Zhang

Core Contributions

A streaming benchmark built from real robot runs plus bare-hand human recordings: 39 scenarios, 34,713 instances, seven tasks spanning recognition, alignment and temporal grounding.
Exposes a split between matching and ordering: GPT-6-Astra scores 98.3% on frame matching but 68.3% on ordering; RynnBrain1.1 drops from 95.4% to 32.9%.
Input ablations show next-action prediction barely changes without visual input (−0.7 pts) — models lean on task priors, so high anticipation scores may not reflect perception.

Show abstract

Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress. We introduce RoboChrono, a benchmark for streaming task understanding comprising 39 scenarios and 34,713 evaluation instances, constructed from real robot executions and complementary bare-hand human recordings. The benchmark evaluates seven tasks grouped into recognition, alignment, and temporal grounding, covering action understanding and anticipation, visual correspondence, temporal ordering, and action localization. Zero-shot evaluation of 18 vision-language models reveals substantial differences across tasks. GPT-6-Astra achieves 98.3% accuracy on Frame Matching but 68.3% on Frame Ordering, while RynnBrain1.1-122B-A10B exhibits a larger gap, reaching 95.4% and 32.9%, respectively. Input ablations on matched questions with five open-weight models further reveal distinct dependencies on visual evidence: removing visual observations reduces Current Action Recognition accuracy by 22.1 percentage points, whereas Next Action Prediction decreases by only 0.7 points. These findings show that strong visual matching does not consistently coincide with strong temporal ordering, and suggest that next-action prediction can be supported by task and action priors even when visual evidence is unavailable. RoboChrono provides a diagnostic setting for examining these differences, highlighting the need for capability-specific evaluation beyond aggregate scores when assessing task understanding in robot manipulation.

#23

h=21

Why Cross-Skeleton Retargeting Is Non-Identifiable: Structural Limits of Generative Motion Models

📅 2026-09-29cs.CVcs.RO👤 Joni Pajarinen (h=21)

Zhiyuan Li, Wenyan Yang, Pekka Marttinen, Joni Pajarinen

Core Contributions

Proves that source-conditioned cross-skeleton retargeting is structurally non-identifiable under standard generative objectives in sparse heterogeneous motion data.
Names two failure modes: gauge non-identifiability (unpaired matching lets skeleton latents rotate freely) and conditional-mean degeneration (action-level pairing collapses to an average motion that ignores the source).
Introduces Source-Instance Fidelity, which tests whether outputs vary the way their sources do; many methods passing action-level tests sit at the source-blind floor.
Relevant to humanoid learning from human/animal motion: 'correct action' does not mean the source motion was actually transferred.

Show abstract

Cross-skeleton motion generation trains generative models to carry action structure and motion intention from one body to another. Yet a target motion that shows the right action has two explanations that the training data cannot tell apart: the model transferred the source clip, or it recovered a typical motion for the requested action. We show that this ambiguity is structural rather than incidental: under standard generative objectives, the source-conditioned retargeting map is non-identifiable in sparse heterogeneous motion domains. Unpaired distribution matching yields gauge non-identifiability: the latent spaces of different skeletons can be transformed relative to one another without changing the training evidence, so different source-conditioned maps fit it equally well. Sparse paired supervision admits the complementary failure mode, \emph{conditional-mean degeneration}: when clips are paired only by action, squared-error training converges to an average target motion that ignores the source clip. To make the missing evidence observable, we introduce Source-Instance Fidelity (SIF), a diagnostic that tests whether outputs differ from one another the way their source clips do, with the target skeleton and action held fixed. Under this diagnostic, methods that succeed at the standard action-level test on animal motion data often sit at the source-blind floor, while the methods that rise above it retain only a partial relational signal. Retargeting therefore needs objectives and evaluations that can identify the source-conditioned map it claims to learn. Project page: https://cross-skeleton-retargeting.netlify.app/.

Humanoid, Legged & Bio-inspired Control

h=62

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

📅 2026-09-29cs.ROcs.CVcs.GR👤 Angjoo Kanazawa (h=62)

Zihan Wang, Zhen Wu, Pieter Abbeel, Rocky Duan, Jitendra Malik

Core Contributions

Attacks the data bottleneck for visual humanoid imitation by generating hundreds of 'counterfactual' human–object videos from a handful of real exemplars with video-to-video models.
The key trick is a contact-anchored real-to-sim reconstruction that tolerates generative artifacts, turning imperfect synthetic clips into physically plausible whole-body trajectories.
Intra-class variation from generated videos lets one policy generalize to unseen boxes, barrels, bins and balls — deployed on a real humanoid with only onboard depth and zero real-world fine-tuning.
Shows video generators can serve as a data multiplier for loco-manipulation when paired with a physics filter, rather than as world models for direct control.

Show abstract

Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach. We propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set. PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. Our contact-anchored real-to-sim pipeline then reconstructs both human and object motions, retargeting this imperfect video data into physically plausible trajectories. The intra-class variability across these counterfactual videos lets us train a single policy that generalizes to unseen objects within each category. We demonstrate the full pipeline by deploying this policy on a real robot without any real-world fine-tuning. Using only onboard depth observations, our humanoid picks up, carries, and drops objects, including boxes, barrels, bins, and balls, across novel instances, sizes, and initial configurations.

h=33

EgoAlign: Bridging the Human-Humanoid Gap for Long-Range Loco-Manipulation

📅 2026-09-29cs.RO👤 Ai-Min Hao (h=33)

Yiming Jiang, Chen Jin, Chongyang Xu, Yilun Chen, Aimin Hao

Core Contributions

Egocentric human demos lack robot states and mismatch body scale; EgoAlign converts them into action/state labels for a real whole-body controller without any robot teleoperation.
Runs the target robot simulator in the loop during collection, preserving locomotion references while adapting upper-body geometry through scale alignment and controller-in-the-loop refinement.
Causal replay reconstructs robot states and motion tokens, so a VLA can be fine-tuned purely on adapted human data and deployed zero-shot on a humanoid for long-range object relocation.
Refinement beats kinematic-only alignment on real pickup success, and human collection is faster on-site than teleoperation.

Show abstract

Egocentric human demonstrations offer an accessible source of task experience, but differences in body scale and controller response, together with missing robot states, limit their value as humanoid training supervision. We present EgoAlign, a data-construction framework that converts these demonstrations into action and state supervision compatible with a general-purpose, continuous whole-body controller, without collecting physical-robot demonstrations. Using the target-robot model and simulator, EgoAlign guides demonstration collection through execution feedback. It preserves locomotion references for visually guided periodic stepping while adapting upper-body interaction geometry through scale alignment and controller-in-the-loop refinement. A final causal replay reconstructs the corresponding robot states and motion-token labels for training with the human observations. We assess the resulting supervision by fine-tuning a vision--language--action model solely on adapted human demonstrations and deploying it zero-shot on a physical humanoid. The resulting policies perform long-range object relocation, navigation to unseen goal positions, and independently evaluated foot interaction. Refinement improves simulated hand alignment and physical pickup success over kinematic alignment alone, while human collection reduces on-site acquisition time relative to teleoperation. https://lambdahumanoid.github.io/EgoAlign/

#29

h=17

CrossBFM: Distilling a Shared Latent Behavior Space Across Humanoid Embodiments

📅 2026-09-29cs.RO👤 V. Ngo (h=17)

Tan-Dzung Do, Tuan Dat Phuong, Nico Bohlinger, Cuc T. Trinh, Siwei Ju

Core Contributions

Forward–Backward behavior foundation models cost hundreds of GPU-hours per robot and produce unrelated latent spaces per embodiment; CrossBFM treats one shared latent space as the transferable asset.
A robot-agnostic encoder distills the behavior space across all humanoids in under one GPU-hour, then latent-conditioned trackers are trained with PPO in ~10 GPU-hours.
Tracking loses only 0.025 rad vs joint-conditioned control, all 41 reward prompts work, and an unseen humanoid recovers up to 89% of seen-robot tracking performance.
Verified on real robots for tracking, goal reaching and reward prompting, including flow-generated latents.

Show abstract

Behavior Foundation Models (BFMs) give humanoids a promptable policy over a latent behavior space, enabling one single vector to represent a motion to imitate, a pose to reach, or a reward to maximize. Forward-Backward representations successfully produce such spaces, but at the cost of hundreds of GPU-hours for a single robot. Moreover, when the training process is repeated for a second robot, it produces a second space unrelated to the first, resulting in embodiment-specific latents that do not unify or transfer. We address these problems with CrossBFM, treating the latent space as the transferable asset for various embodiments. As retargeting provides frame-level cross-embodiment correspondence, we propose a unified encoder architecture with no robot-specific parameters for distilling the behavior space to address all training embodiments simultaneously in less than a GPU-hour. Following this encoder, latent-conditioned trackers turn the distilled latent into whole-body control in a conventional PPO training manner in just 10 more GPU-hours. On three distilled humanoids, all three prompting modes transfer: motion tracking with latent-conditioned policy losing only $0.025$ rad to its joint-conditioned counterpart, smooth goal reaching between poses with no falls, and reward optimization for all $41$ reward prompts. Our experiments further reveal that 1) regressing the encoder on a quarter of the motion corpus costs only $5\%$ of tracking performance and 2) training the encoder on a subset of robots and evaluating on an unseen one recovers up to $89\%$ of the tracking performance of seen robots, demonstrating cross-embodiment generalization to morphologically similar robots. We also verify the pipeline on real robots across all three prompting modes and with flow-based generated latents. Project website: https://dotandung.github.io/crossbfm/

#17

h=24

Predictive Safety Curricula for Robust Legged Locomotion

📅 2026-09-29cs.ROcs.AIcs.LG👤 C. Gehring (h=24)

Ivan Ovinnikov, Pascal Sutter, Christian Gehring, Jordis Herrmann

Core Contributions

Standard locomotion curricula tune difficulty, not exposure to rare safety-critical events; PSC reallocates training experience using a learned distributional safety critic.
The critic prioritizes both terrain contexts and replayed randomization events, leaving reward and loss untouched — easy to drop into existing stacks.
On ANYmal-D hardware, shank collisions fall 63% vs a learning-progress curriculum (lower in every seed); on a production stair-climbing platform, observed shank collisions are eliminated.
Notable for transferring to two production locomotion pipelines, not just a research sim.

Show abstract

Rare but consequential failures can persist in learned locomotion policies for legged robots even when average task performance is high, in part because standard curricula primarily adapt task difficulty rather than the distribution of safety-critical experience. We introduce Predictive Safety Curricula (PSC), a framework for allocating locomotion training experience using learned predictions of future safety cost. PSC trains a distributional safety critic from policy rollouts and uses its predictions to prioritize both terrain contexts and previously encountered randomized events. The resulting curriculum modifies the training distribution while leaving the task reward and policy-optimization loss unchanged. We evaluate PSC in controlled rough-terrain locomotion and in production locomotion systems. PSC improves reliability relative to standard terrain progression, advantage-based replay, and learning-progress curricula, with the largest gains on difficult terrain and under degraded observations. The same allocation principle transfers to two production locomotion stacks. On ANYmal-D hardware, PSC reduces shank-collision incidence by $63\%$ relative to the learning-progress curriculum across three matched training seeds, with a reduction in every seed. On a production stair-climbing platform, PSC eliminates observed shank collisions in the evaluated hardware trials. These results show that learned predictions of future safety cost can provide an effective signal for allocating training experience toward rare failure modes and improving locomotion reliability.

#13

h=28

All You Need Is Low Fidelity: Zero-Shot Sim-to-Real of Learned Robotic Fish Control

📅 2026-09-29cs.RO👤 Robert K. Katzschmann (h=28)

Liam Maloney, Simon Ramchandani, Mike Y. Michelis, Ronan Hinchet, Robert K. Katzschmann

Core Contributions

Challenges the assumption that sim-to-real for soft underwater robots needs high-fidelity CFD: a stateless, quasi-steady fluid model with no wake or added-mass memory suffices.
Transfer works because the policy acts through a band-limited rhythmic generator that cannot leave the frequency band where the fluid model was identified — the constraint, not fidelity, does the work.
A single policy on a single-motor tendon-driven fish performs closed-loop target reaching, disturbance rejection and out-of-distribution tracking in an outdoor pool, unchanged.
Raises a broader design question: how much physics can be moved from the simulator into controller structure.

Show abstract

Complex tasks for underwater robots remain limited by the capabilities of their controllers. Learning a better one for a soft, underactuated robotic fish trades simulator cost against fidelity. We show that an intentionally low-fidelity simulator is enough: a stateless, quasi-steady fluid model with no wake and no added-mass history suffices to learn a \emph{general}, closed-loop controller that transfers to hardware without tuning. Our platform is a soft, single-motor, tendon-driven fish whose policy observes only what the hardware can measure. A staged pipeline grounds the simulator in two independent identifications, fixing the tail dynamics and a stateless fluid model; the policy then acts through a band-limited rhythmic trajectory generator rather than commanding the tail directly. Deployed unchanged in an outdoor pool, a single policy performs closed-loop target reaching, disturbance rejection, and out-of-distribution target acquisition and tracking. The transfer rests on the constraint rather than the fidelity: the generator cannot leave the band over which the fluid was identified. This raises the question of how much of the physics can reside in the controller rather than in the simulator.

Learning from Videos, Demos & Specifications

#10

h=32

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

📅 2026-09-29cs.ROcs.AI👤 Jyotirmoy V. Deshmukh (h=32)

Merve Atasever, Keyan Azbijari, Cagan Bakirci, Bo-Ruei Huang, Tolga Izdas

Core Contributions

Rather than scalar video-similarity rewards or LLM-written reward code, Video2STL extracts parametric Signal Temporal Logic specs from observation-only videos — inspectable, reusable task structure.
A VLM decides the temporal structure; numeric thresholds and time bounds are grounded from successful robot rollouts, separating 'what' from 'how much'.
Short-horizon specs give dense robustness rewards; a causal monitor on the long-horizon spec gives one-shot progress rewards — addressing the credit problem of long tasks.
85.8%/67.0% success-once/at-end vs 65.0%/42.3% for Text2Reward on manipulation, and 100% success for quadruped locomotion across 0.3–2.1 m/s — including transfer from human and animal videos.

Show abstract

Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves $85.8\%$ average success-once and $67.0\%$ success-at-end, compared with $81.5\%/59.5\%$ for native dense PPO and $65.0\%/42.3\%$ for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve $100\%$ success across velocities from $0.3$ to $2.1\,\mathrm{m/s}$ while remaining competitive in high-speed energy efficiency. Project webpage: \href{https://video2stl.github.io/}{video2stl}.

#14

h=27

VidAct: Learning Manipulation from In-the-Wild Videos with Object-Centric 3D Awareness

📅 2026-09-29cs.RO👤 Zhen-Shan Bing (h=27)

Hang Li, Mingxin Zhang, Zihan Wu, Yang Tian, Dong Chen

Core Contributions

Learns manipulation from one monocular video per task, avoiding hand retargeting by canonicalizing reconstructed object motion in the static object frame — works with any camera viewpoint.
Residual trajectory transfer adapts the demo motion to new object configurations without distorting its shape, a known failure of naive warping.
Adds an auxiliary task predicting complete-object point clouds (privileged in sim) while deploying RGB-only, giving dense 3D shape and pose supervision that improves sim-to-real.
Works on human, robot, generated and internet videos with zero-shot real deployment.

Show abstract

Video demonstrations offer a scalable alternative to costly robot data for learning manipulation, yet existing reconstruction-based approaches often rely on constrained camera viewpoints or human-to-robot retargeting, while the reconstructed trajectories are difficult to adapt to new objects configurations without distorting the trajectory shape. Another key limitation is that the resulting policies often lack precise object-level 3D geometry awareness, limiting object grounding and object shape awareness critical for precise manipulation. To bridge these gaps, we propose VidAct, an efficient video-to-robot framework that learns object-centric, 3D-aware manipulation policies from a single monocular video per task and enables zero-shot real-world deployment. VidAct consists of three key components. First, VidAct reconstructs object meshes and motion from arbitrary demo videos and canonicalizes the motion in the static object frame, avoiding embodiment-specific retargeting and accommodating diverse camera viewpoints. Second, VidAct employ residual trajectory transfer for adapting the reconstructed motion to novel object configurations while preserving its motion shape. Finally, as the key policy-learning component, VidAct predicts simulation-provided privileged complete-object point clouds at each frame as an auxiliary task while retaining RGB-only deployment, providing dense object-centric supervision over both object pose and 3D geometry. Experiments on human, robot, generated, and internet videos demonstrate broad video applicability and zero-shot deployment. Per-frame complete-object 3D supervision improves policy generalization and sim-to-real success, while residual trajectory transfer enables reliable trajectory adaptation with better shape preservation.

#12

h=28

BlenDAgger: Blended Shared Control for Interactive Imitation Learning

📅 2026-09-29cs.ROcs.LG👤 Henny Admoni (h=28)

Cailyn Smith, Geoffrey Sun, Henny Admoni, Zackory Erickson

Core Contributions

HG-DAgger hands full control to the human during interventions; BlenDAgger instead blends human and policy actions via shared control, so corrections stay close to the policy's own distribution.
This yields 57% smoother policy↔human transitions and 14% higher similarity to training trajectories — cleaner data for the next DAgger round.
30+ percentage points higher autonomous success than HG-DAgger on two real-world tasks, and faster data collection in a 14-person user study (BF=13.32) with no drop in perceived workload.

Show abstract

Robot policies are frequently trained from human corrections, yet teleoperating a robot to provide corrections is burdensome, and human demonstrators are not always optimal. We propose Blended DAgger (BlenDAgger), an approach for collecting data to train imitation learning policies by using shared control to blend the policy's and demonstrator's actions during interventions. By blending human and policy actions, we aim to improve the autonomous performance of manipulation policies. We validate our approach across five manipulation tasks, two in the real world and three in simulation. Our approach achieves higher autonomous performance by 30 or more percentage points on two real-world tasks compared to a typical human-gated correction approach (HG-DAgger). We also investigate the advantages of BlenDAgger that allow for higher autonomous performance, finding that BlenDAgger results in 57% smoother transitions between policy control and human interventions, and 14% higher trajectory similarity to the training data. In a user study (n=14) on two real-world tasks, we find that BlenDAgger results in faster data collection (BF=13.32), and we do not find a difference in subjective perceptions. These results show that blended shared control leads to higher autonomous performance compared to typical methods for fine-tuning robot policies from fully teleoperated interventions.

#15

h=26

DROM: A Language-Guided Diffusion Framework for Multi-Skill Robotic Manipulation

📅 2026-09-29cs.RO👤 L. Roveda (h=26)

Vincenzo Pomponi, Rocco Felici, Paolo Franceschi, Stefano Baraldo, Oliver Avram

Core Contributions

Uses Dynamic Movement Primitives to augment a handful of demos into large multi-skill datasets, extending spatial generalization beyond the demonstrated workspace.
Extends Motion Planning Diffusion with language cross-attention so one generative model produces many skills, including orientation-sensitive motions that are hard to hand-engineer.
An LLM decomposes operator requests into skill sequences; validated on Franka Panda, FANUC CRX-25iA and MuJoCo, beating MPD and behavior-cloning baselines on long-horizon tasks.

Show abstract

Learning robust manipulation policies for diverse, long-horizon tasks from limited demonstrations remains a fundamental challenge in robotics. We present DROM, a language-guided diffusion framework that enables robots to learn, represent, and compose multiple manipulation skills within a single generative policy. DROM leverages Dynamic Movement Primitives (DMPs) to augment a small set of expert demonstrations into expressive multi-skill datasets, substantially reducing data collection while improving spatial generalization beyond the demonstrated workspace. Building upon Motion Planning Diffusion (MPD), we extend the diffusion architecture to support language-conditioned multi-skill trajectory generation through cross-attention, allowing a single model to generate skill-consistent motions for a diverse set of manipulation primitives, including orientation-sensitive behaviors that are difficult to design using conventional motion planning or hard-coded controllers. For long-horizon manipulation, a large language model decomposes high-level operator requests into executable sequences of skills, enabling natural language interaction and autonomous task execution. We validate DROM on a Franka Emika Panda robot, a FANUC CRX25ia robot, and in MuJoCo simulation across a wide range of manipulation tasks. Experimental results demonstrate that DROM outperforms Motion Planning Diffusion and Behavior Cloning baselines, achieves robust multi-skill generalization, and composes learned skills to reliably execute long-horizon manipulation tasks from natural language instructions using only a limited number of human demonstrations. Datasets, simulation environments, and more at https://github.com/automation-robotics-machines/drom.

Contact-Rich & Physics-Aware Manipulation

h=54

FORM: Robot Manipulation through Direct Material Law Identification

📅 2026-09-29cs.RO👤 U. Topcu (h=54)

Stepan Tretiakov, Ruihan Zhao, Cheng-Hsi Hsiao, Xingjian Li, Adam Thorpe

Core Contributions

Instead of iterative differentiable-simulation fitting, FORM writes weak-form momentum balance as equations that are linear in unknown material parameters, reducing identification to least-squares.
Uses the same MPM discretization as the forward simulator, so identified parameters plug straight into planning with no refit or conversion.
Identification drops from ~10–25 min to 2–5 s from a single interaction, with elastic properties within 3.4% and elastoplastic within 2%.
One interaction suffices for new tasks on hardware: elastic rod insertion, golf putting with a flexible club, dough shaping (72–78% IoU) and pouring with 3.8 mL mean error.

Show abstract

When interacting with an unfamiliar deformable material, a robot lacks prior knowledge of its physical properties and how it will respond to applied forces and motion. Rapid online identification is therefore essential for reliable manipulation. We present FORM (From Observed Response to Material laws), which identifies material properties from a single robot interaction and reuses the recovered model to plan manipulation under new actions and geometries. We use weak-form momentum balance to convert observed material motion and contact forces into linear equations in the unknown material parameters. These equations are assembled using the same material point method discretization as the forward simulator, so identification reduces to linear least-squares solves whose solutions can be used directly for prediction without refitting or conversion. Across four material classes, FORM reduces identification time from roughly 10--25 minutes for iterative baselines to 2--5 seconds, while maintaining competitive accuracy on new motions, initial conditions, and geometries. We demonstrate our approach in simulation and on hardware across four manipulation tasks: elastic rod insertion, golf putting with an elastic club, elastoplastic shaping, and target-volume pouring. In each task, the model identified from a single interaction is reused to plan new motions or manipulate a different geometry. FORM estimates elastic properties within 3.4% and elastoplastic properties within 2%, achieves 72.4--77.8% IoU in dough shaping, and keeps mean pouring error at 3.8 mL across target volumes of 60--160 mL.

#16

h=26

Learning to Explore Hidden Kinematics for Articulated Object Manipulation

📅 2026-09-29cs.RO👤 Kostas Daniilidis (h=26)

Ruiyao Liu, Boshu Lei, Zhuoyang Pan, Kostas Daniilidis

Core Contributions

Active perception for articulated objects usually picks the greedy most-informative action each step, which cannot plan over horizons without unknown contact dynamics; this work amortizes exploration into an RL policy.
The Bayesian belief over joint type/parameters is rendered as a per-point articulation flow field — an inductive bias that generalizes better than latent belief codes or observed flow.
Reward is entropy reduction of the posterior, so informative probing becomes learned behavior.
61.7% vs 44.4% on the new ArticuRiddle set, where appearance deliberately implies the wrong articulation, plus gains on PartManip doors and drawers.

Show abstract

The kinematics of an articulated object is often ambiguous from vision alone. Interaction resolves the ambiguity, and active perception methods exploit this by searching for the single action that most sharpens a belief over the kinematic parameters at each step. Such greedy search cannot be extended over a horizon without forward models of the contact and inertial dynamics, which are themselves unknown. We instead amortize action selection into training. We maintain a belief distribution over joint type and parameters, initialized from a generative prior and updated by Bayesian filtering on the observed part motion. To condition the policy on this belief, we render it as a per-point articulation flow field, the motion that the current posterior predicts for every point on the object. Carrying the inductive bias of articulated motion, this representation generalizes better than a latent encoding of the belief or flow tracked from observation. We train the policy with reinforcement learning, rewarding the entropy that each interaction removes from the posterior, so that informative exploration becomes learned behavior rather than a search at every step. Our method outperforms previous approaches across door and drawer manipulation on the PartManip benchmark, and reaches 61.7% success on ArticuRiddle, a new dataset of objects whose appearance implies the wrong articulation, against 44.4% for the best previous method. Project Website: https://hiddenkinematics.github.io/

#24

h=21

HACo: Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation

📅 2026-09-29cs.RO👤 Peng-Wei Wang (h=21)

Naisheng Ye, Yinzhe Zhou, Junkai Zhao, Yuhang Lu, Checheng Yu

Core Contributions

Combines fingertip tactile sensing with joint-torque feedback, capturing contacts beyond tactile coverage that tactile-only policies miss.
Compliance-regulated teleoperation produces action labels that preserve intent while limiting loads, and the command–state discrepancy is used as auxiliary 'compliant intent' supervision.
A gated haptic cross-attention module grounds action generation in evolving contact state, giving closed-loop force regulation without explicit contact models.
83% mean success vs 35% for the best baseline across friction, fragile curved-surface, torque and deformable tasks (20 trials each).

Show abstract

Contact-rich dexterous manipulation requires policies that translate physical feedback into motion commands while regulating interaction loads across evolving multi-contact interactions. This requires haptic observations of contact state and action supervision showing how commands should adapt. Existing policies often overlook complementary fingertip tactile and joint-torque feedback, while common action targets either encode excessive loading or omit motion constrained by the object. We introduce HACo, a Haptic Active Compliance policy that learns force-regulating actions directly from haptic feedback. Compliance-regulated teleoperation converts operator inputs into controller-executable compliant actions that preserve motion intent while regulating loads. HACo learns these actions directly, using command-state discrepancy as auxiliary compliant-intent supervision. It combines local fingertip tactile responses with joint-torque feedback capturing load transmission through the articulated hand, including contacts beyond tactile coverage. A Compliance Grounding Module uses gated haptic cross-attention to ground action generation in the evolving haptic state, enabling closed-loop force regulation without explicit online contact modeling. We evaluate HACo on a real-world benchmark covering multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation. Across 20 trials per task, HACo achieves an 83% mean success rate, compared with 35% for the strongest evaluated baseline. These results demonstrate active compliance across diverse force-sensitive dexterous manipulation tasks.

Perception, SLAM & Field Robotics

h=56

When to Adapt: Multi-Signal Domain Shift Detection for Efficient Training-Free Adaptation in Open-Vocabulary Segmentation

📅 2026-09-29cs.ROcs.CV👤 P. Jensfelt (h=56)

Michele Antonazzi, Alejandra C. Hernandez, José Araujo, Olov Andersson, Patric Jensfelt

Core Contributions

Training-free test-time adaptation for open-vocabulary segmentation typically adapts every frame — too expensive on robot hardware; this work asks when to adapt instead.
Combines three complementary shift signals — visual change, adapter mismatch and semantic drift — exploiting temporal coherence between consecutive frames to trigger adaptation only when needed.
Keeps segmentation accuracy while substantially cutting adaptation calls on indoor/outdoor real-robot data, making continual adaptation feasible for long-duration missions.

Show abstract

Robust and reliable perception is essential for autonomous robots operating in real-world environments, particularly in long-term missions where environmental conditions may change significantly over time. Although recent advances in Visual Foundation Models (VFMs) have improved open-vocabulary semantic segmentation, these models can still suffer from domain shift, which can significantly degrade performance if they are not adapted to the current environment. Training-free domain adaptation is a relevant paradigm for adaptation, consisting of adjusting the model online using lightweight adapters. Recent approaches apply this on a per-frame basis, which is impractical for deployments on resource-constrained robotic hardware. To tackle this, we propose a multi-signal domain shift detection method for training-free continual test-time adaptation (TF-CTTA) in open-vocabulary segmentation. Our method leverages temporal coherence across consecutive frames by monitoring and combining complementary aspects of domain shift (visual change, adapter mismatch, and semantic drift) to trigger adaptation only when needed. We validate our approach on a benchmark including indoor and outdoor environments and using real robotic data. We demonstrate that our approach maintains segmentation accuracy while substantially reducing adaptations, making training-free adaptation practical and feasible for long-term, real-world robotic deployments.

h=44

Towards Spatial Perception for Heterogeneous Robot Collaboration in Subterranean Mining Environments

📅 2026-09-29cs.RO👤 G. Nikolakopoulos (h=44)

Mario Alberto Valdes Saucedo, Akash Patel, Christoforos Kanellakis, George Nikolakopoulos

Core Contributions

Frames deep-mine exploitation as a heterogeneous handoff: a light Explorer maps and builds a 3D scene graph; a heavily instrumented Inspector plans close-range viewpoints from it.
Detects mineral deposits zero-shot from natural-language prompts with a vision-language segmentation stack — no mine-specific labeled data.
The contribution is the geometric abstraction chain (per-view boxes → cross-view merging → plane fitting → polygons) that turns noisy detections into inspectable targets.
Validated in an active magnesite mine and a test facility under dust and poor lighting — rare field evidence for open-vocabulary perception underground.

Show abstract

The autonomous extraction of deep mineral deposits in abandoned underground mines is fundamentally a multi-agent integration problem. No single platform simultaneously offers the mobility to traverse kilometers of degraded drifts and the sensing payload required to characterize an ore body. This article presents the onboard perception pipeline that bridges two heterogeneous agents within the PERSEPHONE autonomous mining mission. Which consist of a lightweight Explorer robot that maps an unknown mine and generates a 3D scene graph of inspection targets, by running a zero-shot, vision-language semantic segmentation stack that detects mineral deposits directly from natural-language prompts. The map and the graph are then handed to a second Inspector robot, which carries an advanced sensing payload and uses them to plan close-range inspection viewpoints. We detail the complete pipeline, with emphasis on the geometric abstraction that turns raw detections into actionable inspection targets, spanning per-view bounding-box generation, cross-view box merging, plane fitting, and polygon extraction, and we report an extensive field validation in a subterranean test facility and in an active magnesite mine, covering both iron-vein and magnesite mineralization under realistic, perceptually degraded conditions.

#11

h=32

GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion

📅 2026-09-29cs.CVcs.RO👤 K. Krishna (h=32)

Suhani Grover, Astik Srivastava, Viswas Dinesh, Avinash Sharma, K. Madhava Krishna

Core Contributions

Glass is a blind spot for RGB and depth alike; GlassFormer adds mmWave radar, which reflects strongly exactly where optical sensors fail.
Exploits cross-modal inconsistency (radar hit, no depth) to form a spatial prior injected into a lightweight transformer via cross-modal attention.
0.88 mIoU on mixed conditions and 0.59 on a dedicated low-light split, while running in real time on constrained platforms — a practical safety fix for indoor navigation.

Show abstract

Transparent surfaces are ubiquitous in built environments, yet they remain a persistent failure case for robotic perception. RGB cameras perceive the background behind glass rather than the surface itself, while depth sensors such as LiDAR, time-of-flight, and RGB-D often return invalid or background measurements in transparent regions. As a result, systems that rely solely on optical sensing may misinterpret glass walls, doors, or mirrors as free space, compromising safe and reliable navigation. Existing glass segmentation approaches address this by learning visual cues such as reflections, boundaries, and semantic context from RGB images. While effective under favourable lighting and viewing conditions, these cues degrade in low-light environments, under glare, or when glass surfaces are featureless or partially occluded. In this work, we propose a multimodal framework that fuses millimetre-wave radar with RGB-D sensing for real-time transparent surface segmentation. Radar reflects strongly off glass surfaces, providing a geometric cue that remains reliable precisely where vision and depth fail. We exploit this cross-modal inconsistency to generate a radar-guided spatial prior, which is integrated into a lightweight transformer-based segmentation network, GlassFormer, via cross-modal attention. We report results on a mixed-condition test split covering all scene types and a dedicated low-light split designed to stress vision-only methods. GlassFormer achieves 0.88 mIoU on the mixed split, and 0.59 mIoU on the low light split, demonstrating substantial robustness gains over vision-only baselines while maintaining real-time performance on resource-constrained platforms.

#18

h=23

Pow3R-SLAM: Real-Time RGB-D SLAM with 3D Reconstruction Priors

📅 2026-09-29cs.ROcs.CV👤 Sajad Saeedi (h=23)

Christopher Kolios, Ishaan Mehta, Sasa Janjic, Yeganeh Bahoo, Sajad Saeedi

Core Contributions

Extends MASt3R-SLAM to RGB-D by feeding depth as a prior to the Pow3R two-view network rather than fusing it geometrically, so sparse or holey depth still yields well-conditioned pointmaps.
Against MASt3R-SLAM on 24 sequences: 1.6× faster, 15% lower trajectory error and 30% lower Chamfer distance; a hybrid variant runs at 25.3 FPS (2.1× faster).
More accurate than ORB-SLAM3 RGB-D on TUM, 7-Scenes and ETH3D and completes every TUM sequence, though it still struggles in self-similar scenes.

Show abstract

We present Pow3R-SLAM, a real-time RGB-D simultaneous localization and mapping (SLAM) system that uses Pow3R for tracking and mapping. Inspired by MASt3R-SLAM, a recent work on monocular SLAM using two-view 3D reconstruction priors, we extend the work to incorporate depth as a prior on the network's prediction, rather than as geometry to fuse. Where traditional RGB-D SLAM systems struggle with sparsity in the depth images, Pow3R utilizes the available depth to give a better-conditioned pointmap, while inferring the depths in empty regions from the two-view photometric, depth, and intrinsic data. Evaluated against MASt3R-SLAM following its protocol on 24 sequences from TUM, 7-Scenes, and Replica, Pow3R-SLAM runs 1.6x faster in wall time, has 15% lower mean trajectory error, a 3.1x lower unscaled error, and produces denser maps, with a 30% lower Chamfer distance. We also introduce a hybrid variant that runs 2.1x faster than MASt3R-SLAM at 25.3 frames per second (FPS), while maintaining improved tracking and mapping accuracy. Against ORB-SLAM3 in RGB-D mode, Pow3R-SLAM is more accurate on TUM, 7-Scenes, and ETH3D-SLAM, and completes every TUM sequence. While Pow3R-SLAM can struggle on a small set of self-similar scenes, its overall performance shows that adding depth as a prior for two-view 3D reconstruction SLAM can be beneficial. A project webpage is available at: https://ChrisKolios.github.io/Pow3R-SLAM , and code will be made open-source upon acceptance.

Autonomous Driving & Multi-Agent Planning

h=62

V2X-WAM: A Cooperative World Action Model for End-to-End Autonomous Driving

📅 2026-09-29cs.ROcs.CV👤 B. Ran (h=62)

Junwei You, Weizhe Tang, Can Wang, Yan Zhao, Jun Hua

Core Contributions

Prior vehicle–infrastructure methods use roadside data only to improve the current scene representation; V2X-WAM also uses it to predict the consequences of candidate plans.
Candidate trajectories condition future occupancy and flow predictions, which are fed back to refine the plan — closing an action↔world loop for cooperative driving.
Reliability-aware fusion plus a compact quantized infrastructure message reduces communication overhead substantially while improving planning accuracy and safety on a large real-world V2X dataset.

Show abstract

Vehicle-infrastructure cooperation can complement onboard sensing with broader and more informative observations of the traffic environment, providing valuable support for end-to-end autonomous driving. However, existing cooperative driving methods mainly exploit roadside information to enhance the representation of the current scene, while the future consequences of prospective driving actions are rarely modeled explicitly. This limits the ability of the planner to anticipate how its decisions may interact with the evolving traffic environment. To address this issue, we propose V2X-WAM, a cooperative world action model that tightly couples cooperative scene understanding, action generation, and future-world reasoning. V2X-WAM constructs a reliability-aware spatiotemporal representation from vehicle- and infrastructure-side observations, while compressing infrastructure information into a compact quantized message for efficient communication. Based on the resulting cooperative representation, a multimodal planner generates prospective trajectories, which explicitly condition future occupancy and dynamic-flow prediction. The predicted world consequences are then fed back to refine the planned trajectory, forming a closed interaction between action and future-world evolution. Experiments on a large-scale real-world cooperative driving dataset demonstrate that V2X-WAM consistently improves planning accuracy and safety over representative end-to-end cooperative driving methods, while achieving stronger future-world prediction and substantially lower communication overhead. Ablation studies further validate the effectiveness of the proposed design.

#22

h=21

PhysWAM: Physically Consistent World Action Model for Autonomous Driving

📅 2026-09-29cs.ROcs.CV👤 R. McAllister (h=21)

Dhruv Parikh, Fengcheng Yu, Quankai Gao, Jiawei Yang, Junjie Ye

Core Contributions

Joint generation of future video and actions does not by itself enforce geometric agreement; PhysWAM co-denoises multi-view video, metric depth and ego-motion in one flow-matching transformer.
Coupled Point Projection unprojects generated depth, moves it with generated SE(3) ego-motion, and matches it against LiDAR — tying world and action predictions to measured geometry.
Trajectory selection needs only a label-free consensus rule (no learned scorer or simulator), yet planning is strong on NAVSIM v1/v2 and transfers zero-shot closed-loop.
CPP improves both planning and depth, supporting the claim that geometry is the right coupling between world and action.

Show abstract

World-action models (WAMs) jointly predict how a scene will evolve and how an agent should act, however joint generation alone does not necessarily impose a shared geometric constraint on these predictions. We present PhysWAM, a unified world-action model for autonomous driving that co-denoises multiview video, metric depth, and ego motion within a single flow-matching transformer. To ground world and action generation in measured scene geometry, we introduce Coupled Point Projection (CPP) that unprojects the generated depth into 3D points, transforms them using the generated $\mathrm{SE}(3)$ ego motion, and minimizes their distance to LiDAR points transformed using the recorded ego motion. This geometric constraint promotes physical consistency with the measured scene by jointly supervising generated depth and motion alongside their standard flow-matching objectives. At inference, trajectory selection relies only on a simple label-free consensus rule, with no learned scorer or simulator feedback. We evaluate PhysWAM across NAVSIM v1 and v2 planning, zero-shot closed-loop transfer, and future video and metric-depth prediction. Despite PhysWAM's simple selection procedure, it achieves strong planning performance and transfers zero-shot to unseen driving environments. It also generates accurate metric depth and temporally coherent video, with CPP improving both planning and depth prediction. Together, these results demonstrate that the geometric relationship between scene depth and ego motion provides a direct way to couple world and action generation within a simple unified model.

#20

h=22

Learning from Shared-Control Overrides: Context-Driven Acceleration Profile Prediction for Personalized Overtaking

📅 2026-09-29cs.AIcs.LGcs.RO👤 Lounis Adouane (h=22)

Ruizheng Xu, Lounis Adouane, Javier Ibañez-Guzmán, Clément Zinoune

Core Contributions

Reframes driver throttle overrides of adaptive cruise control as implicit supervisory feedback for personalization, instead of treating them as noise.
A hybrid pipeline — clustering override events into prototype acceleration profiles, a context classifier selecting one, and a residual regressor refining it — avoids the over-smoothing of end-to-end regression.
On real public-road data, reconstructed profiles track the driver's expected overtaking behaviour, pointing to fewer manual interventions.

Show abstract

Adaptive Cruise Control (ACC) systems are typically calibrated for an average driver, often resulting in a mismatch between vehicle behavior and individual expectations during time-critical maneuvers such as highway overtaking. When the ACC is perceived as too conservative and inconsistent, drivers intervene through throttle overrides, providing implicit feedback on the system's behavior. This paper reframes these override actions as human-in-theloop supervisory signals and proposes a data-driven framework for personalized vehicle adaptation, termed Context-driven Personalized ACC (CoP-ACC). Rather than relying solely on end-to-end regression, which tends to over-smooth dynamic responses, we introduce a hybrid pipeline combining: (i) unsupervised hierarchical clustering to extract representative acceleration profiles from override events; (ii) a context classifier that maps pre-maneuver driving conditions to the appropriate profile; and (iii) a residual regressor that refines the selected profile into a smooth, personalized acceleration profile tailored to the immediate context. Evaluated on real-world public-road data against a withheld forced-ACC baseline, the approach demonstrates high reconstruction fidelity and generates acceleration profiles that tend toward the driver's expected behavior in potential override contexts. The results highlight the potential of learning from shared-control overrides to enable anticipatory, personalized ACC behavior, reducing manual interventions and improving ride comfort.

#21

h=22

Cascaded consensus splitting for multi-branch contingency games

📅 2026-09-29cs.ROeess.SY👤 David Fridovich-Keil (h=22)

Bastien Lechardoy, Pau de las Heras Molins, Thibault Lahire, Laurent Pautet, David Filliat

Core Contributions

Existing contingency games assume one branching time for all intent uncertainty; this allows multiple branch points so each agent's intent can be revealed asynchronously.
An ADMM solver exploits scenario-level parallelism, taming the combinatorial growth in agents × intents.
In tightly coupled three-agent interactions, the multi-branch plan outperforms single-branch contingency planning while also lowering mean receding-horizon solve time.

Show abstract

Contingency games enable agents to anticipate and plan for other agents' hypothetical intents by constructing trajectories with a shared prefix and intent-dependent branches. While contingency games capture intent uncertainty, existing formulations rely on a single branching time, oversimplifying interactions in which different agents' intentions are revealed at different times. Moreover, the computational cost of such problems grows rapidly with the number of agents and intents, as all scenario-dependent best responses must be solved jointly. We introduce a multi-branch contingency architecture in which sources of intent uncertainty can be resolved progressively at different branching times, allowing the planned trajectories to adapt to the asynchronous revelation of intents. We also develop an ADMM-based solver that exploits scenario-level parallelism. Experiments on tightly coupled three-agent interactions support that the proposed architecture outperforms the conventional single-branch while achieving a lower mean receding-horizon solve time.

#25

h=21

Trajectory-Level Mode Guidance for Controllable Diffusion-Based Multi-Robot Motion Planning

📅 2026-09-29cs.RO👤 Sheng-Ze Cai (h=21)

Tianyou Yu, Shengze Cai, Chao Xu

Core Contributions

Tackles injecting coarse, partial trajectory priors into diffusion planners without collapsing their multimodality.
Guidance is applied in the reconstructed clean-trajectory space with timestep-dependent strength: strong early to steer toward the prior region, released later so diverse solutions survive.
Planning costs enter via gradient refinement and inter-robot collision costs extend it naturally to multi-robot coordination, producing controllable, diverse and safe plans.

Show abstract

Motion planning often admits multiple feasible solutions, making multimodal generation valuable, particularly for flexible multi-robot coordination. Diffusion models naturally learn such trajectory distributions, yet incorporating coarse and partial trajectory priors without restricting generation remains challenging. Such priors indicate a desirable region of the solution space rather than a single solution, motivating conditioned generation that preserves multimodality. In this paper, we guide trajectory generation in the clean trajectory space and progressively incorporate trajectory priors with a timestep-dependent guidance strength. At each reverse diffusion step, the reconstructed clean trajectory provides a unified space for integrating planning costs and partial trajectory priors. Planning costs are incorporated through gradient-based refinement, while the partial prior is progressively injected at the corresponding noise levels with decreasing guidance strength. This guides generation toward the prior in early stages while gradually releasing the constraint to preserve the inherent multimodality of the diffusion model. The framework naturally extends to multi-robot planning by incorporating inter-robot collision costs. Experiments on single- and multi-robot planning tasks demonstrate controllable trajectory synthesis, diverse feasible solutions, and safe multi-agent coordination.