🤖 Robotics arXiv Digest

Tuesday, October 20, 2026

📄 30 papers 📂 7 research areas Generated by Claude

🔭 Research Landscape

The strongest signal today is a push to make learning cheaper by moving it into a model. SuperMimic (#1) replaces cluster-scale model-free RL for humanoid motion tracking with one differentiable world model and analytic gradients, matching a 7-day, 128-GPU baseline in 3 hours on one GPU. The same logic runs through the world-action papers: TaskWAM (#14) adds Mode and State tokens so imagined futures stay on the right task stage, PhysCanvas (#15) turns a visuo-tactile video model into an evaluator and RL environment for pi0.5, and UniWALM (#16) co-denoises video, action and language so generated text serves as memory. The shared bet is that a good learned simulator or predictive representation is worth more than additional real rollouts. Multi-View Latent Action Pretraining (#25) and the predictive-geometry work (#28) attack the same bottleneck from the data side, extracting action signal from action-free video.

A second theme is a sober round of diagnosis. Perceives but Doesn't Brake (#11) shows driving VLAs encode hazards yet let ego-speed history dominate the plan, then fixes it at sampling time without retraining. RoboCasa-Shift (#19) shows that identical end-to-end drops hide different stage-level failures, CAPSafe (#17) finds models recognize unsafe actions at over 90% but plan safely far less often, and JARVIS (#24) finds assembly performance collapsing with horizon. The two GPT-6 Astra papers (#18, #30) are a useful contrast: one finds only 11.1% zero-shot success on geometry-constrained tasks, the other reaches 93.8% on tactile tasks when expert demonstrations are summarized into guidance. Together they suggest general models are strong where the task can be expressed as reasoning plus feedback, and weak where precise motion is the bottleneck.

Third, there is steady progress on structure and guarantees for classical problems. SPAR (#4) learns which pushes to try from search outcomes, #5 and #13 give learned safety and reach-avoid sets either verification-guided or probabilistic rather than worst-case, PUN (#7) uses neural operators for noise-aware planning, and RILC (#8) couples RL with ILC while keeping a stability proof. On the data side, LocoPilot (#9) and the PointHand study (#22) both argue the interface shapes the demonstration, and SceneRig (#10) turns single images into physics-stable simulation scenes. Taken together, the batch points to a field where scale-oriented learning and formal methods increasingly borrow each other's tools.

Humanoid & Dexterous Skill Learning

Getting human-like whole-body and finger skills out of motion data cheaply: differentiable world models, hybrid mimic/generalize training, and VLM-guided contact retargeting.

#1 SuperMimic: Scaling Humanoid Whole-Body Control with...
#3 Learning Adaptable Dexterous Skills via Hybrid Dexte...
#23 VLM-Guided Contact Retargeting and Generative Partne...

World-Action Models, VLAs & Representations

What a policy should predict before acting: stage-aware futures, visuo-tactile imagination, language-as-memory, multi-view latent actions and predictive geometry.

#14 Plausible Future, Wrong Choice: Steering World-Actio...
#15 Physical Canvas: A Vision Tactile World Model For Ro...
#16 UniWALM: Unified World Action Language Model for Emb...
#25 Multi-View Latent Action Pretraining for Cross-Embod...
#28 From Predictive Geometry to Action: Learning Complem...

Evaluating Foundation Models for Control

Diagnosing where VLAs and general multimodal models break: perception-action gaps, safety reasoning, robustness under shift, manual-following and GPT-6 as a controller.

#11 Perceives but Doesn't Brake: A Perception-Action Bin...
#17 CAPSafe: A Benchmark for Contextual Action and Plan...
#18 Large Multimodal Models for Direct Robot Control: GP...
#19 RoboCasa-Shift: Stage-Aware Robustness Evaluation of...
#24 JARVIS: From Visual Manuals to Autonomous Robotic As...
#30 GPT-6-Astra Understands Touch and Outperform Vision-...

Teleoperation, Data & Simulation Assets

Where robot training data comes from: loco-manipulation teleoperation, what operators see, single-image scene reconstruction and rig-agnostic motion-video generation.

#6 Joints, Camera, Action! Co-Generating 3D Motion and...
#9 LocoPilot: Single-Operator Bilateral Teleoperation f...
#10 SceneRig: An Agentic System for Simulation-Ready 3D...
#22 What Novice Operators See Changes What the Policy Le...

Planning, Safety & Control Theory

Principled planning and control under uncertainty: learned push rankers, neural safety certificates, probabilistic reach-avoid, neural-operator planning and RL-augmented ILC.

#4 SPAR: Search-trained Push Action Ranker for Efficien...
#5 Enumeration-Guided Training of Neural Hamilton-Jacob...
#7 Planning under Noise with Neural Operators
#8 Reinforced Iterative Learning Control for elastic jo...
#13 Temporal Difference Learning of Probabilistic Reach-...

Navigation, Perception & Fleet Systems

Spatial intelligence for mobile robots: evidential BEV completion, omnidirectional localization, calibration data synthesis and fleet-scale semantic memory.

#20 SynCalib: Breaking Dataset Shortcuts in LiDAR-Camera...
#21 OmniVLoc: Topo-metric Visual Localization for Omnidi...
#26 EchoNav: Evidential BEV Completion from Historical O...
#27 Industrial ReMEmbR: Scaling Spatiotemporal Memory to...

Applied Systems & Hardware

Domain-specific engineering: lunar rover thermal design, autonomous electrosurgery, and agentic discovery of car-following laws.

#2 Design and implementation of thermal control strateg...
#12 Agentic discovery of interpretable car-following law...
#29 Velocity Optimization via Thermal Imaging Enhances A...

Humanoid & Dexterous Skill Learning

#1 h=58

SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model

2026-10-20 cs.RO 👤 K. Sreenath (h=58)

Qiayuan Liao, Yiyang Shao, Junfeng Long, Kevin Zakka, Roberto Horowitz

Core Contributions

Fits one differentiable world model to logged transitions of a standard rigid-body simulator and trains the tracking policy through it by backprop over short imagined rollouts, replacing high-variance model-free RL with analytic gradients.
Cost drops sharply: 3 hours on a single GPU beats the return of a baseline trained for 7 days on 128 GPUs (over 1000x fewer GPU-hours), with 25 failures on 129,514 motions versus 179.
Scaling holds up: world-model error stays stable from 1 to 264 hours of motion and zero failures are reached after 9.8 hours on two GPUs, which makes motion-tracking scaling laws measurable.
Transfers to a physical Unitree G1 on acrobatic motions, so the differentiable model is not just a simulator artifact.

Show abstract

Tracking large human-motion datasets has become the dominant route to general, expressive humanoid whole-body control, but scaling across many motions is expensive. Current pipelines use model-free reinforcement learning, either merging per-motion specialists through lossy distillation, or brute-forcing a single policy with cluster-scale compute. SuperMimic amortizes learning across all motions with one differentiable world model of their shared dynamics, fit from logged transitions of a standard rigid-body simulator. The world model and the tracking policy train together in one loop, and backpropagation-through-time over short imagined rollouts gives the policy analytic gradients. Fast imagined rollouts and analytic gradients make policy learning efficient. In 3 hours on a single GPU, SuperMimic surpasses the return of a state-of-the-art motion tracker baseline trained for 7 days on 128 GPUs. It reduces GPU-hours by more than $1000\times$ and fails on only 25 of 129,514 benchmark motions, versus 179 failures for the baseline. After 9.8 hours on two GPUs, SuperMimic achieves zero failures. From 1 to 264 hours of motion, the world model's prediction error stays stable, and GPU-hours to a target return grow slowly. Our policy transfers to a physical Unitree G1 across diverse and acrobatic motions. SuperMimic removes compute as the bottleneck to scaling humanoid whole-body control, making it practical to measure motion-tracking scaling laws.

#3 h=48

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

2026-10-20 cs.ROcs.CVcs.GRcs.LG 👤 Shubham Tulsiani (h=48)

Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye, Sungjae Park

Core Contributions

Identifies the trade-off in dexterity learning: goal-reaching RL generalizes but yields unnatural fingers, while motion tracking looks human but cannot deviate from its clip.
Hybrid Dexterous Mimicking trains one shared policy on two environment populations at once, dense-tracking environments that inject the human prior and noised-goal environments with sparse object-success reward that force generalization.
Needs only a few seconds of hand-object trajectory per skill, from mocap or reconstructed from in-the-wild video, and yields grasping, in-hand reorientation with finger gaiting and throw-and-catch.
Shows transfer across simulators and to a real hand, which matters because naive RL dexterity policies often break at that step.

Show abstract

Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.

#23 h=15

VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction

2026-10-20 cs.ROcs.AIcs.CV 👤 Kashu Yamazaki (h=15)

Yuto Shibata, Kotaro Amaya, Kashu Yamazaki, Lalit Jayanti, Yoshimitsu Aoki

Core Contributions

Targets humanoid-human contact where the partner is occluded or out of view, which breaks pose estimation and corrupts mocap from heavy occlusion.
Uses a VLM to find semantically meaningful contacts and retarget them to humanoid-human geometry, then trains privileged policies with contact rewards and distills them into egocentric depth policies.
Adds generative partner-state modeling to predict the partner's motion from occluded views, improving contact recall on four of five interactions.
Runs all five interactions on a real Unitree G1 using onboard sensing only, without external tracking.

Show abstract

Humanoid-human interaction requires robots to operate under severe partial observability: during close physical contact, the human partner is occluded, truncated, or outside the robot's field of view, causing pose estimation to fail and often necessitating external tracking or privileged human state. At the same time, heavy mutual occlusions introduce errors into the human-human motion-capture data used for learning. We present REACT, a framework for learning deployable humanoid-human interaction policies from human-human demonstrations. REACT treats semantic contact structure as the key supervisory signal connecting demonstration processing, interaction learning, and onboard control. First, a VLM-guided contact modeling and retargeting pipeline identifies semantically meaningful contacts and transfers them onto humanoid-human geometry while preserving interaction structure. Using these annotations, privileged interaction policies learn physically grounded behaviors through interaction-aware contact rewards and are then distilled into deployable egocentric depth policies trained entirely from onboard sensing using photorealistic human depth rendering. To address severe partial observability, we introduce generative partner-state modeling objectives that predict partner motion from occluded egocentric views. We evaluate REACT in simulation and on a real Unitree G1 humanoid across five contact-rich interactions. Semantic contact modeling substantially improves preservation of demonstrated interactions, while generative partner-state modeling improves contact recall on four of five interactions, most under heavy occlusion. On real hardware, REACT performs all five interactions using onboard sensing alone.

World-Action Models, VLAs & Representations

#14 h=23

Plausible Future, Wrong Choice: Steering World-Action Models toward Intended Future

2026-10-20 cs.ROcs.AI 👤 Jia-Wei He (h=23)

Zhuohao Li, Jiawei He, Yuxuan Yan, Lang Zhou, Yinghao Li

Core Contributions

Identifies Stage Confusion: in long-horizon tasks where similar observation-action segments recur, world-action models generate futures that are locally plausible but belong to the wrong task stage.
TaskWAM adds Mode Tokens for fine-grained task requirements and State Tokens for execution progress, forming a structured task state that steers generation in closed loop.
Ships StageBench, a controlled multi-stage benchmark with parameterized stage variations and automated scoring, so the failure can be measured.
Plugs into different WAM architectures and embodiments (single-arm, dual-arm, dexterous hands, humanoids), suggesting it fixes a general weakness.

Show abstract

World-Action Models (WAMs) have emerged as a promising paradigm for embodied generalist policies by jointly modeling future world evolution and robot actions across diverse manipulation tasks. However, representing diverse futures is only the first step: a capable WAM must also generate futures that match the requirements of the current task and execution progress. In long-horizon manipulation where similar observation-action segments occur across different tasks and stages, existing approaches may generate locally plausible yet misdirected rollouts to drift toward wrong task stage, i.e., a phenomenon we term Stage Confusion. To address this problem, we propose TaskWAM, a closed-loop, stage-aware framework that represents fine-grained task requirements with Mode Tokens and tracks execution progress with State Tokens, forming a structured task state to steer world--action generation toward intended future. We further introduce StageBench, a controlled multi-stage manipulation benchmark with parameterized stage variations and automated evaluation.Extensive experiments on both simulation benchmarks and real-world tasks demonstrate that TaskWAM effectively reduces confusion and achieves competitive performance with state-of-the-art methods. Moreover, TaskWAM provides strong plug-and-play transfer across heterogeneous WAM architectures and robot embodiments, including single-arm robots, dual-arm grippers, dexterous hands, and humanoid robots.

#15 h=20

Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning

2026-10-20 cs.RO 👤 Yu-Peng Zheng (h=20)

Zebin Xing, Yuhang Zheng, Yupeng Zheng, Yichen Zhang, Zichen Li

Core Contributions

Argues that visually plausible world-model rollouts can misrepresent contact dynamics, making them unreliable for policy evaluation and learning.
Maps RGB, tactile images, marker displacement, task state and actions into one visual space with a shared video VAE and backbone, so touch reuses visual and temporal priors, with forward-kinematics motion images for action conditioning.
Post-trains with scene, modality and state reward models; the state reward raises the mean Pearson correlation between world-model and real success rates from 0.602 to 0.885.
Using the world model for GRPO updates lifts pi0.5 from 59.1% (SFT) to 70.8% real-world success.

Show abstract

Evaluating and post-training robot policies through physical interaction is costly and difficult to scale, motivating action-conditioned world models as interactive simulators. However, visually plausible predictions can misrepresent contact dynamics, providing unreliable feedback for policy evaluation and learning. We introduce PhysCanvas, a state-consistent visuo-tactile world model for contact-rich robot manipulation. PhysCanvas unifies multi-view RGB, tactile images, marker displacement, task state, and actions within a unified vision space. All modalities share a video VAE and pretrained generative backbone, allowing optical touch to reuse visual and temporal priors. Motion images derived through forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. Our consistency-aware post-training uses the frozen Scene Reward Model, Modality Reward Model, and State Reward Model to supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes. The resulting world model supports closed-loop policy rollouts for evaluation and provides imagined interactions and rewards for GRPO policy updates. Across four real-world manipulation tasks, when policies are rolled out in the world model, State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model further increases $π_{0.5}$'s mean real-world success rate from a 59.1% SFT baseline to 70.8%. These results demonstrate the value of unified multimodal modeling and consistency-aware post-training for robot policy evaluation and learning. Project page: https://zebinx.github.io/PhysCanvas_HomePage/.

#16 h=19

UniWALM: Unified World Action Language Model for Embodied Robot Control

2026-10-20 cs.ROcs.AIcs.CV 👤 J. van Baar (h=19)

Rahul Sajnani, Diego Romeres, Mingyuan Zhang, Kai Gao, Carl Qi

Core Contributions

Puts video, action and language into one mixture-of-transformers with per-modality diffusion heads that co-denoise, rather than attaching a VLM planner as a separate stage.
Generated language is fed back at the next step, so subtask and textual memory work as a loop without an external planner or memory bank.
Adds Flow Matching with Correction, which teaches recovery from perturbed action trajectories at no inference-time cost.
Reaches 34.5% on memory-dependent RMBench versus 28.5% for a keyframe-equipped baseline, and on a real Find-the-Item task textual memory lifts success from 11.1% to 61.1% while VLAs stay at or below 33.3%.

Show abstract

Motivated by the human ability to integrate experience across language, perception, and action, we introduce UniWALM, a unified world-action-language model that learns video, action, and language in a single shared representation, where each modality is both a conditioning signal and a supervision target. We ask: can jointly generating language enrich this shared representation, improving manipulation and supporting memory-dependent behavior? Unlike prior approaches that attach a reasoning or vision-language model to a video-action backbone and run them as distinct stages, UniWALM unifies all three modalities inside a single mixture-of-transformers backbone whose per-modality diffusion heads co-denoise synchronously, decoding subtask and textual memory in the same pass as action with no separate planning stage; the generated language is re-consumed on the next step, closing a language-as-memory loop without an external planner or memory bank. To keep the action stream reliable under this richer objective, we further introduce Flow Matching with Correction (FMC), a training objective that teaches recovery from perturbed action trajectories at no inference-time cost. UniWALM consistently improves over same-backbone baselines (e.g., FastWAM) on compositional pick-and-place (MolmoSpaces) and across the memory-dependent subset of RMBench, where it reaches 34.5% without any pretrained language or reasoning model and exceeds Mem-0's keyframe-equipped setup (28.5%) using only a single anchor frame. On a real-world Find-the-Item task, generating textual memory lifts success from 11.1% to 61.1%, while state-of-the-art VLAs stay at or below chance (<=33.3%).

#25 h=13

Multi-View Latent Action Pretraining for Cross-Embodiment Robot Learning from Videos

2026-10-20 cs.RO 👤 Angela P. Schoellig (h=13)

Oliver Hausdörfer, Felix Minzenmay, Mani Majd, Maximilian P. Christof, Shiyuan Zhang

Core Contributions

Argues single-view latent action models resolve 6D robot actions ambiguously, missing out-of-plane motion and occlusions, and fixes this with a multi-camera LAM using cross-attention.
Dual-view pretraining improves decodability of ground-truth 6D actions by 17% over single-view, and scaling single-view data by two orders of magnitude does not recover the gap.
Downstream success rises by 13.5% versus 6.2% for single-view; adding a second viewpoint beats adding up to four more embodiments.
On hardware with human-hand video demonstrations it improves success by 20% over action-only policies across 850 rollouts, while noting video benefits are task-dependent.

Show abstract

Action-free videos are an appealing substitute for costly teleoperated data. Latent action models (LAMs) turn videos into a supervision signal by inferring abstract actions between frames. Existing LAMs observe a single camera view, which resolves 6D robot actions only ambiguously missing out-of-plane movements and potential occlusions. We propose a LAM that ingests synchronized multi-camera views using cross-attention to resolve depth and occlusions before compressing them into latent actions. The latent actions are fed into a policy in which a latent expert and an action expert are coupled by sequential flow matching around a frozen VLM backbone. We further evaluate in strict cross-embodiment settings where task-specific demonstrations come from other embodiments. We show that dual-view LAM pretraining improves the decodability of ground-truth 6D actions by 17% compared to single-view, while scaling single-view data by two orders of magnitude is not sufficient to fully recover out-of-plane motions. The downstream policy improves success rate by +13.5% compared to +6.2% for single-view LAM. A single cross-embodiment is sufficient for skill transfer, and adding a second viewpoint outperforms adding up to four further embodiments. The results transfer to hardware where demonstrations come from a human hand improving success rate by +20% over action-only policies in 850 rollouts. We show that video and robot data are complementary, and that benefits of learning from video are task-dependent. Website: https://learnsyslab.github.io/multiview_latent_action_pretraining/

#28 h=11

From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

2026-10-20 cs.RO 👤 Hao Li (h=11)

Xiaoxu Xu, Hao Li, Hongtao Zhang, Yili Tang, Jiafei Cao

Core Contributions

Asks how predictive geometry should be represented for action generation, noting prior work either adds current geometry or uses future prediction only as auxiliary supervision.
Pretrains on multi-view video, compressing frozen 3D-foundation-model features into predictive geometry, then applies action supervision to geometric transitions to get a complementary motor representation.
Future observations are privileged supervision only, so inference needs no future input; gains include +10.4 absolute points on WidowX evaluation and improvements on LIBERO, LIBERO-Plus, SimplerEnv and real tasks.
Transfers across several VLA and WAM architectures.

Show abstract

Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction. Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation. Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips. During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution. Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions. During downstream post-training, predictive geometry is transferred through geometry queries, while the motor representation guides intermediate features along the action-generation pathway. Future observations are used only as privileged supervision during training and are not required at inference time. Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world bimanual and single-arm tasks, our approach consistently improves downstream policies, including an absolute gain of 10.4 percentage points on the WidowX evaluation. We further demonstrate transfer across multiple VLA architectures and WAMs, with improvements in both future-world and action prediction. The code will be available at https://github.com/xuxiaoxxxx/CPR.

Evaluating Foundation Models for Control

#11 h=27

Perceives but Doesn't Brake: A Perception-Action Binding Gap in Reasoning Driving VLAs

2026-10-20 cs.ROcs.AI 👤 Marco Pavone (h=27)

Matteo Cercola, Katie Z Luo, Simone Formentin, Marco Pavone

Core Contributions

Names a failure mode: reasoning driving VLAs encode in-path hazards in their visual features but longitudinal plans are dominated by ego-speed history and barely react to hazard proximity or the model's own reasoning.
The trajectory generator can brake, so the fault is in coupling rather than capability, which makes a retraining-free fix plausible.
Injects a hazard-conditioned energy into the frozen flow-matching sampler of Alpamayo-1.5, cutting footprint intrusion from 58.1% to 14.5% with exact obstacle geometry and removing collisions in the braking-relevant closed-loop tests.
Shows the plan flips from tracking ego speed to responding to obstacle range, supporting the diagnosis.

Show abstract

Reasoning-augmented vision-language-action (VLA) driving models can perceive and reason about safety-critical hazards yet fail to translate that evidence into safe actions. We characterize this failure as a perception-action binding gap: hazard information is available inside the model but exerts insufficient control over trajectory generation. Using matched simulated counterfactuals and logged driving replay, we show that in-path hazards are actually encoded in the visual representation, but longitudinal plans are dominated by ego-speed history and respond only weakly to hazard proximity or the model's own reasoning trace. Further analysis shows that the trajectory generator is capable of braking but sometimes fails to do so due to the perception-action binding gap. Without retraining or modifying any weights, we inject a hazard-conditioned energy into the frozen flow-matching sampler of Alpamayo-1.5, restoring the missing coupling at generation time. With exact obstacle geometry, guidance cuts footprint intrusion from 58.1% to 14.5% and recovers safe braking whenever sufficient stopping distance remains, eliminating collisions entirely in our braking-relevant closed-loop battery. It also reverses the diagnosed dependence of the plan, from tracking ego speed to responding to obstacle range. These results separate hazard perception from hazard control in driving VLAs, show that unsafe behavior can arise even when the model has both the evidence and the ability to act on it, and propose a simple fix for bridging this binding gap.

#17 h=19

CAPSafe: A Benchmark for Contextual Action and Plan Safety Reasoning in Embodied Agents

2026-10-20 cs.ROcs.CL 👤 Svetlana Stoyanchev (h=19)

Youmna Farag, Svetlana Stoyanchev, Simon Keizer, Mohan Li, Rama Doddipatla

Core Contributions

Separates action-level safety (is this step safe in context) from plan-level safety (is a long-horizon plan safe), with situational hazards, co-occurring hazards and severity labels.
Evaluating nine LLMs shows recognition and planning diverge: best models exceed 90% on unsafe-action recognition, yet the top safe-plan success is 78.19% (Gemini-3-pro) and others fall below 65%.
The result implies that safety filters validated on single actions overstate how safe an embodied agent is over a full task.

Show abstract

Reasoning about safety is essential for embodied agents powered by large language models (LLMs). We introduce the Contextual Action and Plan Safety benchmark (CAPSafe), designed to evaluate safety reasoning at two levels: (1) an action-level dataset that tests whether a single action is safe given its environmental context, and (2) a plan-level dataset that evaluates whether agents can generate safe long-horizon plans for complex hazardous household tasks. CAPSafe models safety conditions where hazards are situational (inferred from object properties), multiple hazards can co-occur within the same environment, and each is annotated with a relative severity level for fine-grained analysis. Evaluating nine closed and open-source LLMs reveals a substantial gap between hazard recognition and safe long-horizon planning. Several models achieve high accuracy in recognizing unsafe actions (exceeding 90% in the best cases), yet safe long-horizon planning remains substantially harder. Gemini-3-pro achieves the highest safe success rate (78.19%), while other models remain below 65%. These results highlight that robust safety reasoning over long action sequences remains an open challenge.

#18 h=19

Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning

2026-10-20 cs.RO 👤 Shang-Hang Zhang (h=19)

Youwei Zhou, Bohan Hou, Yankai Fu, Yaoxu Lyu, Yifan Ye

Core Contributions

Tests GPT-6 Astra as a direct closed-loop manipulation controller, first probing depth perception, embodied in-context learning and tactile correction, then five benchmarks across six capability axes.
Results are mixed by design: 76.0% on RoboTwin 2.0 and 26.92% on RoboDojo, but a clear gap between semantic reasoning and physical execution.
Introduces LIBERO-GeoMotion for geometry-constrained tasks needing coordinated translation and rotation, where zero-shot success is only 11.1%.
Real-robot tests confirm deficits in motion feasibility, precision and safety, and inference latency is a further barrier to deployment.

Show abstract

Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood. This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework. We first probe three fundamental embodied capabilities-depth perception, embodied in-context learning, and tactile understanding-and demonstrate that the model can exploit depth information, acquire simple manipulation strategies from contextual demonstrations, and use tactile feedback to correct its actions. We then evaluate GPT-6 Astra across five established manipulation benchmarks along six capability dimensions. Despite competitive performance on several benchmarks-26.92\% success on RoboDojo and 76.0\% on RoboTwin 2.0-our analysis reveals a pronounced gap between semantic reasoning and physical execution, particularly in consequence-aware action planning, precision, and complex motion coordination. To further examine these limitations, we introduce LIBERO-GeoMotion, a new benchmark of geometry-constrained tasks that require precise pose alignment, coordinated translation and rotation, and anticipation of physical action consequences. GPT-6 Astra achieves only 11.1\% zero-shot success, exposing the difficulty of translating spatial understanding into physically feasible motion. Real-world experiments corroborate these findings: zero-shot execution is feasible for simple tasks, yet persistent deficits remain in motion feasibility, precise control, and physical safety, while substantial inference latency further hinders practical deployment. Overall, our results characterize the emerging embodied capabilities of general-purpose multimodal models and delineate the key challenges toward safe, efficient, and reliable robotic manipulation.

#19 h=19

RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

2026-10-20 cs.RO 👤 Ben Fei (h=19)

Haocheng Wang, Youquan Liu, Weiyu Guo, Aokai Zhang, Donghang Wu

Core Contributions

Challenges aggregate success as a robustness metric by pairing clean and perturbed rollouts that differ in exactly one shifted factor, with stage-level diagnostics.
Evaluates GR00T N1.5 on four long-horizon tasks with four perturbation factors and three severities over 1,040 rollouts.
Equal drops can hide different failures: camera L2 and Gaussian blur L1 both give a 0.188 success-rate drop, but first-stage failure rates are 19.0% and 36.4%.
Severity is not always monotonic, and one camera shift shows zero net change while 8 of 80 matched episodes flip outcome, so averages can hide real sensitivity.

Show abstract

Vision-language-action (VLA) robustness is commonly summarized by aggregate task success, which obscures how individual executions change under distribution shifts and where failures emerge in long-horizon tasks. We introduce \textbf{RoboCasa-Shift}, a diagnostic robustness benchmark built on RoboCasa365 that combines controlled single-factor shifts, matched clean-perturbed rollouts in which each pair shares the same underlying settings and differs only in the target shift, and simulator-grounded stage-aware diagnostics. We evaluate GR00T N1.5 on four long-horizon tasks that pass a predefined clean-performance gate under four perturbation factors and three calibrated severity levels, including conditions that drive the evaluated policy to complete task failure, totaling 1,040 formal rollouts. Robustness responses vary substantially across factors and tasks, and increasing severity does not always produce monotonic degradation. For example, one camera condition yields $Δ\mathrm{SR}=0$ despite 8 of 80 matched episodes changing outcome. Stage-aware analysis further shows that the same end-to-end degradation can correspond to different execution-depth profiles. Camera $L_2$ and Gaussian blur $L_1$ both yield $Δ\mathrm{SR}=0.188$, yet their first-stage failure rates are 19.0\% and 36.4\%, respectively. Our findings suggest that VLA robustness should be viewed as a multi-dimensional property of execution rather than a single end-to-end performance measure, motivating evaluation protocols that jointly capture outcome degradation, paired outcome sensitivity, and failure progression. The code is publicly available at https://github.com/HaochengWang23/RoboCasa-Shift.

#24 h=14

JARVIS: From Visual Manuals to Autonomous Robotic Assembly

2026-10-20 cs.RO 👤 V. M. Baskaran (h=14)

Fan Ding, Fucai Ke, Kevin Yuchen Ma, Xuewen Luo, Wenyu Liang

Core Contributions

Introduces a benchmark for following visual LEGO manuals with no language instruction: 236,679 tasks over 8,987 scenes, and a 50-task suite for interactive robot evaluation.
A structure-aware score separately measures geometric accuracy, structural relations, build order, partial progress and completion.
JARVIS-agentic composes robot primitives into programs and uses execution feedback for verification and recovery, with no task-specific trajectories or weight updates.
Across nine models, scores fall sharply as assembly horizon grows and execution errors stay hard to correct; prior experience helps but repeated reflection gives inconsistent gains.

Show abstract

Visual manuals communicate procedural knowledge through stepwise diagrams, but translating these instructions into reliable long-horizon robotic execution remains challenging. We introduce JARVIS, a method-agnostic benchmark for autonomous visual-manual-guided execution without additional language instructions. JARVIS comprises 236,679 LEGO assembly tasks across 8,987 scenes, with a carefully curated JARVIS-v1 suite of 50 tasks for interactive robotic evaluation. Tasks span varying initial structural complexities and assembly horizons, requiring agents to interpret manuals, ground actions in an evolving workspace, and execute the intended procedure. A structure-aware evaluation protocol captures geometric accuracy, structural relations, construction order, partial progress, and completion. We further introduce JARVIS-agentic, a tool-grounded embodied agent that composes robot primitives into executable programs and uses execution feedback for verification and recovery, without task-specific action trajectories or model parameter updates. Across nine multimodal and reasoning models, task-oriented primitives increase the average assembly score. Performance nevertheless declines sharply with assembly horizon, and execution errors remain difficult to correct. Prior-task experience improves initial performance, while repeated reflection provides inconsistent benefits, highlighting persistent limitations in connecting visual procedural reasoning with reliable physical execution.

#30 h=10

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

2026-10-20 cs.ROcs.CL 👤 Dong-Yu Zhang (h=10)

Ziyong Wang, Yupei Lin, Runxuan Huang, Liang Lin, Dongyu Zhang

Core Contributions

Tests whether a general model can use touch with no robotics-specific pretraining, by summarizing expert demonstrations into task guidance and tactile references that GPT-6-Astra uses for contact assessment and correction.
Gets 93.8% mean success on five UniVTAC tasks without per-task fine-tuning, comparable to state-of-the-art tactile policies and higher on some tasks.
On Insert Hole, tactile feedback lifts success from 31% to 99%, and execution analysis shows contact-guided probing and correction.

Show abstract

General-purpose multimodal models have shown promise as direct robot action policies, yet whether they grasp the physics underlying real-world interactions and can apply that understanding to guide action remains underexplored. We investigate this question using contact-rich manipulation as a representative testbed, examining tactile understanding as a concrete instance of this broader capability. Contact-rich manipulation requires interpreting tactile feedback and adjusting actions after obstruction. Existing learned tactile policies acquire these capabilities through specialized pretraining and per-task optimization, with substantial data and training costs. We present GPT-as-Tactile-Policy, which summarizes expert demonstrations into task guidance and tactile references for contact assessment and corrective action. GPT-6-Astra combines this context with multimodal observations and interaction history to generate control commands and adjust actions through contact feedback. Without additional robotics-specific pretraining or per-task policy fine-tuning, our method achieves 93.8\% mean success across five UniVTAC tasks. Overall performance is comparable to state-of-the-art policies, with higher success on selected tasks. On Insert Hole, tactile feedback improves success from 31\% to 99\%. Execution analysis further demonstrates contact-guided probing and correction, supporting manipulation progress through explicit expert experience and online feedback.

Teleoperation, Data & Simulation Assets

#6 h=38

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

2026-10-20 cs.CVcs.AIcs.GRcs.LGcs.RO 👤 O. Litany (h=38)

Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik

Core Contributions

Closes the gap between video models that show motion and animation pipelines that need replayable 3D motion for a specific rig, including non-human skeletons.
Keeps motion in a separate model that supports arbitrary skeletons and links it to a pretrained video model through a few cross-attention layers, so motion can be pretrained on motion-only data instead of scarce paired video.
Learns to locate each joint in the frames, which lets it recover a camera and register the 3D motion into the scene.
Builds about 200K training pairs by placing animated 3D assets into scenes with a generative video editor, and supports video-to-rig and rig-to-video plus keyframe control.

Show abstract

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.

#9 h=28

LocoPilot: Single-Operator Bilateral Teleoperation for Loco-Manipulation Policy Learning

2026-10-20 cs.RO 👤 A. Langella (h=28)

Antonio Langella, Giovanni Minelli, Giovanni B. Dessy, Lorenzo Amatucci, Giulio Turrisi

Core Contributions

Fills the gap between robot-free demonstration interfaces, which miss platform-specific signals, and on-robot teleoperation, which needs a way to command base and arm together.
One operator drives a Go2 quadruped with a Piper arm using a leader arm plus two handheld controllers, and feels estimated interaction forces through a passivity-constrained haptic channel without a force/torque sensor.
Records a unified 11D action shared by teleoperation and deployment, so a policy can run with no retargeting or extra inverse kinematics.
Validated end to end by fine-tuning MolmoAct2 on pick-and-place demonstrations and deploying on the physical robot.

Show abstract

Learning loco-manipulation policies requires demonstrations that capture coordinated base and arm motion on the target embodiment. Robot-free interfaces allow demonstrations to be collected independently of the target robot, increasing flexibility but without directly capturing platform-specific commands, proprioception, and body-mounted observations. On-robot teleoperation allows to capture these signals directly, but requires an interface capable of commanding the complete system concurrently. This paper presents LocoPilot, a single-operator bilateral teleoperation and data-collection framework implemented on a 12-DoF Unitree Go2 quadruped equipped with a 6-DoF AgileX Piper L manipulator, an Intel RealSense D435 wrist camera, and an Intel RealSense D405 head camera. A second 6-DoF Piper L arm, used as the leader and two handheld controllers provide simultaneous control of the base, body posture, manipulator, and gripper. The follower arm interaction forces are estimated without a dedicated force/torque sensor and rendered to the operator through a passivity-constrained haptic channel. A ROS~2 pipeline records synchronized multimodal observations together with a unified 11D action shared by teleoperation and policy deployment. Compared to prior work, LocoPilot uniquely unifies robot-in-the-loop legged data collection, force feedback without motion-capture instrumentation, and the same action interface at deployment. We evaluate the interface across multiple loco-manipulation tasks and validate the complete collection-to-deployment pipeline by fine-tuning MolmoAct2 \cite{MOLMO} on pick-and-place demonstrations and deploying on the physical robot without action retargeting or additional inverse kinematics. Code available at: https://github.com/iit-DLSLab/LocoPilot

#10 h=27

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

2026-10-20 cs.ROcs.AIcs.CV 👤 Matthew Tancik (h=27)

Rundong Luo, Yunong Liu, Di Cao, Matthew Tancik, Shyamal Buch

Core Contributions

Argues that visual similarity is not enough for simulation-ready scenes, since post-hoc physics settling often moves objects and yields scenes that look right but behave wrongly.
An agent combines geometric, visual and physical tools, then runs edit-and-validate loops inside a physics simulator until the scene is stable.
Improves segmentation, appearance and 3D placement, and the scenes predict real-robot policy outcomes more accurately than a baseline and better support replay of recorded manipulations.

Show abstract

Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.

#22 h=18

What Novice Operators See Changes What the Policy Learns: A User Study of Keypoint Hand Visual Representation for Teleoperated Demonstration

2026-10-20 cs.ROcs.HC 👤 Masashi Hamaya (h=18)

Amane Yamaguchi, Shohei Kamiguchi, Masashi Hamaya, Shigeo Yoshida, Takuji Narumi

Core Contributions

Asks a question rarely studied: how the way a robot hand is rendered to a teleoperator changes both the demonstrations and the policies trained on them.
In a simulated task with 18 novices and 4 experts (2,200 trials), emphasizing thumb and index finger gave 441 successes in 600 trials versus 320 for the real hand shape.
With diffusion policies on hand and object keypoints, those data went from the weakest condition (46.3% with point-cloud input) to the strongest (57.3%), a gain of +11.0 points across all five seeds, while the all-keypoint condition changed by at most +0.7.
The takeaway is that demonstration value depends jointly on what the operator sees and what the policy observes.

Show abstract

Imitation learning increasingly relies on novice operators, whose demonstrations fall short of experts' in number and inter-trial consistency. Operators act on what they are shown, yet how the rendering of the robot affects their demonstrations, and the policies trained on them, has not been examined. We study this question with PointHand, a teleoperation interface that renders a multi-fingered robot hand as sparse keypoints at its fingertips and joints, the same keypoints a policy can observe. In a simulated mug-in-box task, 18 novices and 4 experts collected 2,200 trials under three renderings: the real shape of the hand, all keypoints emphasized, or the thumb and index finger alone emphasized. The novices shown the thumb and index finger emphasized collected 441 successful demonstrations from 600 trials, against 320 for those shown the real shape. We then trained diffusion policies on these demonstrations, replacing the scene point cloud and proprioceptive state they observe with the hand keypoints and two object keypoints. Within 400 steps these demonstrations went from the weakest of the three novice conditions under the point-cloud policy (46.3%) to the strongest under the keypoint policy (57.3%), a gain of +11.0 percentage points in all five seeds. Those collected with all keypoints emphasized changed by at most +0.7 percentage points, and in no more than three seeds. The value of a demonstration therefore depends on the rendering shown during collection and on what the policy observes together. A shared representation can pay off on both sides, in the number of successful demonstrations and in the success rate of the policy trained on them. Project page: https://omron-sinicx.github.io/point-hand/

Planning, Safety & Control Theory

#4 h=48

SPAR: Search-trained Push Action Ranker for Efficient Navigation Among Movable Obstacles

2026-10-20 cs.RO 👤 Kostas E. Bekris (h=48)

Dhruv Metha Ramesh, Tri-An Nguyen, Junshan Huang, Jingjin Yu, Kostas E. Bekris

Core Contributions

Targets the setup-push problem in NAMO: a push that clears nothing itself but enables a later opening, which makes naive greedy ranking fail.
Supervises a contact-token transformer with outcomes from searches over up to two pushes per object, then reuses it unchanged to guide search or act directly as a policy.
SPAR Search cuts median simulator calls 7.0x and planning time 3.8x; at a five-push depth limit it reaches 50.7% success in 100 simulator calls versus 20.2% for random search.
On a real robot SPAR Policy succeeds 37/40 trials with no physics digital twin, beating both SPAR Search (33/40) and random search (27/40).

Show abstract

Navigation Among Movable Obstacles (NAMO) requires a robot to rearrange objects that block access to its goal. For a pushing robot, progress may depend on a setup push that enables a later opening without itself clearing a passage. The number of candidate sequences can grow exponentially with sequence length, making physics-based search costly. We introduce the Search-trained Push Action Ranker (SPAR), which learns to prioritize push primitives from simulated search outcomes. Searches over sequences of up to two pushes per object provide supervision that distinguishes immediate openings from setup pushes. A transformer trained with regression and ranking losses scores primitives using contact tokens conditioned on scene geometry and reachability regions. The same model guides simulator evaluations in SPAR Search and selects actions for direct execution in SPAR Policy. Both modes solve successive keyhole problems that open access to neighboring free-space regions within a NAMO planner. Across successful runs on $300$ simulated two-push keyhole problems, SPAR Search reduces median simulator calls by $7.0\times$, leading to a $3.8\times$ reduction in planning time relative to Random Search. Without retraining, the same ranker guides search on $600$ keyhole problems which have no solution within two pushes. At a depth limit of five pushes, SPAR Search achieves $50.7\%$ success within $100$ simulator calls, compared with $20.2\%$ for Random Search. These gains carry over to NAMO planning: on hard simulated scenes, SPAR Search achieves $47\%$ success within $100$ simulator calls, compared with $21\%$ for Random Search. In real-robot experiments, SPAR Policy succeeds in $37/40$ trials without a physics-based digital twin, compared with $33/40$ for SPAR Search and $27/40$ for Random Search.

#5 h=39

Enumeration-Guided Training of Neural Hamilton-Jacobi Safety Certificates

2026-10-20 cs.ROcs.LG 👤 E. Bartocci (h=39)

Mihaela-Larisa Clement, Tosca Pierro, Ezio Bartocci, Luca Marzari

Core Contributions

Neural Hamilton-Jacobi certificates are hard to train because unsafe regions are hard to characterize; this uses probabilistic enumeration of safety constraints to find forward-invariance violations directly.
The resulting small set of counterexamples captures the failure boundary better than adversarial-attack samples, and fine-tuning on it enlarges certified regions while cutting certification time on all benchmarks.
Builds a deployment-time shield from the refined certificate and reports zero safety violations in Isaac Sim and on real autonomous racing, against violations for a nominal controller and an LQR backup.

Show abstract

Hamilton-Jacobi (HJ) reachability analysis offers formal safety guarantees for control systems, but training accurate neural certificates remains difficult, particularly when unsafe regions are hard to characterize. We propose a verification-guided training framework that addresses this by using probabilistic enumeration of safety constraints to detect forward-invariance violations, i.e., states where the learned certificate incorrectly classifies unsafe behavior. This produces a compact, targeted set of counterexamples that more precisely captures the failure boundary than inputs generated by standard adversarial attack methods. Crucially, using this representation to fine-tune a pretrained HJ value function yields larger certified regions and reduces certification time across all tested benchmarks. We further introduce a shielding mechanism based on the refined certificates to enforce safety constraints at deployment time. We evaluate the proposed safety filter in a realistic, physics-based Isaac Sim environment and through real-world autonomous racing experiments, achieving zero safety constraint violations compared to a nominal controller and a backup policy based on Linear Quadratic Regulation (LQR). These results demonstrate the effectiveness of the proposed shielding mechanism in ensuring safe operation under high-speed, safety-critical conditions.

#7 h=33

Planning under Noise with Neural Operators

2026-10-20 cs.LGcs.RO 👤 Anima Anandkumar (h=33)

Michael Chen, Hong Chul Nam, Xi Deng, Anima Anandkumar

Core Contributions

Frames planning under execution noise as a screened-Poisson stochastic-control problem and swaps zero-Dirichlet obstacle boundaries for reflecting Neumann ones so sampled paths keep exploring rather than dying at walls.
Extends Walk on Stars to higher dimensions to solve the PDE without a configuration-space mesh, then trains a neural operator on noisy, partially converged Monte Carlo labels to amortize it.
Gets 95-100% success on the 4D replanning benchmark with the lowest runtime, and at the highest noise level gains 10.7-60.6% in success while running 3.29-32.15x faster.

Show abstract

Planning under uncertain control execution noise requires reliable feedback policies across diverse environments. We introduce Planning Under Noise (PUN), a neural operator that learns noise-aware value fields for planning in unseen geometries. Building on an established screened-Poisson formulation of stochastic optimal control, we address two challenges: specifying boundary conditions that support navigation across varying geometries and efficiently computing fields with reliable gradients. Zero-Dirichlet obstacle boundaries terminate sampled paths at walls, weakening the goal signal along long routes and making it harder to estimate reliably. We instead impose reflecting Neumann conditions on obstacles and a Dirichlet condition at the goal, allowing sampled paths to continue exploring free space until they reach the target. To solve the resulting PDE without a configuration-space mesh, we extend a Monte Carlo (MC) solver, Walk on Stars (WoSt), to higher dimensions. However, local MC fluctuations can reverse estimated gradient directions and cause planning failures, making direct planning require costly sampling. PUN amortizes this computation by learning geometry-to-field mappings from noisy, partially converged Monte Carlo labels, avoiding expensive convergence for every training example. At inference, test-time scaling uses bounded local search guided by the predicted global field. We evaluate PUN on planning tasks across dimensions. Across four noise levels, PUN matches or outperforms all compared planners, achieving 95-100% success on the 4D replanning benchmark while maintaining the lowest runtime. At the highest tested noise level in 4D, PUN achieves 10.7-60.6% relative gains in success rate over other methods while running 3.29-32.15 times faster.

#8 h=30

Reinforced Iterative Learning Control for elastic joint arms: Guaranteed cross-trajectory generalization under model uncertainties

2026-10-20 cs.RO 👤 M. Garabini (h=30)

Yuri De Santis, Michele Pierallini, Franco Angelini, Manolo Garabini

Core Contributions

Resolves a known tension: ILC gives precise, compliance-preserving tracking but must relearn per trajectory, while RL generalizes but is sample-hungry and imprecise.
Couples them concurrently, with ILC supplying high-quality samples and RL acting as a generalizing best-effort policy, so good tracking appears from the first trials.
Provides a proof that the RL component does not break ILC stability and can reduce the asymptotic error bound under stated conditions.
Validated on an underactuated 4-DoF serial elastic arm across several trajectories and model mismatches.

Show abstract

Trajectory tracking for compliant robots is challenging due to nonlinear dynamics and modeling uncertainties, often requiring learning-based methods to avoid explicit parameter identification. Iterative Learning Control (ILC) ensures theoretically high precision and preserves the passive compliance of the robot. However, traditional ILC requires a separate learning phase for each new trajectory. Conversely, Reinforcement Learning (RL) generalizes to new trajectories, but practically suffers from high sample complexity and limited tracking precision. This work proposes Reinforced Iterative Learning Control (RILC) that combines the strengths of both approaches to achieve fast adaptation and guaranteed tracking convergence across diverse trajectories. By establishing a concurrent coupling between the two controllers, RILC overcomes the high initial errors and slow adaptation of traditional ILC, enabling high-performance tracking from the very first trials. In this architecture, ILC provides high-quality samples, while RL contributes as a generalizing best-effort policy across iterations. Additionally, a theoretical proof is presented to show that the RL component preserves ILC stability and, under specific conditions, can also reduce the error bound asymptotically. The efficacy of this framework is demonstrated through simulation and experiments on an underactuated 4-degree-of-freedom serial elastic robot using different trajectories and modeling mismatches.

#13 h=24

Temporal Difference Learning of Probabilistic Reach-Avoid Value Functions

2026-10-20 cs.ROeess.SY 👤 A. Del Prete (h=24)

Elias Fontanari, Pietro Noah Crestaz, Matteo Saveriano, Andrea Del Prete

Core Contributions

Notes that classical and neural Hamilton-Jacobi reachability assumes adversarial bounded disturbances, which is conservative when disturbances are really stochastic and requires the disturbance set up front.
Learns a Probabilistic reach-avoid Value Function by self-supervised temporal-difference learning, applicable to black-box systems and unbounded stochastic noise.
Defines reach-avoid sets as states with a chosen success probability, and on systems up to a Hopper locomotion task the sets are less conservative and more accurate than adversarial HJR sets under non-adversarial noise.

Show abstract

Reach-avoid sets, defining the states from which a task can be completed while avoiding unsafe regions, are fundamental tools for deploying robots in environments subject to disturbances and uncertainty, as well as for runtime task-feasibility monitoring, allowing prediction of whether a state is safe and will lead to successful task completion. Classical Hamilton-Jacobi Reachability (HJR) and its recent neural extensions (e.g., Lipschitz Continuous Reachability Learning, DeepReach) treat disturbances adversarially, computing the reach-avoid set under worst-case disturbances from a bounded set. This guarantees safety but can yield overly conservative sets when disturbances are stochastic rather than adversarial, and additionally requires the disturbance set to be known or specified in advance. We propose Probabilistic reach-avoid Value Function (PVF), a self-supervised method, applicable to black-box systems, for computing reach-avoid sets as the states with a certain reach-avoid probability under stochastic, potentially unbounded disturbances. We evaluate our approach against HJR-based baselines on systems of increasing complexity, up to a Hopper locomotion task, and show that our probabilistic sets are less conservative and more accurate than adversarial HJR sets when disturbances are non-adversarial.

Navigation, Perception & Fleet Systems

#20 h=19

SynCalib: Breaking Dataset Shortcuts in LiDAR-Camera Calibration via Extrinsic-Controllable Synthesis

2026-10-20 cs.CVcs.RO 👤 Jian-Hao Jiao (h=19)

Huihua Zhu, Shuyang Zhang, Dongqing Chen, Zhuo Yu, Jianhao Jiao

Core Contributions

Diagnoses a dataset shortcut: learned LiDAR-camera calibration networks overfit to the fixed nominal extrinsics in their training sets and fail on new rigs.
Synthesizes multi-modal calibration data with controllable extrinsics from omnidirectional images that have depth ground truth, so extrinsic diversity can be chosen.
Models trained only on synthetic data transfer zero-shot to real camera-LiDAR configurations, and code, models and a benchmark are promised.

Show abstract

Online LiDAR-camera calibration remains a fundamental yet challenging task in the field of robotics. Recently, deep learning-based data-driven approaches have emerged, offering high-precision, targetless calibration capabilities. However, existing methods often suffer from a critical dataset shortcut phenomenon due to the lack of extrinsic diversity in their training data. Networks tend to overfit to fixed nominal extrinsic configurations, leading to severe performance degradation and poor generalization in real-world deployments. To address this limitation, we propose a highly generalizable data generation framework tailored for training LiDAR-camera calibration networks. Our framework leverages existing omnidirectional image datasets annotated with depth ground truth to synthesize multi-modal calibration data under rigorous extrinsic constraints. Extensive experiments demonstrate that the data generated by our pipeline significantly enhances the generalization and robustness of typical learning-based calibration networks. Furthermore, we conduct comprehensive real-world evaluations across various camera-LiDAR configurations, validating the zero-shot transfer capability of models trained exclusively on our synthetic data. To facilitate future research and community use, the code for data generation, the pre-trained models, and online benchmark datasets will be made publicly available.

#21 h=19

OmniVLoc: Topo-metric Visual Localization for Omnidirectional Cameras

2026-10-20 cs.CVcs.RO 👤 Jian-Hao Jiao (h=19)

Zhuo Yu, Shuyang Zhang, Huihua Zhu, Wenjia Ouyang, Zuo Zuo

Core Contributions

Exploits the surround view of omnidirectional cameras by combining global place recognition and monocular depth in a topo-metric pipeline.
Solves a scale-aware pose graph and spherical bundle adjustment to recover a metric pose for each query frame.
Adds hierarchical verification across stages to reject erroneous outputs, which is what keeps it reliable under illumination and content changes.

Show abstract

Omnidirectional cameras provide surround-view information, significantly overcoming the field-of-view constraints for traditional visual localization systems. However, existing localization systems fail to fully exploit the characteristics of omnidirectional cameras, struggling to deliver accurate and robust localization performance amidst scene content variations and illumination-induced appearance changes. In this work, we propose OmniVLoc, a surround-view localization system based on a topo-metric framework. By integrating state-of-the-art global place recognition and monocular depth estimation methods, we introduce a multi-constrained visual localization paradigm. By solving a scale-aware pose graph and performing precise spherical bundle adjustment, we accurately recover the metric-level pose for each query frame.Furthermore, a hierarchical verification mechanism is designed to filter out erroneous outputs across different stages, effectively bolstering the overall system robustness. Extensive experiments on both public and our custom datasets validate the superior accuracy and reliability of OmniVLoc. To benefit the community, our source code and dataset will be made publicly available upon publication.

#26 h=13

EchoNav: Evidential BEV Completion from Historical Observations for Visual Navigation

2026-10-20 cs.RO 👤 Chi Zhang (h=13)

Diwen Liu, Yiyang Shao, Borui Wang, Chi Zhang, Yunhui Liu

Core Contributions

Predicts a local evidential bird's-eye-view map from short RGB sequences, with no depth, camera pose input or explicit geometric warping.
Adapts frozen VGGT-Omega features and merges historical residual proposals with cellwise weights and a learned no-update option, so it can both recall seen structure and infer occluded space with confidence.
Reaches 89.6% success in sequential two-goal simulation (32.5 points above the best baseline) and 73.3% over 60 wheeled-robot trials versus 33.3% for the best baseline, with demos on quadruped and humanoid.

Show abstract

Visual navigation in cluttered environments requires robots to recover previously observed structure and infer obstacles and free space that remain occluded. We present a framework that combines temporal aggregation and occupancy completion in an evidential bird's-eye-view (BEV) representation predicted from short RGB sequences. A decoder adapts frozen VGGT-Omega features and combines historical residual proposals using cellwise weights and a learned no-update option. The BEV predictor produces a local map in the latest ego frame without depth or camera-pose inputs or explicit geometric warping. The evidential model treats observed free space separately and predicts occupancy with associated confidence across the remaining supported region. We train the predictor on simulated RGB sequences paired with complete and visibility-masked occupancy annotations. Our experiments evaluate navigation in simulation and on a wheeled robot, with additional demonstrations on quadruped and humanoid robots. In simulated sequential two-goal navigation, our system achieves 89.6% success, outperforming the strongest evaluated baseline by 32.5 percentage points. Across 60 wheeled-robot trials in 20 scenes, it achieves 73.3% success, compared with 33.3% for the strongest evaluated baseline.

#27 h=11

Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets

2026-10-20 cs.ROcs.AIcs.SE 👤 Abrar Anwar (h=11)

Jad Yammine, Malek Wahidi, Abrar Anwar, Yan Chang, Mihir Acharya

Core Contributions

Extends NVIDIA ReMEmbR to warehouse fleets by generating scene descriptions only on demand instead of captioning everything at write time.
Each Jetson encodes clips into retrieval embeddings, vision tokens and reconstruction latents without uploading raw frames, and reuses the models to blank out people for GDPR-compliant PII abstraction.
Splits Cosmos3-Edge at the vision-language boundary, putting the encoder on the robot and the decoder on a server, and cached-token decoding matched the unsplit model in all 450 tested decodes.

Show abstract

Warehouse robot fleets continuously observe operational conditions, but preserving these observations as reusable semantic memory incurs substantial computation, communication, and storage costs. For example, explaining recurring aisle blockages can require reviewing video from multiple robots across several shifts. We present IndustrialRemembr, a shared spatiotemporal memory system that extends NVIDIA ReMEmbR to robot fleets and reduces server-side computation by generating scene descriptions only on demand. On each robot, an NVIDIA Jetson encodes video clips into retrieval embeddings with Cosmos-Embed1, vision tokens with Cosmos3-Edge, and reconstruction latents with the Wan2.2 VAE, without uploading raw camera frames. The encoding models are re-used to detect people in image patches, zeroing them out to support PII abstraction for GDPR-compliant deployments. We split Cosmos3-Edge at its vision-language boundary, placing the vision encoder on the robot and the language decoder on an on-premises server. A Nemotron agent retrieves memories by content, position, and time and queries the decoder using cached vision tokens. This design eliminates caption generation at write time, while cached-token decoding reproduces the unsplit model's output in all 450 tested decodes.

Applied Systems & Hardware

#2 h=49

Design and implementation of thermal control strategy for micro-size lunar exploration rover hakuto

2026-10-20 cs.RO 👤 Kazuya Yoshida (h=49)

Toshiki Tanaka, Takuto Oikawa, Keeni Shruti, Teruhito Iida, Kazuya Yoshida

Core Contributions

Treats a 4 kg micro-rover as six radiating surfaces and solves the energy balance over solar flux, lunar radiation, albedo and regolith conduction to choose avionics placement and solar-panel angle.
The mass cap forces thermal design to be done jointly with power generation rather than bolted on afterward.
Closes the loop with detailed numerical simulation and a final thermal-balance test of the flight model in vacuum, which is the evidence flight reviews actually require.

Show abstract

The research focuses on the thermal interface design of the HAKUTO Rover Flight Model (FM) and its evaluation using the final thermal vacuum experiment. Initially, an analytical thermal model was developed, dividing the rover into six distinct radiating surfaces. The surface temperature of the rover was then determined based on its orientation relative to the solar radiative flux, as well as other thermal factors such as lunar radiation, solar albedo, and conduction from the lunar regolith. By considering the impact of these heat sources, suitable mounting locations for the avionics and the angle of the solar panels were determined to optimize the rover's thermal performance and power generation while ensuring the rover's mass remained under 4 kg. After the preliminary thermal analysis, numerical computational simulations were conducted to further elaborate on the thermal response of the final HAKUTO Rover FM design, which was then verified through a thermal balance test in a vacuum environment.

#12 h=25

Agentic discovery of interpretable car-following laws from multi-source trajectory data

2026-10-20 cs.RO 👤 Ke-Shuang Tang (h=25)

Jiamei Xu, Jie Sun, Keshuang Tang, Jian Sun

Core Contributions

Uses an LLM agent to propose car-following model structures from traffic-flow knowledge and reconstruction errors, paired with evolutionary symbolic search and physics-aware multi-objective scoring.
Discovers the Adaptive Equilibrium Response Model, which combines state-dependent relative-speed response with spacing-dependent equilibrium restoration and admits closed-form equilibrium and flow-density relations plus stability conditions.
Cuts spacing RMSE by 9.11%, 7.59% and 21.73% versus the strongest of seven classical baselines on Shanghai, NGSIM I-80 and Waymo data, while keeping the interpretability black-box models lose.

Show abstract

Car-following (CF) models is a fundamental component of microscopic traffic simulation and underpin the analysis of collective traffic dynamics, while its modeling becomes more challenging due to diverse human driving behaviors and the increasing presence of automated vehicle. Analytical CF models offer interpretable representations of longitudinal vehicle interactions, but their prescribed functional structures may limit the representation of heterogeneous driving behavior. This study proposes an agent-centered, physics-aware framework for the empirical discovery of CF laws from multi-source trajectory data. The framework iteratively combines agent-guided hypothesis generation, informed by traffic flow knowledge and state-dependent reconstruction errors, with evolutionary symbolic search and physics-aware multi-objective evaluation. Using NGSIM US-101, highD, and exiD, the framework discovers the Adaptive Equilibrium Response Model (AERM), which couples state-dependent relative-speed response with spacing-dependent equilibrium-speed restoration. AERM admits analytical equilibrium speed-spacing and flow-density relationships; its stability conditions are derived for the zero-delay formulation, and platoon simulations examine disturbance propagation. Evaluated against seven classical CF models on Shanghai naturalistic driving data, NGSIM I-80, and Waymo trajectories, AERM attains the lowest spacing and speed reconstruction errors under trajectory-specific calibration, reducing spacing RMSE by 9.11%, 7.59%, and 21.73% relative to the strongest baselines, and remains competitive under dataset-level shared parameters. These results demonstrate the fitting flexibility and analytical tractability of the discovered CF law, providing an interpretable basis for microscopic traffic simulation.

#29 h=11

Velocity Optimization via Thermal Imaging Enhances Autonomous Tumor Resection

2026-10-20 cs.RO 👤 M. Kam (h=11)

Naveed D. Riaziat, Jiawei Ge, Ethan Kilmer, Michael W. Mikula, Paul Maria Scheikl

Core Contributions

Replaces fixed-velocity electrosurgery with closed-loop velocity control driven by real-time thermal imaging, balancing tissue damage against margin accuracy.
Needs no data-driven training or iterative tuning, which eases adoption for a safety-critical procedure.
On ex vivo porcine tissue with the UR10e-based ASTR system, thermal injury drops by up to 52% and cut accuracy improves by up to 59% versus two fixed-velocity methods, without slowing the task.

Show abstract

Objective: To demonstrate and evaluate THERMO, a thermal-velocity optimization algorithm for autonomous electrosurgery, in ex vivo porcine tissue using the UR10e-based ASTR system. Methods: Thermography for Electrosurgical Rate Modulation via Optimization (ThERMO), a closed-loop algorithm that uses real-time thermal imaging data to minimize thermal injury while ensuring accurate cut margins, was tested on ex vivo porcine tissue. Histological analysis was performed to detect thermal injury extent at a cellular level, and 3D photogrammetric measurement was used to verify margin accuracy. ThERMO's performance was compared against two previously used fixed velocity methods. Results: ThERMO outperformed both fixed velocity comparison methods without requiring data-driven training or iterative tuning, reducing thermal injury by up to 52%, improving cut accuracy by up to 59%, and maintaining task completion speed. Conclusions: Closed-loop thermal feedback via ThERMO improves autonomous robotic electrosurgery by simultaneously reducing thermal injury and improving margin accuracy, reinforcing its value over fixed velocity approaches for tumor resection applications. Significance: This work advances autonomous robotic tumor resection by demonstrating that real-time closed-loop thermal feedback control can substantially reduce collateral tissue damage and improve surgical margin accuracy.