🤖 Robotics arXiv Digest

Wednesday, October 21, 2026
📄 30 papers 📂 6 research areas Generated by Claude

🔭 Research Landscape

Today's batch is dominated by one question: what should a robot foundation model predict, and what should it be grounded in before it acts? Five papers extend the World Action Model (WAM) recipe of jointly generating video and actions, each adding a different missing ingredient. AW-0 (#9) adds executable code as a third generative stream so skills become reusable programs; PhysiCalWorld (#10) adds a one-shot physics descriptor so rollouts stop drifting on unseen mass or friction; Relational Geometry Forcing (#18) shows that aligning pointwise 3D features actually hurts action learning and that pairwise relations are the fix; CW-WAM (#25) adds contact wrench as a predicted modality; and DepState-JEPA (#30) grounds a latent state in metric depth. Read together, they say that pixels alone are an under-constrained target, and the field is converging on adding structure (programs, physics, geometry, force) at training time while keeping inference cheap.

The second theme is the post-deployment life of a policy. TIDE (#13), SimPT (#20), ACT-Flow (#15) and ASTEROID (#3) all try to absorb corrections or exploration without a heavy actor-critic stack: provenance-conditioned supervised learning from human interventions, simulator-compiled progress graphs that automate those interventions, a critic used only to weight residual targets during training, and privileged-teacher distillation on the student's own on-policy history. Post-hoc kernel-variance heads (#6) supply the missing trigger, telling a single VLA when it is out of distribution at roughly 50x lower scoring cost than ensembles, and FlowVLA (#21) shows the flip side: some tasks like pouring need an explicit physical-completion estimate that a vision-only policy cannot see. The shared bet is that supervised objectives on well-chosen data can replace much of online RL.

A third thread is skepticism about what success metrics reveal. NavBind (#23) finds that higher navigation success does not mean a VLA follows its instruction, the PAPAV survey (#2) counts that only 5 of 64 agent benchmarks assess Anticipate and 3 assess Verify, TARSphere (#17) finds general MLLMs can beat embodied-tuned ones, and the thermal-HRI study (#22) documents hallucinated actions. Meanwhile the agentic-harness papers (SERA #28, CausGround #12, the recursive self-improvement position paper #11) push the opposite direction, treating a frozen VLM or VLA as a component to be wrapped in editable memory, causal knowledge and self-revision. Expect the next round of work to pair these: harnesses that improve, evaluated by diagnostics that test whether the improvement is real.

World-Action Models & Predictive Representations

What a manipulation world model should predict and be grounded in: code, calibrated physics, relational geometry, contact wrench and depth-grounded latent state.

  • #9 Agentic World Action Models
  • #10 PhysiCalWorld: In-Context Physics Calibration for Wo...
  • #18 Relational Geometry Forcing for World Action Models
  • #25 CW-WAM: A Structured Contact-Wrench World Action Mod...
  • #30 DepState-JEPA: Learning Depth-Grounded State Represe...

VLA Post-Training, Uncertainty & Online Learning

Making deployed policies improve and know their limits: in-context exploration, cheap epistemic uncertainty, human-in-the-loop online and simulated post-training, residual correction and flow-aware completion.

  • #3 Asymmetric On-Policy Distillation Learns In-Context...
  • #6 Post-hoc Epistemic Uncertainty for Vision-Language-A...
  • #13 Training in Deployment: Human-in-the-loop Online Pos...
  • #15 Learning to Absorb Corrections: Critic-Weighted Resi...
  • #20 SimPT: An Automated Simulation Framework for Interve...
  • #21 FlowVLA: Learning Flow-Aware Physical Completion for...

Agentic Planning & Embodied Reasoning

Foundation models as robot brains: a capability framework, self-improvement as a research agenda, causal grounding, long-horizon social benchmarks and self-evolving harnesses.

  • #2 Multimodal Embodied Agents: A Unified Capability-cen...
  • #11 Toward Recursive Self-Improvement in Physical AI
  • #12 CausGround: Causal Interaction Grounding for Adaptiv...
  • #17 TARSphere: A Robot-Oriented Full-Stack Framework for...
  • #28 From VLM Planners to Robotic Agents: Self-Evolving H...

Navigation, Perception & Mapping

Spatial competence for mobile platforms: active perception, floor-plan registration, balance-aware RL, LiDAR pretraining, instruction-binding diagnostics, aerial benchmarks and flow-matching + MPC.

  • #4 Object- and Task-Aware Active Perception for Safe Na...
  • #14 Reading the Room: Scale-ambiguous Semantic Registrat...
  • #16 State-Dependent Objective Coordination for End-to-En...
  • #19 RangeNav: Task-Aligned LiDAR Range-Image Pretraining...
  • #23 NavBind: Diagnosing Instruction-Action Binding at Ro...
  • #27 AirPlayground: Benchmarking Aerial Navigation across...
  • #29 Integrating Flow Matching and Model Predictive Contr...

Manipulation Control, Retargeting & Scene Reasoning

Classical-meets-learned control and reasoning for arms and humanoids: payload-generalizing feedforward, distance-field RL, streaming retargeting and open-world shelf organization.

  • #5 Payload generalization on feedforward control action...
  • #8 Distance Field Driven Reinforcement Learning for Rea...
  • #24 SMR: Learning streaming motion retargeting for human...
  • #26 OMAR: Open-World Multi-Attribute Reasoning for Scene...

Design, Assistive Devices & Failure Analysis

Hardware-in-the-loop and human-facing robotics: computational LEGO walker design, prosthetic grasp disambiguation and VLM failure modes on thermal video.

  • #1 Make it Walk: Computational Design of LEGO Walkers
  • #7 Extending Affordance Segmentation for Wearable Robot...
  • #22 Failure Modes of Vision-Language Models for Human Ac...
World-Action Models & Predictive Representations
#9 h=24

Agentic World Action Models

2026-10-21 cs.RO 👤 Roei Herzig (h=24)
Zhuoyang Liu, Dantong Niu, Botong Zhang, Boning Shao, Jing Wang
Core Contributions
  • Fuses two usually separate ideas: world action models give physical foresight but no compositional reuse, while code-based agents give reuse but no predictive grounding. AW-0 models code, future video and actions as three coupled streams in one mixture-of-transformers.
  • Code plays a dual role, structuring multi-step prediction inside the world model and storing skills as parameterized routines that can be reused later.
  • Introduces an automated curation pipeline that labels robot and human videos with aligned programs and consolidates recurring routines into a skill library.
  • Improves over VLA and WAM baselines on RoboTwin 2.0 and real bimanual tasks, with gains in language following, semantic reasoning and robustness.
Show abstract
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.
#10 h=21

PhysiCalWorld: In-Context Physics Calibration for World Models

2026-10-21 cs.ROcs.AIcs.CV 👤 Nima Fazeli (h=21)
Yunhao Luo, Wentao He, Yinpei Dai, Hyelin Nam, Hongze Fu
Core Contributions
  • Diagnoses why world models drift on novel scenes: mass distribution, joint damping and friction are not visible in a single frame, so predictions regress to the training prior.
  • Calibrates by in-context prompting: a short exploratory interaction clip is encoded once into a compact physics-descriptor token that conditions later predictions, with no fine-tuning or analytical physics model.
  • Trains end to end with the latent prediction loss alone, with no physics labels, yet the descriptors form a physically structured latent space that generalizes beyond the training ranges.
  • Improves rollout accuracy and visual planning success across four environments, and since the token is extracted once, the inference overhead is negligible.
Show abstract
Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that determine transition dynamics, such as an object's mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene's actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.
#18 h=16

Relational Geometry Forcing for World Action Models

2026-10-21 cs.RO 👤 Jian-Hua Yang (h=16)
Yuan Xu, Yixiang Chen, Qisen Ma, Jiabing Yang, Peiyan Li
Core Contributions
  • Finds that the common approach of pointwise alignment to a 3D foundation model helps video features but degrades action learning, because dense geometric targets over-constrain compact action features.
  • Relational Geometry Forcing aligns pairwise feature relations instead, with three terms: intra-frame, cross-view, and action-temporal relations that tie action dynamics to scene geometry evolution.
  • It is applied only at training time, with no extra geometry branch or inference module, so deployed cost is unchanged.
  • Gives consistent gains across diverse manipulation scenarios and WAM paradigms in simulation and the real world.
Show abstract
World Action Models (WAMs) have shown strong potential for robot manipulation by jointly learning visual dynamics and executable actions. However, their video and action objectives do not explicitly constrain the 3D geometry that is crucial for precise manipulation. Recently, geometry-aware WAMs introduce depth or 4D prediction branches, whereas representation alignment methods supervise intermediate WAM features with pretrained 3D foundation models. Yet we find that such pointwise alignment benefits the video features but degrades action learning, since dense geometric feature targets overly restrict compact action features. To address this limitation, we propose Relational Geometry Forcing (RGF), which transfers geometric priors from a frozen 3D foundation model by aligning pairwise feature relations instead of pointwise features. Specifically, RGF applies three complementary relational objectives, where intra-frame and cross-view relations transfer scene geometry and multi-view correspondence to the video features, while action-temporal relations couple action dynamics to geometric scene evolution. Applied only during training, RGF requires no explicit geometry branch or extra inference module. Extensive experiments in simulation and the real world demonstrate the effectiveness of Relational Geometry Forcing, showing consistent gains across diverse manipulation scenarios and WAM paradigms.
#25 h=10

CW-WAM: A Structured Contact-Wrench World Action Model for Force-Sensitive Manipulation

2026-10-21 cs.RO 👤 Lu-Qi Wang (h=10)
Houzhu Ding, Xinda Qi, Luqi Wang, Ming Yang
Core Contributions
  • Argues that force-sensitive tasks need contact information vision cannot supply, and makes contact wrench a co-predicted modality alongside video and motion.
  • A pair-causal attention mechanism structures motion-wrench dependencies, inspired by hybrid motion-force control, and a long fixed-length wrench memory retains interaction history.
  • Predicted wrench chunks serve as force references for a 20 Hz outer-loop admittance controller, bridging slow model queries and fast force feedback.
  • Insertion success rises from 25% to 70% for square pegs (1 mm clearance per side) and from 40% to 95% for power adapters, and force-reference RMSE drops from 4.8 to 2.7 N for insertion and 2.0 to 1.3 N for wiping.
Show abstract
Force-sensitive manipulation requires contact and load information that is not provided explicitly by visual observations. We present CW-WAM, a world action model that jointly predicts video, motion, and contact wrench. Inspired by hybrid motion-force control, it contains a unified interaction module that treats both motion and contact wrench as modeling targets and predicts time-aligned motion-wrench chunks as control references. A pair-causal attention mechanism structures the motion-wrench dependencies, while a long fixed-length wrench memory retains past interaction information. The predicted wrench chunk acts as a force reference for a 20-Hz outer-loop admittance controller within a slow-fast control system that maintains force feedback and smooth motion between model queries. CW-WAM improves contact prediction and increases the insertion success rate from 25% to 70% for square pegs with 1-mm clearance per side and from 40% to 95% for power adapters, compared with WAM without wrench input. It also improves lifting-height accuracy across different loads. Our framework with an admittance controller reduces force-reference RMSE from 4.8 to 2.7 N for peg insertion and from 2.0 to 1.3 N for wiping, respectively, with smaller force fluctuations during successful execution. Project website: https://cw-wam-project.github.io/
#30 h=9

DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

2026-10-21 cs.ROcs.CV 👤 Jian-Hua Sun (h=9)
Xiujian Liang, Liuyuan Na, Zehao Du, Jiacheng Xu, Weixi Song
Core Contributions
  • Builds on V-JEPA 2.1 with a dual-stream RGB-depth predictor, using depth both as an observation and to ground both modalities in metric 3D space, summarized in a shared latent state called DepState.
  • Modality-private features are retained for decoding, and ray-conditioned readouts predict local features without target depth at inference, so depth need not be available at test time for prediction.
  • After an action-conditioned transition the same decoders predict future RGB and depth features, with robot-state readouts, object-trajectory supervision and forward-kinematics joint-TCP consistency shaping the state.
  • Ablations credit depth content, metric coordinates and future-state matching, with additional control gains from FK regularization and stronger RGB-encoder transfer after RGB-D adaptation.
Show abstract
Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions. Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes. Guided by this intuition, we introduce \textbf{DepState-JEPA}, a dual-stream RGB--Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, \textbf{DepState}. Built on V-JEPA~2.1, the framework uses depth both as a complementary observation stream and to ground both modalities in metric 3D space, aggregating shared information into DepState while retaining modality-private features for decoding. Ray-conditioned readouts predict local features without target depth; after an action-conditioned transition, the same decoders predict future RGB and depth features using the predicted state and fixed initial private features. Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint--TCP consistency. Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success. Within-model ablations identify contributions from depth content, metric coordinates, and future-state matching, with additional control gains from FK regularization.
VLA Post-Training, Uncertainty & Online Learning
#3 h=46

Asymmetric On-Policy Distillation Learns In-Context Exploration

2026-10-21 cs.ROcs.AIcs.LG 👤 Roozbeh Mottaghi (h=46)
Sriyash Poddar, Yanda Bao, Jacob Krantz, Matthew Chang, Xavier Puig
Core Contributions
  • Targets partial observability: imitation of a privileged expert averages actions over hidden states and never explores, while RL is too sample-hungry. ASTEROID labels the student's own on-policy history with the privileged expert's actions so exploration emerges from supervision.
  • The key trick is asymmetry: the expert never needs to explore, yet a history-conditioned student trained on its own contexts learns to search while uncertain and exploit once the hidden state is revealed, behaving like posterior sampling.
  • Across eleven environments (discrete exploration, manipulation, navigation, Procgen) it reaches about 2x the return under fixed exploration budgets with about 100x fewer interactions than RL baselines, and beats frontier models at exploration.
  • Transfers from simulation to vision-denied real-robot manipulation, showing that in-context exploration can be trained cheaply and then used on hardware.
Show abstract
Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation averages expert actions across hidden states and fails to learn exploration. In this work, we propose ASTEROID, a simple and effective student--teacher supervised learning framework that can learn test-time exploration. Although a privileged expert never needs to explore, we demonstrate that a history-conditioned student can learn to do so when trained with on-policy contexts: the student first collects on-policy context under partial observability, and the privileged expert provides action labels conditioned on that context. By distilling expert actions directly conditioned on the student's on-policy history, the student learns to systematically resolve partial observability, i.e., search while uncertain and exploit once the hidden state is revealed. This enables efficient test-time exploration (similar to Bayesian posterior sampling) without incurring the impractical training-data costs of reinforcement learning. Across a diverse range of eleven different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments, ASTEROID achieves ~2x higher returns under fixed exploration budgets, uses ~100x fewer interactions than RL baselines, and outperforms frontier models at exploration. Finally, our method transfers policies trained in simulation to vision-denied real-world robot manipulation, exhibiting efficient exploration and increasing success by +40% over standard distillation. Overall, it provides a simple yet surprisingly effective framework to learn efficient test-time exploration.
#6 h=32

Post-hoc Epistemic Uncertainty for Vision-Language-Action Policies

2026-10-21 cs.ROcs.LGstat.ML 👤 T. Neumuth (h=32)
Johannes Keller, Albrecht M. Bloße, Adrian Lindenmeyer, Thomas Neumuth, Daniel Schneider
Core Contributions
  • Replaces costly policy ensembles for epistemic uncertainty with post-hoc kernel-variance heads: a Gaussian-process conditional variance computed from the embeddings of a single trained policy, with no gradient updates and the policy left untouched.
  • On SmolVLA with LIBERO-10 it matches a two-policy ensemble in correlation with task success and in active-learning demonstration selection, while cutting uncertainty-scoring time by up to about 50x.
  • Shows that the standard LIBERO-10 protocol mostly rewards detecting differences in coverage between tasks, so controlled perturbations are proposed as a better test of within-task uncertainty.
  • Modality-specific heads flag unfamiliar cameras, instructions or states, while fused heads catch unfamiliar combinations of individually familiar inputs, which is what a deployed system needs to decide when to ask for help.
Show abstract
Vision-language-action (VLA) policies need to recognize gaps in their training knowledge to request useful demonstrations or assistance in new situations, especially when failures are costly. Disagreement within an ensemble of independently trained policies can guide these decisions, but requires training and running multiple policies. We propose post-hoc kernel-variance heads that estimate epistemic uncertainty from the embeddings of a single policy. Each head computes the conditional variance of a Gaussian process conditioned on embeddings from the training demonstrations. Fitting requires no regression targets or gradient updates and leaves the policy unchanged. In experiments with SmolVLA on LIBERO-10, the kernel variance heads perform competitively with two-policy ensemble scores in task-level correlation with task success, in demonstration acquisition during active learning, and in referral to a stronger supervisor policy as a proxy for human assistance. For SmolVLA, the head on the vision-language backbone reduces uncertainty-scoring time by up to approximately a factor of 50 compared with ensemble scores, while also substantially reducing setup costs. We also show that the established evaluation protocol on LIBERO-10 primarily rewards recognition of coverage differences between tasks, while execution variability weakens assessment of within-task rankings. Thus, controlled perturbations provide a complementary evaluation: modality-specific heads identify unfamiliar camera, instruction, or state inputs, while fused heads can detect unfamiliar combinations of individually familiar inputs. With a supervisor proxy trained on the evaluated perturbation families, the decoder head improves within-task referral over ensemble disagreement. These findings show that useful uncertainty estimates for adaptive robot learning can be obtained from a single policy.
#13 h=20

Training in Deployment: Human-in-the-loop Online Post-Training for Robot Manipulation

2026-10-21 cs.RO 👤 Xia-Tian Zhu (h=20)
Chun Gu, Xiaofei Wei, Yiming Chen, Bozhou Zhang, Juntong Zhang
Core Contributions
  • Targets the mismatch between offline teleoperated demonstrations and the contact-rich recovery states a deployed policy actually visits; offline post-training adapts slowly and online RL needs fragile critics.
  • TIDE uses provenance-conditioned supervised learning, updating the policy online from human interventions and policy rollouts collected in deployment, with no learned critic, reward model or action scorer.
  • On four real bimanual tasks it improves milestone-normalized progress and full-task completion over the warm-start policy and over matched extra offline demonstrations.
  • Each online post-training session finishes in under an hour with asynchronous learning and checkpointing, making adaptation-in-deployment practical.
Show abstract
Large pretrained robot foundation models provide strong priors for manipulation, yet adapting them to target tasks with offline teleoperated demonstrations often fails to cover the contact-rich, recovery, and precision-critical states encountered during deployment. Existing methods mitigate this issue by augmenting offline demonstrations with human-in-the-loop data or online reinforcement learning. Despite this, offline post-training adapts slowly because it cannot immediately absorb deployment experience, while online RL typically relies on value functions or auxiliary actor-critic models, increasing both training complexity and instability. We present TIDE (Training In DEployment), a human-in-the-loop online post-training system. TIDE introduces a provenance-conditioned supervised learning framework that updates the policy online from human interventions and policy-generated rollouts collected during deployment, without a learned critic, reward model, or action scorer. Across four real-world bimanual manipulation tasks, TIDE improves milestone-normalized task progress and binary full-task completion over the warm-start policy and matched additional offline demonstrations, and performs comparably to the fixed-policy collection baseline while making updated checkpoints available during deployment. Each task-specific online post-training session, from the start of deployment rollouts until the final checkpoint is available, completes in less than one hour while learning and checkpoint handling run asynchronously.
#15 h=18

Learning to Absorb Corrections: Critic-Weighted Residual Flow for Vision-Based Insertion

2026-10-21 cs.RO 👤 Jin-Rong Yang (h=18)
Yiqing Yuan, Zhi Li, Jinrong Yang, Mingshan Sun, Mei Yu Zhi
Core Contributions
  • Tackles precise vision-based insertion by letting an Action Chunking Transformer absorb successful executions while a conditional residual flow policy learns corrections relative to the updated prior.
  • A chunk-level critic scores final composed actions and weights residual candidates for flow matching, but is used only in training, so deployment needs just RGB and one residual sample per chunk.
  • Reaches a five-task mean success of 66% after 20,000 post-initialization steps, 18 points above the strongest baseline, across plug, USB and key insertion, cable routing and needle threading.
  • With the residual disabled, the base ACT improves from 16% to 76% mean success, showing the online loop also improves the prior rather than only patching it.
Show abstract
Vision-based insertion requires correcting alignment errors while retaining useful behavior learned from demonstrations. We present ACT--Flow, an online learning framework that couples an adaptive Action Chunking Transformer (ACT) with a conditional residual flow policy. Following anchor-guided warmup, ACT absorbs successful executed actions through behavioral cloning, while residual targets are recomputed relative to the updated prior. A chunk-level critic learns from successful and failed interactions and evaluates final composed actions. Its estimates weight generated residual candidates for flow matching, complemented by supervision from successful real executions. Candidate scoring is confined to training; deployment uses RGB observations and samples one residual per action chunk. We evaluate the framework on power-plug, USB, and key insertion, together with cable routing and needle threading. Results show a five-task mean success rate of 66% at 20,000 post-initialization interaction steps, exceeding the strongest baseline by 18 percentage points at this checkpoint. Evaluations at matched checkpoints with the residual disabled show ACT's mean success increasing from 16% to 76%. Component ablations further support the complementary roles of behavior absorption, value weighting, and successful-action supervision.
#20 h=12

SimPT: An Automated Simulation Framework for Intervention-Based VLA Post-Training

2026-10-21 cs.RO 👤 Zi-Peng Ye (h=12)
Caicheng Wang, Zipeng Ye, Zhexuan Zhou, Zilong Huang, Xi Zhang
Core Contributions
  • Removes the human and the real robot from algorithm iteration for intervention-based VLA post-training by building the whole loop in simulation: rollout, corrective data collection, training and standalone evaluation.
  • An Agent-Compiled Progress Graph tracks task progress from privileged simulator state and keeps stable snapshots, so a CuRobo expert recovers from the most advanced feasible stage. This cuts recovery attempts by 75.0% and raises recovery success from 93.4% to 98.9%.
  • Compares RECAP, HG-DAgger, DPO+SFT and FlowPRO on seven RoboTwin tasks, then develops RECAP-CF-LPF with cross-fitting and low-pass filtered value predictions (76.50% to 79.36% in simulation).
  • The simulation finding transfers: real-robot Ethernet cable insertion rises from 72% to 96% over 25 scenarios.
Show abstract
Human-in-the-loop (HIL) post-training improves vision-language-action (VLA) policies through expert corrections, but costly human supervision and repeated real-robot evaluation limit algorithm exploration and development. We introduce SimPT (Simulation-based Post-Training), a simulation framework that automates intervention-based VLA post-training by integrating policy rollouts, corrective data collection, policy training, and standalone evaluation. To automatically determine intervention timing and recovery strategies, we develop an Agent-Compiled Progress Graph that tracks task progress using privileged simulator state and retains stable snapshots during policy execution. After a rollout fails, a CuRobo-based expert attempts recovery from these snapshots, starting at more advanced task stages and falling back to earlier ones as needed to generate corrective trajectories. We implement and compare four post-training methods, RECAP, HG-DAgger, DPO+SFT, and FlowPRO, across seven RoboTwin manipulation tasks. Relative to a fixed-interval snapshot baseline, the Progress Graph reduces recovery attempts by 75.0% and increases recovery success from 93.4% to 98.9%. Using SimPT, we develop RECAP-CF-LPF, a refinement of RECAP that combines cross-fitting and low-pass filtering of value predictions. This refinement increases the overall success rate from 76.50% to 79.36% in held-out simulation trials. In real-robot HIL post-training, it increases the success rate from 72% to 96% across 25 scenarios of an Ethernet cable-insertion task. These results demonstrate the utility of SimPT for exploring and developing VLA post-training algorithms.
#21 h=11

FlowVLA: Learning Flow-Aware Physical Completion for Quantitative Robotic Pouring

2026-10-21 cs.RO 👤 Tu-Tian Tang (h=11)
Liwen Yang, Yutian Zhang, Jiaqi Zhai, Li Yang, Tutian Tang
Core Contributions
  • Names a concrete failure of whole-task VLAs: during steady pouring, image and pose barely change while mass accumulates, so the decision to return is ambiguous.
  • FlowVLA separates motor generation (a shared π0.5 action expert) from physical-completion estimation, fusing vision, joints, wrist wrench and a recurrent force belief into a transferred-mass estimate that triggers return.
  • Achieves 13.1 g goal-balanced MAE, lower than all tested VLA baselines, with 53.7% of attempts both completing and landing within ±10% of requested mass.
  • Interpolates to held-out 150/250 g targets without retraining and is checked on three further container-fluid settings, supporting force-aware completion as a general VLA module.
Show abstract
During steady pouring, images and robot pose may remain nearly unchanged while transferred mass accumulates, making the return decision ambiguous for a whole-task vision-language-action (VLA) policy. We present FlowVLA, which separates motor generation from physical-completion estimation. A shared $π_{0.5}$ action expert learns approach, pour, and return with boundary-aware supervision. A target-independent estimator fuses visual, joint, wrist-wrench, and recurrent force-belief histories to maintain an estimate of transferred mass. The controller initiates return when this estimate plus predicted residual outflow reaches the requested mass. In coffee-pot/water experiments, FlowVLA achieves 13.1 g goal-balanced mean absolute error (MacroMAE), lower than all tested VLA baselines. Across all attempts, 53.7% both complete the task and finish within +/-10% of the requested mass. The same model interpolates to held-out 150/250-g targets without retraining. Independently trained models further evaluate the formulation in three additional container-fluid settings.
Agentic Planning & Embodied Reasoning
#2 h=53

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

2026-10-21 cs.ROcs.AIcs.CVcs.MAcs.MM 👤 M. Shou (h=53)
Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An
Core Contributions
  • Proposes PAPAV (Perceive, Anticipate, Plan, Act, Verify) as a function-level lens that lets computer-use agents, robotic systems and embodied agents be compared without arguing about architectures.
  • Argues that moving from screens to bodies changes the constraints (partial observability, unverifiable outcomes, irreversibility), so Anticipate and Verify matter far more for robots than for GUI agents.
  • Audits 64 benchmarks and finds evaluation concentrated on Act: only 5 assess Anticipate and 3 assess Verify, so task success hides the capabilities most responsible for reliability.
  • Useful as a checklist for benchmark designers: it names which capability a new benchmark is actually measuring rather than reporting success alone.
Show abstract
Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics. We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback. This raises a central question: what changes when a multimodal agent moves from computer-use to robot-use? We introduce PAPAV, a unified capability-centric framework of five recurring functions: Perceive the current state, Anticipate action effects, Plan a feasible course, Act through an interface or body, and Verify the outcome. Defined by function rather than architecture, these capabilities let PAPAV identify what all three share and then compare MMEAs with MMAs and with RSs. Five physical constraints, from partial observability to unverifiable outcomes, leave fewer choices fixed in advance and less room to reverse errors. Across 64 benchmarks, explicit evaluation centers on Act; only 5 assess Anticipate and 3 assess Verify, leaving both largely hidden behind task success. These constraints also frame the open challenges we identify. PAPAV therefore provides a common basis for designing reliable physical agents and measuring progress across all five capabilities rather than task success alone.
#11 h=21

Toward Recursive Self-Improvement in Physical AI

2026-10-21 cs.ROcs.AI 👤 Ziyang Luo (h=21)
Meng Luo, Zechen Bai, Shanqing Xu, Yanlin Li, Kaixin Li
Core Contributions
  • A position paper that asks what recursive self-improvement, where a system improves its own improvement process, means for physical AI, where trials are costly, observations partial and mistakes sometimes irreversible.
  • Traces pathways from improving robot capabilities to improving the processes that produce them, and states when gains carry forward: feedback must address the remaining learning need, the receiving component must be able to use it, and the gain must survive task or hardware change.
  • Finds existing physical studies, mostly in manipulation, show behavior improvement, benefits to later learning and procedure improvement, but none yet isolate gains attributable to recursion.
  • Distills evaluation principles and research directions, giving the community a concrete test for claims of self-improving robots.
Show abstract
Recursive self-improvement (RSI) describes a system that improves not only its behavior but also the process by which it improves. Recent digital agents test this idea by rewriting their own code and checking each version with executable tests. Bringing these mechanisms into the physical world requires learning from partial observations through costly trials, while accounting for changing operating conditions and potentially irreversible consequences. We position RSI within robot learning and trace pathways from improving robot capabilities to improving the processes that generate them. We examine how physical experience can strengthen these improvement mechanisms and when the resulting gains can support further self-improvement of the mechanisms themselves. Gains carry forward only when feedback addresses remaining learning needs, the receiving component can use the change, its value survives changes in tasks or hardware, and continued practice remains safe and affordable. Existing studies, concentrated in manipulation, support behavioral improvement, benefits to later learning, and procedure improvement. Some systems also change the improver's knowledge or tools, but the physical studies reviewed here have yet to isolate gains attributable to recursion. We synthesize these findings into evaluation principles and research directions for machines that, trial by trial, safely improve the way they improve.
#12 h=20

CausGround: Causal Interaction Grounding for Adaptive Embodied Planning

2026-10-21 cs.RO 👤 Zhong-Yuan Wang (h=20)
Yijun Shen, Diqi He, Jinbang Guo, Wanyue Zhang, Jiaming Mi
Core Contributions
  • Addresses the fact that MLLMs do not know environment-specific action preconditions and effects, and that existing fixes need costly retraining or reuse unstructured experience poorly.
  • CausGround distills interaction experience into reusable, non-parametric action-centered causal knowledge over a factorized world state of object attributes and relations.
  • At planning time it reasons backward over preconditions and forward over effects, so the MLLM plans against grounded dependencies instead of guessing.
  • Raises VirtualHome task success from 24.9% to 53.3%, RoboCasa closed-loop planning from 43.0% to 57.0% and SWITCH interaction accuracy from 25.8% to 31.9%, with real-robot demonstrations.
Show abstract
Multimodal large language models (MLLMs) show promising reasoning capabilities, but reliable physical interaction requires planning that accounts for when actions are executable and how they change the world. These preconditions and effects are environment-dependent and often not captured by MLLMs, making adaptation challenging. Yet existing approaches either require costly environment-specific training or rely on unstructured interaction experience that is difficult to reuse. Therefore, we introduce CausGround, a non-parametric causal interaction grounding framework that distills interaction experience into reusable action-centered causal knowledge. CausGround identifies action preconditions and effects and organizes them in a factorized world-state representation capturing object attributes and inter-object relations. At planning time, task-relevant causal dependencies are grounded in current observations to support backward reasoning over preconditions and forward reasoning over effects for action selection and multi-step planning. We evaluate CausGround on simulated high-level planning, interaction reasoning, and real-world robot manipulation. On adapted VirtualHome and RoboCasa benchmarks, CausGround improves vision-based high-level planning across MLLM backbones, increasing average task success from 24.9% to 53.3% on VirtualHome and closed-loop planning success from 43.0% to 57.0% on RoboCasa. On SWITCH, average interaction reasoning accuracy increases from 25.8% to 31.9%, indicating improved action selection, recovery, and verification. Moreover, real-world experiments show that CausGround reasons backward from MLLM-interpreted goals to derive executable high-level action sequences for sequential manipulation. These results demonstrate that reusable action preconditions and effects support long-horizon embodied planning across diverse environments without parameter updates.
#17 h=16

TARSphere: A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments

2026-10-21 cs.RO 👤 Zhu-Cun Xue (h=16)
Yicheng Xu, Zhikai Xu, Ziyuan Chen, Yuhang Dong, Tao Zhang
Core Contributions
  • Moves embodied evaluation away from fixed goals toward an evolving world: TARSphere simulates embodied NPCs with constrained dialogue and an event-driven state machine that updates tasks from actions and information exchanged.
  • TARSphere-Harness lets a frozen MLLM act through multi-level executable skills with hierarchical memory and execution feedback, a reusable brain for long-horizon tasks.
  • TARSphere-Bench has 360 episodes over 30 indoor scenes with five paired conditions: perceptual robustness, disruption adaptation, task update, human coordination and intent understanding.
  • Across 8 MLLMs, general-purpose models can beat embodied-tuned ones, task updates and human interaction remain hard, and frequent collisions and falls argue for reporting execution safety alongside success.
Show abstract
Robots must act under physical and social constraints while adapting to changes in their environment and human needs. Existing embodied-agent systems often focus on specified goals or abstract execution into symbolic actions, limiting evaluation in an evolving world. We introduce TARSphere, a robot-oriented full-stack framework for physically and socially grounded long-horizon embodied agents. TARSphere-Infra runs robot policies in a physics simulator with embodied NPCs and constrained dialogue. An event-driven finite-state machine (FSM) uses completed actions and delivered information to update task states and evaluate outcomes. TARSphere-Harness, its embodied brain harness, lets a frozen multimodal large language model (MLLM) act through multi-level executable skills, using observations, hierarchical memory, and execution feedback to revise its decisions. On this foundation, agent-in-the-loop curation aligns task semantics, causal dependencies, and scene geometry to construct TARSphere-Bench: 360 episodes across 30 indoor scenes. Five paired conditions assess perceptual robustness, disruption adaptation, task update, human coordination, and intent understanding for navigate-and-report and fetch-and-deliver tasks. Our evaluation of 8 MLLMs shows that general-purpose models can outperform embodied models, while task updates and human interaction remain difficult. Frequent collisions and falls further motivate evaluating execution safety alongside task success, even with a shared action interface and decision budget.
#28 h=9

From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

2026-10-21 cs.RO 👤 Jun-Ting Dong (h=9)
Tingwen Du, Haoyu Guo, Hanxue Zhang, Xuanxu Wang, Junting Dong
Core Contributions
  • Casts the high-level VLM as a robotic agent with an editable harness that maintains task context, so long-horizon tasks with non-Markovian dependencies become tractable with a fixed VLA executor.
  • SERA self-evolves the harness: environment rollouts are used to propose and evaluate harness revisions, and the selected configuration is frozen before held-out testing to keep evaluation clean.
  • On RoboMME, autonomous harness selection reaches 51.38% overall success against 44.38% without self-evolution.
  • Suggests that improving the prompt-and-memory scaffolding around a VLM is a cheaper route to history-dependent manipulation than retraining the policy.
Show abstract
Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.
Navigation, Perception & Mapping
#4 h=46

Object- and Task-Aware Active Perception for Safe Navigation of Mobile Manipulators in Semi-Static Environments

2026-10-21 cs.RO 👤 Steven L. Waslander (h=46)
Jingxing Qian, Xintong Du, Jun Yang, Siqi Zhou, Steven L. Waslander
Core Contributions
  • Treats stale maps in semi-static environments as a safety issue: the hand camera is steered during navigation to inspect uncertain or potentially changed regions, instead of trusting the last visit's map.
  • Formulates an object- and task-aware next-best-view objective that favors observations near the planned path and around objects likely to have moved, rather than generic information gain.
  • Embeds the perception objective in hierarchical-task MPC so camera viewpoint is adjusted via kinematic redundancy while base trajectory tracking keeps higher priority, so looking never compromises safe motion.
  • Real-world tests show better map consistency, lower task-dependent map entropy and lower navigation risk than passive or non-task-aware baselines.
Show abstract
Mobile manipulators must navigate safely in cluttered, partially unknown environments across logistics, manufacturing, and service applications. Semi-static changes between robot visits can lead to outdated scene knowledge and increased collision risk. We propose an active perception framework that enables a high degree-of-freedom (DoF) mobile manipulator to inspect uncertain or safety-critical regions with its hand camera during navigation. A probabilistic, object- and task-aware next-best-view (NBV) formulation prioritizes observations near the desired path and around uncertain or potentially changing objects. The perception objective is integrated into hierarchical-task model predictive control (HTMPC), exploiting kinematic redundancy for viewpoint adjustment while preserving the higher priority of base-trajectory tracking. The system unifies navigation, map maintenance, and active perception in a closed loop. Real-world tests show improved map consistency, reduced task-dependent map entropy, and lower navigation risk than passive or non-task-aware baselines. Additional simulation and real-world sequences demonstrate responses to controlled semi-static scene changes.
#14 h=18

Reading the Room: Scale-ambiguous Semantic Registration of Sparse VSLAM Maps to Floor Plans in Partially Built Environments

2026-10-21 cs.ROcs.CV 👤 E. Szymańska (h=18)
Bjorn Stragier, Emilia Szymańska, J. Micah Prendergast, Lukas Rosenberger Schmid
Core Contributions
  • Solves a construction-site problem: registering a scale-ambiguous, sparse, semantic VSLAM map to a 2D floor plan when the building is only partly built and visually repetitive. Prior methods handled at most two of these four difficulties.
  • Defines a single robust, differentiable, class-balanced semantic-geometric energy used consistently for initialization, matching and local refinement.
  • Offers two global initializations, one from a single inaccurate user correspondence and one from a cross-modal matching network, each finding candidates that are then refined directly on the energy.
  • On 33 sequences from three active sites it reduces mean absolute trajectory error by 4.5x over the strongest baseline, and 7.6x with an anchor.
Show abstract
Many robotic construction applications, such as progress tracking, require registering a sensor trajectory to a prior 2D floor plan. Visual capture is attractive for its low cost and availability, but aligning a visual simultaneous localization and mapping (VSLAM) map to a floor plan is difficult; VSLAM recovers geometry only up to an unknown similarity transform, the sparse point cloud is cross-modal to the symbolic floor plan, sites are only partially built during capture, and construction sites are visually repetitive, degrading map quality while removing distinctive features. Existing methods address at most two of these challenges. We thus introduce Reading the Room(RR), a global-local approach to register scale-ambiguous, sparse, semantic VSLAM maps to partially built floor plans. Our key idea is to define a single robust, differentiable, class-balanced semantic-geometric energy function that governs all stages of the pipeline. We develop two global initialization schemes, leveraging a single inaccurate user-provided correspondence if provided, or estimating the initialization globally using a cross-modal matching network. Both schemes are designed to find initial candidates with respect to the energy function, which is then directly optimized for local refinement. We evaluate RR on 33 sequences across three active construction sites against classical and learning-based baselines, achieving a 4.5x improvement in mean absolute trajectory error over the strongest baseline in the general unconstrained setting, which increases to 7.6x when an anchor is provided.
#16 h=17

State-Dependent Objective Coordination for End-to-End Balance and Navigation

2026-10-21 cs.RO 👤 Yun-Fan Ren (h=17)
Xuhao Huang, Xitao Xu, Yuxin Zheng, Yunfan Ren, Zheng Gong
Core Contributions
  • Observes that on a two-wheeled inverted pendulum, navigation and balance objectives conflict in a way that depends on pitch, so a fixed reward weighting is wrong in some states.
  • Introduces a pitch-conditioned scalarization coefficient β(s) that shifts weight between stabilization and navigation during PPO training, with no PID/LQR layer or extra hierarchy.
  • Gradient analysis shows persistent directional interference and state-dependent scale differences between the objectives, which motivates the state-dependent weight.
  • Achieves 98% and 95% success on a fixed and a random high-density obstacle map with small attitude excursions, plus zero-shot hardware transfer using only onboard sensing.
Show abstract
Traditional robotic navigation systems separate mapping, planning, and low-level control through cascaded modules, introducing computational overhead and accumulated latency. End-to-end reinforcement learning offers a compact unified policy that maps onboard observations directly to control actions. However, on inherently unstable platforms such as two-wheeled inverted-pendulum robots, navigation progress and obstacle avoidance can conflict with attitude stabilization in a strongly state-dependent manner. We propose a lightweight state-dependent objective coordination method for integrated navigation and control. A pitch-conditioned scalarization coefficient $β(s)\in[0,1]$ adjusts the relative contributions of balance stabilization and navigation objectives during training, without extra control hierarchies or auxiliary attitude-stabilization controllers, such as proportional-integral-derivative (PID) or linear-quadratic regulator (LQR) controllers. To support zero-shot deployment, simulation training models execution delay, action hold, asynchronous and noisy observations, and dynamics uncertainty. Gradient analysis reveals persistent directional interference and state-dependent variation in the objectives' relative scales. Comparisons with alternative reward-weighting, constraint, training, and control-structure strategies show that PPO+$β(s)$, where PPO denotes proximal policy optimization, achieves 98% and 95% success rates with small attitude excursions on the fixed obstacle map and the randomly generated high-obstacle-density map, respectively. Zero-shot hardware experiments further demonstrate bounded attitude behavior and consistent obstacle avoidance using only onboard observations. Training and evaluation code will be made publicly available upon acceptance.
#19 h=13

RangeNav: Task-Aligned LiDAR Range-Image Pretraining for Robot Navigation

2026-10-21 cs.ROcs.LG 👤 César Cadena (h=13)
Samuel Foo Enze, Manthan Patel, Fan Yang, Linus Kramer, Pascal Roth
Core Contributions
  • Addresses two RL obstacles at once: sparse-reward learning from raw 360° LiDAR is sample-inefficient, and 3D point-cloud pretrained encoders are too heavy for massively parallel simulation.
  • RangeNav is a 183k-parameter range-image encoder with anisotropic convolutions and circular padding that respect the extreme aspect ratio and azimuthal seam, pretrained purely on synthetic data and then frozen.
  • Pretraining combines range reconstruction, semantic segmentation and BEV prediction, and the navigation-aligned semantic and BEV terms lift success over reconstruction-only by 32.03 pp in training-like and 23.73 pp in out-of-distribution test environments.
  • Zero-shot deployment on a Unitree B2-W quadruped handles dense crowds, low light and construction zones where limited-FOV depth policies struggle.
Show abstract
Autonomous mobile robots operating in dense crowds and cluttered sites require $360^\circ$ perception to react to surrounding hazards. While panoramic LiDAR offers full-surround coverage, learning representations within reinforcement learning (RL) is sample-inefficient under sparse rewards, and existing 3D pretrained point-cloud encoders are too memory-heavy for massively parallel RL simulation. We present RangeNav, a compact (183k parameters) LiDAR range-image encoder pretrained entirely on synthetic data. RangeNav uses anisotropic convolutions and circular padding to handle the extreme aspect ratio and azimuthal seam of range images. We pretrain the network using a navigation-aligned objective combining range reconstruction, semantic segmentation, and bird's-eye-view (BEV) prediction, then freeze it for policy learning. In simulation, adding semantic and BEV supervision increases success rates over a reconstruction-only baseline by an average of 32.03 and 23.73 percentage points (pp) in procedurally generated training and out-of-distribution realistic test environments, respectively. Zero-shot real-world deployments on a Unitree B2-W quadruped demonstrate robust navigation through dense crowds, low-light settings, and construction zones where limited-field-of-view depth policies struggle. Code and models will be open-sourced.
#23 h=11

NavBind: Diagnosing Instruction-Action Binding at Route Forks in Navigation VLAs

2026-10-21 cs.RO 👤 Yu-Run Chen (h=11)
Josh Qixuan Sun, Xiaoying Xing, Yurun Chen, Huaiyuan Weng, Wentao Zhu
Core Contributions
  • Argues that navigation success cannot show whether a VLA follows the instruction or leans on visual cues and history, and introduces NavBind to test instruction-action binding directly.
  • Builds 3,252 route-fork contexts from R2R-CE and RxR-CE where the traveled route and agent state are identical but two instructions specify different branches, isolating the instruction's effect.
  • Compares generated action sequences, likelihoods of reference actions and shifts in visual attention and attribution under the two instructions.
  • Finds that higher reported navigation success does not imply more faithful instruction following, so benchmark rankings may reward shortcuts.
Show abstract
Vision-language-action (VLA) navigation models translate visual observations and instructions into actions. Navigation success assesses whether an agent reaches the goal. However, reaching the goal successfully does not establish whether the model genuinely follows the instructions or relies on visual cues or navigation history. We introduce NavBind to diagnose instruction-action binding: whether changing the instruction influences the probabilistic distribution of the actions, with the navigation context fixed. We construct 3,252 route-fork contexts by recombining route-aligned instruction segments from R2R-CE and RxR-CE. Each context pairs full instructions that share the description of the route already traveled but specify different branches. Both are evaluated under the same agent state and navigation history. We compare generated local action sequences, the likelihoods of fixed reference action sequences under both instructions, and changes in visual attention and attribution. Across the evaluated models, higher reported navigation success does not imply that the model more faithfully follows the instructions.
#27 h=10

AirPlayground: Benchmarking Aerial Navigation across Diverse Reconstructed Environments

2026-10-21 cs.RO 👤 Ming-Sheng Li (h=10)
Rangya Zhang, Yujing Shang, Zhuoyuan Yu, Mingsheng Li, Qingsong Yan
Core Contributions
  • Fills a gap in aerial navigation resources, which lacked both dense flight demonstrations and broad environments for closed-loop evaluation.
  • Builds 94 photorealistic 3D Gaussian Splatting environments with aligned collision geometry, and an automated expert pipeline yielding 94K demonstrations and 46.5M synchronized 50 Hz samples.
  • Benchmarks seven baselines across four policy paradigms in three settings; scaling ACT training from 5 to 70 scenes raises median success from 30.6% to 41.2% on seen and 16.8% to 29.4% on unseen environments.
  • Supports analysis of how offline prediction error relates to closed-loop success, plus an extensible API for new policies and protocols.
Show abstract
Large-scale aerial resources provide a foundation for learning navigation across diverse environments. However, existing resources remain limited in providing (1) densely sampled flight demonstrations for learning different policy formulations and (2) broad environment coverage for standardized closed-loop evaluation. To address these limitations, we introduce AirPlayground, a unified benchmark for training and evaluating aerial navigation policies across diverse environments. We first construct 94 photorealistic 3D Gaussian Splatting (3DGS) environments spanning indoor, urban outdoor, and vegetation settings, with aligned collision geometry. Building on these environments, we develop a scene-adaptive pipeline for automated expert flight collection, yielding 94K demonstrations and 46.5M synchronized observation-state-action samples at 50 Hz. We further provide an extensible API that supports training different policy formulations on this shared corpus and evaluating them under unified protocols in the corresponding environments. We benchmark seven baselines spanning four policy paradigms across three navigation settings in seen and unseen environments, and further examine scaling behavior with ACT. Scaling ACT's training corpus from 5 to 70 scenes raises median success rates on seen and unseen environments from 30.6% and 16.8% to 41.2% and 29.4%, respectively. Through this shared training and evaluation framework, AirPlayground supports analysis of how offline prediction errors relate to closed-loop outcomes. Our analyses highlight the roles of prediction errors, execution protocols, and scene interactions in closed-loop performance.
#29 h=9

Integrating Flow Matching and Model Predictive Control for Safe Robot Navigation

2026-10-21 cs.ROeess.SY 👤 João Carvalho (h=9)
Philipp Holzmann, Maik Pfefferkorn, Joao Carvalho, Ali Younes, An Thai Le
Core Contributions
  • Pairs the diversity of flow-matching path generation with constraint-aware execution, since learned planners alone cannot guarantee collision avoidance on deployment.
  • A model-predictive path-following controller adapts motion online to state and input constraints using updated local sensor information.
  • An event-triggered replanning scheme feeds the controller's solutions back into the next path generation, helping with obstacles unseen at planning time.
  • Simulated mobile-robot navigation shows robust performance in cluttered scenes, a template for adding safety layers to generative planners.
Show abstract
Learning-based path planners based on diffusion and flow matching can generate diverse trajectories from demonstrations but typically lack mechanisms for constraint satisfaction and collision avoidance during deployment. This work proposes a framework that combines flow-matching-based path planning with model predictive path-following control to integrate data-driven path generation with real-time, constraint-aware execution. The flow-matching model captures multimodal path distributions, while the predictive controller adapts motion online to satisfy state and input constraints and avoid obstacles using updated local sensor information. An event-triggered re-planning scheme incorporates predictive control solutions into subsequent path generation, promoting collision-free navigation in environments with previously unseen obstacles. Simulation results for mobile robot navigation demonstrate robust performance across cluttered environments. The framework provides a systematic approach to combining flow matching with predictive control for adaptive, data-driven navigation.
Manipulation Control, Retargeting & Scene Reasoning
#5 h=35

Payload generalization on feedforward control actions for trajectory tracking in robots with rigid and elastic joints

2026-10-21 cs.ROeess.SY 👤 G. Grioli (h=35)
Giorgia Pazzi, Giorgio Simonini, Michele Pierallini, Paolo Salaris, Giorgio Grioli
Core Contributions
  • Addresses the practical problem that learned feedforward compensation breaks when the payload changes, especially with elastic joints where analytical models are poor.
  • Derives the minimal number of example payloads needed to generalize, which turns data collection from guesswork into a bounded calibration procedure.
  • Given only a new payload's parameters, it produces the feedforward action for the same trajectory without a dynamic model and independently of the underlying controller, so it can sit on top of existing industrial loops.
  • Validated on a standard Franka Panda and a Panda with an elastic wrist, covering both rigid and compliant joint cases.
Show abstract
Industries employ robots for repetitive tasks due to their high precision and performance, typically achieved through feedforward control actions based on an accurate robot description. However, an exact dynamic model can be difficult to obtain, especially for robots with elastic elements or when the payload changes. Learning-based feedforward approaches are promising for trajectory tracking under model uncertainties, but typically face generalization issues when task parameters vary. This paper proposes a novel method to generalize the feedforward control action with respect to payload parameters, independently of the underlying controller. Firstly, the minimal number of example objects needed for payload generalization is derived. Then, an algorithm is proposed to execute the same trajectory with a new payload, given only its parameters, without knowledge of the robot's dynamic model and even in the presence of elastic joints. The effectiveness of the proposed algorithm is validated in simulations and experiments on tracking tasks with different robots and controllers. Two sets of experiments are performed: on a standard Franka Emika Panda robot, and on a Franka Emika Panda robot equipped with an elastic wrist.
#8 h=27

Distance Field Driven Reinforcement Learning for Reactive and Robust Collision Avoidance

2026-10-21 cs.RO 👤 Teresa Vidal-Calleja (h=27)
Taiga Ikeshima, Sheila Sutjipto, Lan Wu, Francisco J. Naranjo-Campos, Taiki Ishigaki
Core Contributions
  • Puts the Euclidean Distance Field directly into the RL observation, giving the policy continuous geometric clearance cues instead of making it infer proximity from raw points.
  • That richer observation allows a streamlined reward for obstacle avoidance that works with or without Artificial Potential Field guidance.
  • Trained only on randomized static obstacles, the policy transfers zero-shot from simulation to a real 7-DoF redundant manipulator and avoids dynamic obstacles online.
  • Suggests distance-field observations are a compact way to shrink the sim-to-real gap for reactive collision avoidance.
Show abstract
Robust and reactive collision avoidance remains a central challenge for deploying Reinforcement Learning (RL) policies on real robotic manipulators, particularly when transferring behaviours learned in simulation to the physical world. This paper presents an RL framework that integrates the Euclidean Distance Field (EDF) directly into the observation space, providing the agent with continuous geometric cues that support stable and responsive avoidance behaviours. By leveraging the rich spatial information encoded in the EDF, the method enables a streamlined reward formulation that achieves effective obstacle avoidance both with and without Artificial Potential Field guidance. The proposed approach is validated on a 7-DoF redundant manipulator, demonstrating zero-shot transfer from simulation to real hardware. Despite being trained exclusively with randomised static objects, the deployed policy generalizes to previously unseen real environments and exhibits robust online avoidance of dynamic obstacles. These results highlight the potential of EDF-based representations to mitigate the sim-to-real gap and support reliable and real-time collision-aware behaviour in complex and changing environments.
#24 h=11

SMR: Learning streaming motion retargeting for humanoids from human semantic

2026-10-21 cs.RO 👤 Chun Gu (h=11)
Chun Gu, Juntong Zhang, Jingyu Li, Li Zhang
Core Contributions
  • Targets two weaknesses of humanoid retargeting: kinematic pipelines ignore dynamics, and earlier physics-supervised networks are bidirectional full-sequence models tied to prescribed kinematic solutions.
  • SMR converts human motion to robot-aware task-space targets, lets physics experts track them in simulation under contact and actuator limits, and trains a strictly causal Transformer on the executed trajectories that pass semantic and physical validity checks.
  • Because the student sees only current and past frames, it streams references to a whole-body controller, enabling real-time teleoperation-style use that offline retargeters cannot support.
  • Semantic-conditioned experts give higher semantic fidelity and physical validity than direct optimization, and the retargeter beats kinematic and physics-refined baselines in closed loop under a common controller.
Show abstract
Retargeting human motion to humanoids requires preserving motion semantics while accounting for the robot's physical constraints. Kinematic pipelines prescribe a robot trajectory through geometric matching, without establishing whether it can be executed under the robot's dynamics. Prior physics-supervised neural methods train bidirectional full-sequence models on physics-refined kinematic trajectories, remaining tied to prescribed kinematic solutions and requiring noncausal context. We propose SMR, a streaming neural motion retargeter learned from semantic-conditioned physics experts. We convert human motion into robot-aware task-space targets for body landmark positions, orientations, and velocities. Physics experts track these targets directly in simulation under contact, dynamics, and actuator constraints. We train a single strictly causal Transformer on their executed state trajectories that complete the motion and pass semantic and physical validity checks. At each incoming frame, the student uses only current and past human observations to generate a robot motion reference for a downstream whole-body controller. Experiments demonstrate that semantic-conditioned physics experts generate motion with higher semantic fidelity and physical validity than direct optimization-based retargeting, while the resulting streaming retargeter achieves improved closed-loop performance over kinematic and physics-refined baselines under a common whole-body controller.
#26 h=10

OMAR: Open-World Multi-Attribute Reasoning for Scene-Conditioned Shelf Organization

2026-10-21 cs.RO 👤 Nils Dengler (h=10)
Benedikt Kreis, Rohit Menon, Benno Wingender, Muhammad Ehsan Ullah, Nils Dengler
Core Contributions
  • Addresses open-ended household organization where the robot gets no instruction, no object vocabulary and no object-receptacle mapping, and must infer the latent organization of a shelf.
  • Builds an instance-level 3D scene graph by having a VLM propose labels and attributes such as content and packaging type, grounded across views with segmentation and depth.
  • An LLM reasons over the graph to rank scene-consistent placements for a novel object, and the ranking then goes through geometric and kinematic feasibility checks.
  • Improves instance mapping and placement coherence over single-view language-grounding and geometry-only baselines, without class-specific retraining.
Show abstract
With the advent of foundation models, robots are increasingly capable of understanding and executing explicitly specified human tasks. Extending these capabilities to open-ended tasks, however, requires robots to infer appropriate actions from the observed scene. Household organization exemplifies this challenge, as robots must often act without detailed instructions, predefined object vocabularies, or known object-receptacle associations. While robots may follow instructions or learn organization preferences from demonstrations, inferring the latent organization of an occasionally changing scene and placing a novel object consistently with it remains challenging. We propose an open-world multi-attribute reasoning pipeline that grounds unknown objects across views and constructs an instance-level 3D scene graph for placement reasoning. First, a vision-language model proposes labels together with attributes such as content and packaging type, which are grounded through segmentation, depth, and multi-view association. Then, an LLM reasons over this representation to infer the latent shelf organization and rank scene-consistent object placements without explicitly user-provided preferences. The resulting placement rankings guide subsequent geometric and kinematic feasibility checks for execution. Our experimental evaluation shows that our approach leads to improved instance mapping and placement coherence over single-view language-grounding and geometry-only baselines, enabling autonomous placement of previously unseen objects without predefined object-receptacle mappings or class-specific retraining.
Design, Assistive Devices & Failure Analysis
#1 h=67

Make it Walk: Computational Design of LEGO Walkers

2026-10-21 cs.RO 👤 Chi-Wing Fu (h=67)
Siqi Li, Fancheng Gao, Chi-Wing Fu, Peng Song
Core Contributions
  • Replaces fixed single-actuator LEGO walking mechanisms with a gear-linkage design in which 3D-printed non-circular gears shape each foot trajectory, so gait shape is a design variable rather than a fixed property of the kit.
  • Gives every leg its own motor and coordinates timing with a simple adaptable controller, which is what lets one build produce multiple gaits instead of one.
  • Co-optimizes morphology and control against a target locomotion and a target appearance, turning toy-style walker design into a computational design problem.
  • Validated on a variety of fabricated walkers with different gaits and looks, making it a useful template for computational design of modular, educational and low-cost legged robots.
Show abstract
LEGO walkers are walking automata constructed from modular LEGO Technic bricks. They exhibit intriguing locomotion and are also highly reconfigurable through the modular Technic bricks, making them widely applicable in education, recreation, and robotics research. However, existing LEGO walkers are designed mostly by reusing existing single-actuator walking mechanisms. Although the locomotion is stable, their fixed walking gaits limit their adaptability and versatility. This paper presents the first computational solution capable of designing LEGO walkers that not only walk stably, but also support a variety of walking gaits. Our idea is twofold. First, we control the foot motion trajectory of each leg mechanically via a specifically-designed gear-linkage mechanism, in which we incorporate a pair of 3D-printed non-circular gears to work with a LEGO Technic mechanism. Second, we drive each leg by an independent motor and synchronize the timings of multiple leg motions via a simple yet adaptable control algorithm, enabling stable and versatile locomotion. Importantly, we take this modeling as a foundation to co-design the morphology and control of the LEGO walkers to meet the target locomotion and appearance. Leveraging our approach, we created a variety of LEGO walkers, demonstrating various walking gaits and appearances, and fabricated some of them to show their locomotion capability in the real world.
#7 h=29

Extending Affordance Segmentation for Wearable Robots to Multi-Object Images

2026-10-21 cs.CVcs.RO 👤 R. Zunino (h=29)
Simone Lugani, Matteo Montani, Paolo Gastaldo, Rodolfo Zunino, Edoardo Ragusa
Core Contributions
  • Tackles a gap for prosthetic grasp pipelines: affordance segmentation works for single objects but fails to pick the intended object when several are present.
  • Adds a laser pointer to the hardware setup as a cheap disambiguation signal, shifting the problem from hard scene understanding to recognizing the pointed object.
  • Retrains the CNNs with a mixed-reality strategy that reuses existing high-quality masks to procedurally generate realistic multi-object images, avoiding new annotation.
  • Improves target disambiguation accuracy by up to 9% over existing solutions while preserving computational cost, which matters on wearable embedded hardware.
Show abstract
This paper proposes a refinement to existing vision-based portable pipelines for the recognition of graspable object parts. Such pipelines can effectively support automatic grasp selection and preshaping in upper-limb prosthetic devices. However, realistic scenarios in which multiple objects are present still represent a challenge. The proposed method exploits a minor modification to the conventional hardware setup and a custom training approach to extend the applicability of vision-based pipelines. A laser pointer is introduced in the setup as an effective mechanism for object disambiguation. Accordingly, a suitable retraining of the deep convolutional neural networks supporting recognition of graspable parts is implemented. A mixed reality strategy allows the reuse of available high-quality segmentation masks to generate realistic images procedurally, extending the range of scenarios supported by the final device. Experimental results confirm that the proposed pipeline improves target disambiguation accuracy by up to 9% compared with existing solutions, while preserving the computational cost.
#22 h=11

Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI

2026-10-21 cs.CVcs.RO 👤 Hang-Rui Hu (h=11)
Han Hu, He Chen
Core Contributions
  • Evaluates seven locally deployable VLMs, including physical-AI models, on two person-centric thermal datasets, since thermal sensing offers low-light and privacy-preserving perception for HRI.
  • Finds models recognize broad activity categories but fail on fine-grained actions like coughing, sneezing and yawning, build inaccurate timelines and hallucinate object actions.
  • Simple thermal preprocessing and extra reasoning give only limited, model-dependent gains, and abstention behavior varies even under explicit instructions.
  • Using paired RGB as a reference localizes the failures to the thermal domain gap, a caution for deploying RGB-trained VLMs on thermal robots.
Show abstract
Thermal cameras enable robots to perceive human activities in darkness while revealing less personally identifiable appearance information and fewer details of the surrounding environment than RGB cameras. As vision-language models (VLMs), originally developed for RGB imagery, are increasingly integrated into embodied systems for scene understanding and high-level planning, thermal sensing offers a promising way to extend activity perception to low-light and privacy-sensitive settings. However, the failure modes of VLMs in thermal human activity understanding for human-robot interaction (HRI) remain underexplored. We fill this gap by evaluating seven locally deployable VLMs, including models developed for physical AI and robotics, on two person-centric thermal datasets. Our diagnostic tasks examine fine-grained action recognition, activity timeline reconstruction, and the ability to withhold unsupported answers. Paired RGB recordings serve as a performance reference, while controlled variations in frame sampling, thermal input presentation, and reasoning are used to assess whether the observed errors can be mitigated. Our results show that models can recognize broad activity categories but struggle with fine-grained actions such as coughing, sneezing, and yawning. They also produce inaccurate event timelines and hallucinate actions involving objects. Simple thermal input preprocessing and additional reasoning offer only limited, model-dependent gains. Abstention behavior also varies across models even under explicit instructions, limiting their reliability for practical robotic use.