Robotics 46
☆ STA: A Split Temporal Adversary Method for Evaluating Robustness of Multimodal RL-based Driving Policies
Multimodal reinforcement learning (MRL) offers a promising paradigm for autonomous driving by jointly leveraging complementary sensor observations to learn effective decision-making policies. However, its effectiveness critically depends on temporal alignment across modalities, making temporal inconsistency a fundamental vulnerability. Even subtle misalignment or latency can corrupt fused representations and ultimately lead to decision failures. Existing studies have explored adversarial attacks to induce cross-modal temporal inconsistencies, thereby enabling systematic evaluation of the temporal robustness of MRL policies. However, existing methods remain limited in generality, realism, and adaptivity. To address these limitations, we propose the Split Temporal Adversary (STA), which formulates multimodal temporal attacks as a unified decision process that jointly optimizes attack timing and modality selection under realistic policy-aware and budget-constrained settings. STA includes two mechanisms: a) a unified multimodal temporal perturbation mechanism that delays observations via historical states, removing the need for modality-specific noise; b) a split multimodal adversarial decision mechanism that jointly optimizes attack timing and modality selection through adaptive optimization while leveraging the victim policy's clean actions to identify sensitive decision patterns. We provide a theoretical analysis of the induced distribution shift and a modality selection principle. Experiments in the CARLA simulator show that STA more than doubles the collision rate of the strongest baseline against PPO while using 17.8\% fewer attack attempts, and that it generalizes to a VLM-based driving policy. These results reveal temporal consistency as a critical vulnerability, highlighting temporal robustness as essential for safe autonomous driving.
comment: 10 pages, 6 figures, submitted for publication
☆ Robotic Reward Models in the Foundation-Model Era
Robotic reward models (RRMs) use foundation models to replace per-task reward engineering with reusable, task-conditioned evaluation. Yet under one name, their outputs answer different questions, from whether a task succeeded to which execution is better. This survey organizes 114 methods into five semantic families by the question their native outputs answer: Outcome, Absolute Progress, Relative, Temporal Value, and Hybrid. Within this taxonomy, we review how model design determines what an evaluator can judge, how raw experience becomes supervision, and how existing evaluation practice tests the result. The review exposes a gap: agreement with demonstration order and downstream policy gains do not show whether a reward tracks task fulfillment. We therefore introduce RRM-Bench-1K, 1,117 real-robot videos of closed-loop executions with task-state and camera-evidence labels, and four contrast suites that separate the credited condition from its usual cues. A unified study of 27 evaluators shows that the strongest final scores rank whole executions well enough for selection and filtering. Dense credit, however, follows the progression of an execution more closely than its task state, and no evaluator signals when the decisive view is missing. How much fidelity a reward model needs therefore depends on the decision it drives. We outline directions from real-failure supervision to metrics that separate task state from chronology, and open-source RRM-Hub, a toolbox in which 14 existing RRMs can be readily applied and new evaluators tested on the same capabilities.
comment: Project page: https://ethanliang99.github.io/awesome-robot-reward-model
☆ Selective Sensor Acquisition for Multimodal Place Recognition
Multimodal Place Recognition (PR) can improve robustness, but most systems rigidly read and process every sensor for every query, even when the primary sensor is sufficient, with implications for energy consumption, second sensor update rate, and domain-specific considerations such as passive stealth. We introduce selective, controllable sensor acquisition, where a user specifies a desired second sensor usage rate and the system acquires the additional sensor only when the primary sensor's estimated performance is sufficiently poor. An online threshold adjustment achieves the requested usage rate by adapting to recent primary sensor performance, directing additional sensing towards queries where fusion is most likely to help without calibration specific to sensor modality, PR method, or environment. The approach requires only a PR method that produces a match score and applies to both single-frame and sequence-based PR. Our system, across multiple datasets, PR descriptors, and sensor combinations, recovers 55% of the available PR performance gain while using the second sensor on only 25% of queries, and 81% while using it on 50%. We further analyse alternative acquisition decision mechanisms and usage-rate control methods, and examine performance when the second sensor is stronger than, comparable to, or weaker than the primary sensor.
comment: 8 pages, 6 figures
☆ AVD: A Multi-Sensor Teleoperated Wheelchair Platform with Safety-Aware Control
Electric wheelchairs predominantly rely on manual joystick interfaces, which present critical barriers to independent mobility for individuals with severe motor and visual impairments. This paper presents AVD (Autonomous Vehicle Drive), a dependable teleoperation architecture for assistive mobility that unifies perception, communication, embedded control, and remote supervision in a single low-cost platform built entirely from commercially available hardware. The central engineering contribution is a safety-oriented design framework: control authority follows a static hierarchy, a manual emergency stop and the occupant's joystick pre-empt remote commands, and the on-board sensing layer has stop-only authority. The compute design partitions primary and safety functions and specifies a heartbeat-driven safe-state transition for the fault cases considered. The sensing stack this framework depends on is characterized against measured data: per-channel acquisition latency, a 2,025-scan LiDAR degradation campaign spanning five target configurations and a control under water mist, airborne dust, and direct aperture occlusion, and attitude measurements across static inclines to 47 degrees. The lowest pooled environmental-condition mean retained 76.6% of the clear-baseline forward-sector yield, and no scan lost the sector completely. A logged vibration-induced inertial dropout occurred while LiDAR and camera logging continued, illustrating complementary channel behavior. The result is a documented prototype and an explicit validation path for a deployable wheelchair implementation.
☆ ACiT: Proprioceptive-Prior Action Chunking with Inverted Transformers
Visuomotor policies condition action prediction on visual and proprioceptive observations. While historical proprioceptive sequences carry rich dynamic priors, current methods treat them as equal to high-dimensional vision, causing the proprioceptive signal to be overwhelmed. To identify what is lost in equal-conditioning fusion, we analyze proprioceptive data and find that strong inter-joint coupling is the key action-determining structure. This finding aligns with the neural sensorimotor loop, in which proprioception first forms a body-centered prior and vision is later calibrated via parietal remapping. Inspired by this, we propose ACiT, Proprioceptive-Prior Action Chunking with Inverted Transformers, which reinforces historical joint-coupling as the primary condition with vision as a later calibration. This design is instantiated through three modules: Inverted State Encoding builds a proprioceptive prior via per-joint tokens and inter-joint self-attention. State-Visual Grounding lets joint tokens query visual evidence, making vision goal- and calibration-only. Inverted Action Chunking decodes per-joint trajectories independently. ACiT improves success rates on most LIBERO, RoboTwin 2.0, and real-world tasks, with the largest gains on those that require dynamics and coordination. The same conditioning design transferred to the VLA backbone pi_0-ACiT yields significant gains with better execution efficiency.
☆ TaF-Foundation: Wrench-Grounded Cross-Sensor Tactile Representations
Jiaming Jiang, Yuzhe Huang, Baokun Yang, Hao Liang, Li Gao, Haoran Guo, Pei Lin, Shengcheng Luo, Chenxi Xiao, Wanlin Li, Ziyuan Jiao
Tactile sensing is essential for contact-rich manipulation, yet heterogeneous sensing mechanisms and observation formats hinder cross-hardware transfer. Since physical contact is governed by Newtonian mechanics, we ground tactile representations in six-dimensional wrenches, establishing a common physical reference across sensing mechanisms. Specifically, we pair tactile observations from all sensors with wrench measurements, enabling shared physical supervision without requiring direct tactile-to-tactile pairing between sensors. To support learning at scale, we develop a dedicated acquisition platform and collect over 100 million paired tactile observations and wrench measurements from 24 sensors spanning four families of sensing principles. Building on this dataset, our framework combines sensor-specific input adaptation with a shared tactile encoder. By learning to reconstruct wrenches, the encoder maps observations from different sensors into a shared, continuous feature space grounded in contact physics. To leverage these representations for manipulation, we introduce TaF-Policy, which integrates tactile features with visual observations and robot states to generate action chunks. We evaluate the learned representations through wrench estimation, cross-sensor retrieval, few-shot adaptation to unseen sensors, and robotic manipulation. On the core benchmark, our method reduces single-frame force RMSE by 17.1% and improves cross-sensor retrieval R@1 by 13.11 percentage points over the strongest baseline. In manipulation experiments, TaF-Policy achieves average success rates of 51.33% across three simulated tasks and 46.25% across four real-world tasks, compared with 47.33% and 23.75%, respectively, for the strongest baselines evaluated on the same task sets. Project page: https://mrjiangjm.github.io/TaF-Foundation/.
comment: 17 pages, 6 figures, including supplementary material
☆ SG-ODOM: Event-Driven Proprioceptive State Estimation with Stability-Gated Contact Updates for Legged Robots
Francesco Iotti, Chiyun Noh, Franco Angelini, Manolo Garabini, Ayoung Kim, Maurice Fallon, Frank Dellaert
Reliable proprioceptive state estimation for legged robots fuses inertial sensing with kinematic sensing from leg encoders. The latter assumes a foot is stationary during the stance phase; however, physical foot touchdown does not necessarily provide a stationary reference for odometry, particularly during impact and load transfer. Repeatedly applying kinematic corrections throughout the same contact phase can introduce redundant or unreliable information. We present SG-ODOM, a proprioceptive odometry framework that uses stability-gated contact selection to schedule kinematic updates. Following physical touchdown, a stability gate accepts the contact for state estimation only after the foot reaches a sufficiently stable support condition. The resulting validated touchdown event triggers an estimator correction, while additional timeout updates bound the maximum duration of uninterrupted IMU propagation. The framework is evaluated on multiple legged platforms, both quadrupedal and humanoid robots, against representative state-of-the-art proprioceptive estimators. Across the evaluated sequences, SG-ODOM demonstrates competitive or superior estimation accuracy, reducing Relative Pose Error (RPE) by up to 25% compared to the baseline estimator on outdoor sequences from the GrandTour dataset. Source code: https://github.com/ori-drs/SG-ODOM. Video: https://youtu.be/-583_RyuhqQ
☆ Design of the Secure Smart Scooter with Automatic Lane Detection
The authors present a prototype of an electric scooter that uses computer vision to determine driving lanes and automatically modify vehicle configuration. Scooters steer on sidewalks, bike paths, or car lanes. The government imposes speed limits on the lanes in which electric scooters drive. Featuring computer vision, electric scooters can determine driving lanes and automatically modify vehicle configuration. However, such systems' safety is still a concern, and research into such vehicle systems' safety and security aspects is proceeding. First, we will present our vehicular setup. Then, we will present our implemented scooter with safety constraints. Finally, we present our security prototype for such an electric scooter.
comment: 4 pages, Published in 16th IEEE International Conference on Service Operations and Logistics, and Informatics (IEEE SOLI 2022)
☆ ARMFUL: Learning Whole-Arm Grasp Configurations for Single-Arm Pickup of Large Objects
Many everyday objects, such as storage boxes, baskets, and chairs, are too large or heavy for a gripper to grasp, yet a single arm can still pick them up by wrapping around and supporting them. Such grasps are hard to learn: wrapping motions are difficult to teleoperate, and existing datasets contain little whole-arm pickup data. We introduce WAG (Whole-Arm Grasp Dataset), the first large-scale dataset for single-arm pickup of large objects, with 1.7~million joint configurations across 793 objects. We sample configurations directly in joint space, execute each through squeeze and lift primitives in simulation, and stress-test survivors under active shaking and external forces. To demonstrate the utility of WAG and accomplish the whole-arm pickup task, we develop ARMFUL, a whole-arm pickup pipeline that pairs a diffusion model trained on WAG, generating whole-arm grasps from the object's point cloud, with a motion planner that reaches them collision-free before the same squeeze-and-lift execution. In simulation, ARMFUL reaches 85.4\% lift success on training and 69.9\% on held-out objects, and it lifts the object in 70 of 100 trials on a physical Franka arm across 10 IKEA objects.
☆ Learning and Verifying Progress: Closed-Loop Agentic Planning with Progress-Enhanced Execution for Long-Horizon Vision-Language Navigation
Kangjie Zhou, Sukai Wang, Yuhan Xiao, Qinghang Liu, Zhejia Wen, ChunXuan Jiao, Wei Shan, Guanghui Ren, Maoqing Yao, Chang Liu
Long-horizon vision-language navigation (VLN) requires agents to track instruction progress and recover from local errors before they propagate across subtasks. However, executors trained on expert trajectories encounter policy-induced state shifts during deployment, and their stop signals do not reliably indicate subtask completion. We introduce Learning and Verifying Progress for VLN (LeaP-VLN), an agentic VLN system that couples progress-aware execution with independent progress verification through plan--execute--verify--recover cycles. LeaP-VLN leverages progress both explicitly, through a progress verifier that verifies subtask completion before the agent advances and triggers dual-phase recovery upon route deviation, and implicitly, by co-training the executor on progress reasoning and action prediction to enhance the executor's progress awareness and corrective control. To adapt the executor to this closed-loop interface, we propose a two-stage progress-enhanced curriculum: behavior cloning on expert demonstrations, followed by joint training on demonstrations, expert action corrections, and progress-reasoning supervision. Policy-generated trajectories expose the executor to both intra-subtask drift and inter-subtask state shifts, aligning training with the deviations encountered during closed-loop execution. Extensive experiments on R2R-CE and RxR-CE demonstrate state-of-the-art success rates among monocular methods trained without additional navigation datasets, while real-world experiments further validate the effectiveness of LeaP-VLN in long-horizon navigation.
☆ Instructions Before Actions: Prospective and Retrospective Language Supervision for Driving Vision-Language-Action Data
Driving vision-language-action (VLA) models can accept navigation instructions, yet predicting a driver's trajectory does not establish that a model follows the instruction. We study how driving-language supervision is collected and evaluated, distinguishing prospective instructions issued before an action from retrospective labels written after observing it. We develop ADVLAT-Engine to synchronize spoken instructions, road-scene video, and vehicle trajectories, and compare three navigation apps on 48 replayed routes. On 87 maneuvers announced by both Google Maps and a passenger, the passenger uses more scene-dependent language. We evaluate four open driving VLAs before and after fine-tuning on prospective navigation-app drives. Fine-tuning reduces passenger-drive trajectory error by 23-59%, but instruction effects remain inconsistent. At real intersections, counterfactual instructions reveal stronger responsiveness: fine-tuned Alpamayo~2 Super gains 31 and 18 accuracy points at 15 and 5~m, while Qwen-Drive follows alternative maneuvers more reliably with a discrete command than with a sentence. We additionally collected retrospective annotations from four people for the driving recordings and describe controlled comparisons of annotation content, timing, and downstream learning; those analyses are pending. Our findings separate maneuver prediction from instruction-following and establish a framework for comparing prospective and retrospective supervision.
☆ JEPTERA: A Platform Agnostic JEPA-Assisted Model Predictive Planner for Urban UAV Navigation
Urban navigation for Unmanned Aerial Vehicles (UAVs) requires planning that accounts for environmental geometry, vehicle dynamics, and moving obstacles. This paper presents JEPTERA, a navigation architecture that integrates a Joint-Embedding Predictive Architecture (JEPA), Model Predictive Control (MPC), PID control, and a safety layer. JEPA predicts the UAV's motion response to candidate commands for MPC action selection, while the safety layer checks trajectories against occupied space and predicted obstacle motion. Simulator adapters and platform-specific YAML configurations separate the planning core from simulator interfaces. An ablation study evaluated six configurations across three gym scenarios, totaling 180 runs. The complete architecture achieved the highest aggregate success rate, completing 28 of 30 runs. Additional tests completed one route across three city maps without collisions. Separate integration tests demonstrated operation in Gazebo/ROS 2, MuJoCo, and CARLA without modifying the core planning logic. These results support the feasibility of the integrated architecture for simulated urban UAV navigation and its portability across simulator interfaces.
comment: 6 pages, 3 figures. Submitted to conference
☆ Stability Principle Inherent in Wheel Gait of Planar X-shaped Walker Generated Using Constant Torque Drive and Mechanical Stoppers ICRA 2026
Since the late 19th century when the first walking toys were developed, it has been known that mechanical stoppers at the hip joint are crucial for generating stable passive dynamic walking. Recent research on passive-dynamic and limit-cycle walkers has also confirmed that mechanical stoppers at the hip and knee joints are effective for generating stable walking motion, but theoretical research on how this mechanical constraint enhances the overall gait stability remains insufficient. This paper introduces a planar X-shaped walker equipped with mechanical stoppers at the hip joint, and investigates the effect of the mechanical constraint on the stability of the wheel gait generated by constant torque drive on a downslope. By simply falling forward while using the stoppers to constrain itself to the target impact posture, the robot can generate a highly stable wheel gait. We divide the motion of one step into four phases, derive approximate analytical solutions for the state error transition function matrix in each phase using a linearized model, and analyze the increase or decrease in the state error norm using metrics such as its maximum singular value. Numerical simulations demonstrate that while two phases are unstable in terms of the increase in the state error norm, the remaining two phases are stable, resulting in overall asymptotic stability.
comment: 8 pages, 6 figures. Published in the Proceedings of IEEE ICRA 2026. This author version corrects the author names in one reference; the scientific content remains unchanged
☆ ExtremeDrive: A Real-World Dataset and Benchmark for Critical Driving
Xinhan Li, Yuhong Jiang, Shiyue Zhao, Junzhi Zhang, Chengkun He, Ruihai Ma, Shijie Zhao, Wenxuan Cheng, Jinbo Lv, Tao Liu
Existing autonomous driving datasets and benchmarks provide extensive resources for routine driving, yet real-world data covering safety-critical scenarios and extreme environmental conditions remain limited. To address this gap, we present ExtremeDrive, a real-world dataset and benchmark covering emergency maneuvers, complex traffic interactions, and adverse conditions such as slippery or snow-covered roads, alongside routine driving. The dataset comprises 589 driving scenes curated from a larger pool of real-world driving recordings, with calibrated and aligned multisensor data and reviewed annotations of road structure and traffic participants. We further establish a benchmark to evaluate planned trajectories through simulated vehicle execution, measuring safety-related metrics and driving performance. Experiments at equal training set sizes show that incorporating critical driving data reduces trajectory prediction errors and improves mean critical-success rates in critical scenarios. ExtremeDrive provides a unified resource for learning and evaluating autonomous driving in safety-critical scenarios and extreme environmental conditions.
☆ OrthoMark: Place Recognition with Point-Anchored Orthographic Height Mark
LiDAR place recognition provides the loop closures of Simultaneous Localization and Mapping(SLAM). Perspective from Z-axis, a gravity-aligned scan is an orthographic map of the scene, and a revisit from another position and heading moves this map rigidly in the plane, a motion that a polar Fourier spectrum ignores and phase correlation recovers. Along the vertical, the heights of the returns describe the structures, but which heights are observed depends on the range, because the sensor's vertical field of view cuts nearby structures low; height thus misleads a global key, yet is decisive once two scans are aligned and compared cell by cell where they overlap. We present \method, a training-free method built on one principle: two LiDAR scans can only be compared through features computed from the same geometric surfaces in both scans. \method retrieves candidates with a translation- and rotation-invariant polar spectrum of the bird's-eye view (BEV), 1.9,KB per keyframe, and verifies each candidate by estimating its planar pose through phase correlation and scoring it with \emph{point-anchored height columns}: every BEV cell keeps a 16-bit mask of the heights of its own returns; within a session, the score is weighted by the expected overlap of the two scans' sensing ranges, computed from their distance. We show that this vertical link pays off in verification but hurts in retrieval, where what a height slice contains depends on the range to the structure. For single robot, \method has the highest AP of the training-free methods without 3D registration on the KITTI development sequences; across robots, on all multi-robot sequences, it matches or exceeds RING++ at 1/350 of its descriptor size and closes more inter-robot loops than the learned BEVPlace++ on all tests.
☆ sqrtVINS-Based 360 Visual-Inertial SLAM Systems: Hilti-Trimble SLAM Challenge 2026 Entry ICRA 2026
This report describes our submission to the 2026 Hilti-Trimble SLAM Challenge. The system is a 360-camera visual-inertial pipeline that uses only the provided dual-fisheye image streams and IMU measurements. First, raw images are preprocessed with low-light enhancement and static/dynamic masking, and then fed into a $\sqrt{\text{VINS}}$ frontend as the local odometry, with deep learning-based SuperPoint features, ELSED line segments, and LightGlue matching. The resulting trajectory is refined using a GTSAM-based backend that performs tightly-coupled factor graph optimization with SALAD global descriptor and visual bundle adjustment (BA). The backend performs local-to-global structureless BA, tightly regularized intrinsic and extrinsic refinement, and global scale adjustment supported by the known inter-camera baseline and cross-view constraints. The design targets long indoor sequences with dark regions, repetitive corridors, dynamic objects, and limited geometric parallax, while keeping the pipeline reproducible in the provided ROS2 workspace. Fig. 1 demonstrates that our proposed method aligns well with the floorplan across several indoor sequences.
comment: 2026 Hilti x Trimble SLAM Challenge at ICRA 2026 (2nd Place in the SLAM Category)
☆ Vision-Language Grounded Semantic Navigation for Autonomous Mobile Robots
Language-guided navigation enables mobile robots to interpret natural language commands and navigate toward user-specified targets in unstructured environments. However, integrating open-vocabulary visual perception with classical robotics pipelines for mapping and navigation remains a challenging problem. In this work, we present a complete ROS2-based system that combines frontier-based exploration, open-vocabulary object detection using Grounding DINO, depth-based 3D localization, and semantic mapping to enable end-to-end vision-language grounded navigation in simulation. The system accepts natural language commands such as ``go to the yellow box,'' detects candidate objects from RGB images using text prompts, estimates their positions in the world frame, and generates navigation goals executed by the Nav2 stack. In a Gazebo-based indoor environment with a TurtleBot3 robot, the system autonomously explored the room, detected three of the five target objects, and successfully executed both natural language navigation commands tested, without task-specific training. The results indicate that open-vocabulary grounding models can be integrated into classical navigation pipelines, though performance is limited by detection uncertainty, depth-based localization error, and incomplete exploration coverage. These findings highlight both the feasibility and current limitations of deploying vision-language grounded navigation systems using lightweight, CPU-only architectures.
comment: 7 pages, 4 figures
☆ AsymEEF: Bimanual Manipulation with Asymmetric End Effectors
Teleoperated demonstrations are costly to collect and rarely reusable across end-effectors with different kinematics and action spaces. This burden grows in bimanual manipulation, where each end-effector pair may require new data. We present AsymEEF, a framework for executable cross-end-effector demonstration generation and unified bimanual policy learning. Given a small source dataset, AsymEEF transfers demonstrations to heterogeneous combinations of two-, three-, and five-fingered end-effectors without additional teleoperation. Kinematic teacher anchors provide geometric supervision, while sparse real-world X-Calib anchors correct discrepancies in grasp execution. A shared encoder maps wrist poses and actuator states to a morphology-agnostic latent space, and a morphology-conditioned decoder reconstructs target commands. Generated trajectories undergo replay filtering for task-level executability. The representation also supports a unified bimanual visuomotor policy trained across multiple pairs. Across four simulated and six real-world tasks, AsymEEF improves replay success over adapted transfer baselines, achieves a mean replay-valid expansion gain of 16.6 times in simulation, and is evaluated through downstream policy training on real robots. Project website: https://asymeef.vercel.app/.
☆ WTR-Drive: World-Task Decomposition with Structured Risk Supervision for End-to-End Driving
Proposal-conditioned scene predictions can lack a common object-level reference across candidate trajectories. Aggregate planning scores also leave the objects and times associated with violations implicit. We present WTR-Drive (World-Task-Risk Driving), a Drive-JEPA-based planner that separates shared scene estimation from proposal-conditioned reasoning. A proposal-independent World Memory estimates object states, map structure, and future motion by re-reading the original visual features at each refinement round. Proposal feedback updates a residual Task branch without writing back to World Memory. This branch guides refinement through candidate-specific retrieval and gated visual feedback. A final Task readout supplies candidate-conditioned features for scoring. We derive structured risk targets from existing candidate-scoring rollouts without additional simulator calls. For at-fault collision and time-to-collision violations, the targets encode event existence, association with shared object slots, and the first violation time bin. On the NAVSIM navtest split, WTR-Drive achieves 94.66 PDMS on v1 and 88.24 EPDMS on v2 (expert-frame sequential evaluation). These scores are 0.96 and 0.44 points above the published Drive-JEPA values, respectively. Candidate-level analysis shows that higher best-candidate scores can coexist with lower selected-trajectory scores, revealing a gap between candidate quality and selection effectiveness.
comment: 8 pages, 5 figures, 3 tables. Submitted to IEEE Intelligent Vehicles Symposium (IV) 2027
☆ VoxSAOR: Semantic-Aware Occupancy Robustification via Aleatoric Uncertainty Modeling
Vision-based 3D occupancy prediction provides a dense semantic representation of scenes for autonomous driving and embodied perception, yet remains vulnerable to occlusion, depth ambiguity, and projection errors that induce spatially varying reliability in voxel features. Existing methods typically map voxel representations to semantic predictions without explicitly modeling such observation-induced feature variations. To address this limitation, we propose VoxSAOR, a semantic-aware occupancy robustification framework that models aleatoric uncertainty through feature-space proxies. It first quantifies Local Semantic Consistency (LSC) to derive a voxel-wise structural stability prior and uses cross-view semantic consensus to filter unreliable semantic correspondences. Within each semantic category, spatially matched stable and ambiguous voxels form stable-to-ambiguous feature transitions, which are modeled by a class-conditional diagonal Gaussian distribution, capturing both feature displacement and dispersion. Guided by these proxy distributions, Aleatoric Regime Ensemble Learning (AREL) trains independently parameterized prediction heads using feature perturbations at progressively increasing scales, with perturbation strength modulated by structural reliability. At inference, proxy modeling and sampling are disabled, and all heads process the same observed features; their output probabilities are then averaged to produce a deterministic consensus prediction without requiring sampling. Experiments across multiple datasets and occupancy backbones demonstrate the effectiveness of VoxSAOR in improving semantic occupancy prediction, long-tail recognition, and predictive calibration. The source code will be made publicly available at https://github.com/MengfeiD/VoxSAOR.
comment: The source code will be made publicly available at https://github.com/MengfeiD/VoxSAOR
☆ Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models NeurIPS 2026
Robot policy pretraining based on human videos is crucial for improving the policy's generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose LSA, a hierarchical robot policy pretraining framework that includes three phases: Lift, See, and Act. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder's attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The project page is available at: https://yiyuan00.github.io/Lift-See-Act.
comment: Accepted at NeurIPS 2026
☆ ReRead: Wrong Predictions, Useful Representations - Correcting Frozen Robot Policies by Re-reading Their Own Representations
When a pretrained robot policy fails on challenging tasks, common remedies include retraining the backbone, adding an external visual encoder, or introducing a memory module. We argue that wrong predictions do not imply wrong representations: a policy can retain the required information yet fail to read it out as the correct action. To test this view, we propose ReRead: with the backbone frozen, a corrector re-reads the current and two previous caches to predict an action residual. The corrector sees no images, is trained solely on offline demonstrations from all 50 RoboTwin 2.0 tasks, and adds only 4.17% latency. Ablations and interventions show that the correction relies on the backbone's own representations: none of four external visual encoders adds further gain, and swapping in past caches from other trajectories sharply increases action error. On the 11 hardest RoboTwin 2.0 tasks, ReRead improves the success rates of LingBot-VLA, $π_{0.5}$, and Fast-WAM by 23.45, 13.09, and 8.00 percentage points, respectively. In paired evaluation on all 50 tasks, it raises Fast-WAM's average success rate from 90.73% to 92.66%. On a Flexiv dual-arm robot, it raises success on a phase-ambiguous task from 0% to 36%.
☆ Stochastic Self-Distillation for Flow-Based Vision-Language-Action Models
Flow-based vision-language-action (VLA) models achieve strong performance in robotics through imitation learning, yet fine-tuning them with on-policy reinforcement learning (RL) is limited. In group-based RL methods like Group Relative Policy Optimization (GRPO), every rollout step requires multi-step ODE integration, and the sampler's limited stochasticity leads to rollout groups with low variance. We propose VLA-SSD, which addresses both high inference latency and limited stochasticity. In particular, our method self-distills a pretrained flow-matching action head into a stochastic one-step policy by conditioning a single-jump generator on a Brownian path. We derive a conditional action density via our stochastic one-step policy, enabling efficient post-training via likelihood-ratio RL algorithms. We show that distilling VLA-SSD from the $π_0$ VLA reduces the action head's forward-pass time by a factor of 7.7, at the cost of a 5.5% relative decrease in the success rate on the LIBERO benchmark. However, by leveraging the increased sample diversity of VLA-SSD during group generation, post-training a $π_{0.5}$ VLA using GRPO yields a 20.6% relative improvement.
☆ XMimic and XSoma: Learning Humanoid Motion Tracking and Locomotion in Under a Minute
We present XMimic and XSoma, which learn humanoid motion tracking and locomotion on rough terrain in under a minute of training. Both follow SuperMimic, which learns a world model of the robot and optimizes the policy with analytic gradients through short rollouts imagined in this model. Existing implementations of this method do not make use of larger GPUs, as every training iteration is a long chain of small GPU operations that run one after another. XMimic and XSoma therefore shorten this chain and run the steps of every iteration at the same time on up to four GPUs. We compile the complete training loop with JAX, including the MuJoCo Warp simulator. For locomotion, XSoma leaves only the dynamics of the robot to the world model and computes everything else exactly, including the terrain height under the feet. It learns with a horizon of one second and without a critic. On four H100 GPUs, XMimic learns to track all 40 LAFAN1 clips with the Unitree G1 in 0.86 minutes, 3.7 times faster than BlinkMimic on one GPU. XSoma reaches the tracking target that Holosoma's tests set for FastSAC in 0.77 minutes, 12.3 times faster than FastSAC on one GPU, while needing only an eighth of its environment steps.
☆ Making Koopman Control Guarantees Hold on Robots
A Koopman controller can stabilize its learned latent-space linear model yet fail to stabilize the nonlinear plant. The learned model propagates predictions in latent space, while the deployed controller repeatedly re-encodes the measured physical state. We show that nominal closed-loop stability transfers to the nonlinear plant when the learned and true closed-loop dynamics agree to first order on the embedding's tangent space at the target. We derive a robustness margin for approximate matching and show that a stable open-loop latent model cannot exactly match an unstable equilibrium. We introduce Tangent-Invariant Koopman (TIK), which estimates a local plant linearization from offline transitions and enforces first-order matching to this estimate. TIK recovers the fitted LQR gain at the target, while a bilinear predictor captures state-dependent actuation during large maneuvers. Across four simulated unstable benchmarks, all 40 Standard Koopman controllers have stable latent closed loops, but their deployed nonlinear systems are locally unstable. In contrast, all 40 TIK controllers locally stabilize their respective nonlinear plants. An offline stability test using empirical curvature estimates identifies stable TIK controllers on two cart-pole systems, but is inconclusive under high noise. Under explicit model-error and curvature bounds, an invariant terminal set guarantees constraint satisfaction after entry under local feedback. TIK achieves 88\% constrained cart-pole swing-up success with a 0.25 ms planning budget and controls simulated seven-joint arms within 1-5 ms per step.
comment: https://walleclipse.github.io/tik-koopman/
♻ ☆ Parameter-Dependent LMI Synthesis for Semi-Global Differential ISS Trajectory Tracking of Nonholonomic Mobile Robots Under Multiplicative Wheel Slip
This paper presents a parameter-dependent linear matrix inequality (LMI) synthesis framework for trajectory tracking of nonholonomic mobile robots subject to multiplicative wheel slip. A sampled polytopic convex formulation jointly targets four design objectives on the variational dynamics in Kanayama coordinates: a prescribed exponential decay rate, an H-infinity disturbance-to-error gain, regional pole placement, and a weighted gain bound acting as a design-time actuator budget. The inverse metric and the auxiliary gain are parameterized affinely in the reference linear and angular velocities. A Slip-to-Disturbance Lemma provides an explicit closed-form upper bound on the additive disturbance equivalent to bounded multiplicative wheel slip in Kanayama coordinates, connecting the physical slip mechanism to the convex synthesis. The dissipation inequality is established at the sixteen vertices of the parameter-rate polytope, and a Lipschitz-based residual condition is given as a sufficient criterion for extending feasibility to interior points; the regions where the criterion succeeds for the synthesized controller, and where it does not, are both reported. An integral representation of the Kanayama residual lifts the variational certificate to a pointwise exponential-plus-ISS bound on the trajectory error directly from the LMI. The certified envelope is the semi-global ball at sigma_bar = 0.10; variable-terrain tests at slip ratios up to plus-minus 50% are reported as empirical observations outside that envelope. In the certified regime, the LPV controller and a fixed-gain LMI baseline produce quantitatively similar tracking records; on the severe-slip probes, the parameter-dependent design retains modest reproducible margins over both the fixed-gain and the classical Kanayama baselines.
comment: 23 pages, 18 figures, 50 references
♻ ☆ Racing the Clock: Teaching VLNs Budget-Conditioned Object Search
How do action budgets shape vision-language navigation (VLN) exploration, and what expert supervision best teaches budget-aware search? We study budget-conditioned object search: finding as many distinct instances of a requested category as possible within a budget, with discovery defined geometrically by camera field of view and proximity. Distinct discoveries provide a denser evaluation signal than single-target success. We cast expert design as a choice of scene abstraction: a single budget-indexed Bellman recursion, applied to different abstractions of the full environment, yields experts that differ in what they know. An object-level Bellman expert is given every target location and reduces search to budgeted routing. A room-level Bellman expert retains observed room connectivity, doorway evidence, and cross-scene semantic priors but withholds the contents of unexplored rooms, so it must search for targets that its student also cannot see. We distill both experts into Uni-NaVid under matched training conditions and also evaluate greedy-frontier supervision on held-out HM3D scenes. Changing only the stated budget changes learned trajectories. Despite higher object-level expert return, room-level supervision achieves the highest mean collection among evaluated VLNs at every tested budget. At 2000 steps, it finds 72.7% of reachable targets, versus 68.5% under greedy-frontier supervision, 65.5% under object-level supervision, and 44.8% for the pretrained VLN. These results show that expert-return rankings need not predict supervision quality for VLN exploration, and motivate choosing an expert's abstraction with the student's information in mind rather than by expert return alone.
comment: 12 pages, 4 figures, 3 tables
♻ ☆ Learning-based Autonomous Oversteer Control and Collision Avoidance
Oversteer, wherein a vehicle's rear tires lose traction and induce unintentional excessive yaw, poses critical safety challenges. Failing to control oversteer often leads to severe traffic accidents. Although recent autonomous driving efforts have attempted to handle oversteer through stabilizing maneuvers, the majority rely on expert-defined trajectories or assume obstacle-free environments, limiting real-world applicability. This paper introduces a novel end-to-end (E2E) autonomous driving approach that tackles oversteer control and collision avoidance simultaneously. Existing E2E techniques, including Imitation Learning (IL), Reinforcement Learning (RL), and Hybrid Learning (HL), generally require near-optimal demonstrations or extensive experience. Yet even skilled human drivers struggle to provide perfect demonstrations under oversteer, and high transition variance hinders the accumulation of sufficient data. Hence, we present Q-Compared Soft Actor-Critic (QC-SAC), a new HL algorithm that effectively learns from immature demonstration data and adapts rapidly to new conditions. To evaluate QC-SAC, we introduce a benchmark inspired by real-world driver training: a vehicle encounters sudden oversteer on a slippery surface and must avoid randomly placed obstacles ahead. Experimental results show that QC-SAC attains near-optimal driving policies, significantly surpassing state-of-the-art IL, RL, and HL baselines. Hardware-in-the-loop simulation (HILS) validation with physical steering hardware further demonstrates that QC-SAC can operate beyond a software-only simulation, providing an intermediate validation step toward real-vehicle implementation. To the best of our knowledge, this is the first work that achieves safe autonomous control of vehicles in oversteer while simultaneously avoiding obstacles.
comment: 14 pages, 8 figures, 4 tables. Accepted for publication in IEEE Transactions on Intelligent Vehicles
♻ ☆ ESI: An Embodied Self-Improvement Flywheel for LLM-VLA Agents
Independently pretrained language models and vision-language-action (VLA) policies provide complementary building blocks for embodied agents, but connecting them does not automatically produce coordinated behavior. The VLA may generate actions that diverge from the LLM's intended subtask and, upon regaining control after a correction, undo the promising state restored by the LLM. We introduce ESI (Embodied Self-Improvement), an automated flywheel that uses environmental interaction to align reasoning and action. ESI treats the modular agent as the unit of post-training, repeatedly refining the prompts that guide the LLM, the shared skill interface that expresses its intent, and the VLA parameters that realize that intent, then redeploying these components together. Our key insight is that collaboration itself can supply supervision for improving this interface. The LLM's subtask decisions provide semantic supervision for what the policy should execute, while its effective corrections and subsequent student continuations provide action-level supervision for recovery and handoff. The student also learns from its own effective execution, allowing locally useful experience to contribute even when the overall episode fails. On ten selected RoboDojo tasks, ESI achieves a mean Score of 62.03 and a mean success rate of 49.41% using an LLM-VLA agent without generating task-specific control code. On both metrics, it outperforms the published reference results of all hybrid embodied agents, including the Agent+WAM system GO-3 Nano Preview (47.27 Score and 36.60% success), as well as all compared code-based methods.
comment: Technical report. 25 pages, 7 figures. Code: https://github.com/ZERITH-ZLab/ESI
♻ ☆ LLM-Flax: LLM-Generated Rules and Object Guidance for Neuro-Symbolic Task Planning
Adapting a neuro-symbolic task planner to a new domain can require hand-authored symbolic rules and solved training instances for a learned object scorer. This paper presents LLM-Flax, which applies large language models (LLMs) to offline rule generation, failure recovery, and object scoring within the Flax planning pipeline. Generated rules undergo validation and corrective feedback before reuse, while runtime guidance operates within a shared planning budget. Experiments on MazeNamo and two additional domains show that the evaluated rule configuration improves planning success over manually authored rules while retaining the trained object scorer. Across eight MazeNamo settings, the mean success rate increases by approximately three percentage points. Controlled comparisons distinguish the effects of rule content from the choice of generator. Runtime recovery and object scoring do not establish an overall performance advantage under the evaluated planning budgets. These findings demonstrate the value of reusable LLM-generated rules and identify information quality and inference efficiency as central challenges for runtime guidance.
♻ ☆ HINT: Human-Intent Inception for Agentic Robot Manipulation
Mingyu Mei, Haojie Xu, Shihao Jin, Zibo Dai, Qihao Cheng, Zhengrui Lv, Hongjie Fang, Shirun Tang, Guang Chen, Xinyue Zhao, Huiliang Shen, Zaixing He
Humans can perform complex manipulations given a simple intent through an overall instruction, while continuously adapting to evolving visual observations. However, current vision-language action (VLA) models and other action policies struggle to realize this high-level intelligent behavior under dense, evolving visual inputs and sparse language guidance. Visual correlations can then dominate semantic intent, leading actions to follow visual shortcuts rather than human goals. We present HINT (Human-INTent INcepTion), an agentic framework inspired by the human manipulation principles: semantic intent changes sparsely at manipulation-pattern transitions, whereas continuous control primarily depends on the evolving object-hand relationship. HINT invokes semantic reasoning only at pattern transitions to resolve the current subtask and target, then maintains this commitment through multi-view grounding and visual tracking. We explore two visual interfaces-image-space semantic highlighting and attention-prior injection-to communicate the tracked intent to the action policy without introducing additional trainable parameters into the foundation action model. Experiments across three long-horizon tasks and out-of-distribution variants show that HINT substantially improves intent understanding, task progress, and end-to-end success across two foundation policies while preserving low-latency control. Project page: https://robot-hint.github.io/
comment: Title updated
♻ ☆ Parallel Branch Model Predictive Control on GPUs
We present a GPU-based solver for trajectory planning problems using branch model predictive control. Building on iterative LQR methods, we adopt a multiple-shooting formulation for the system dynamics and use an augmented Lagrangian method to handle general stage-wise constraints. This design enables straightforward warm-starting. The constraint-handling capability of our solver is validated on two challenging trajectory planning problems. In addition, we develop two tailored inner LQR solvers that exploit the tree-sparse structure. The solvers offer different levels of parallelism, making them appropriate for different tree sizes. The numerical results demonstrate that, compared to a high-performance CPU-based solver, our approach achieves superior performance on large-scale problems.
comment: 11 pages, 8 figures
♻ ☆ CtRoFlow: Monocular Camera-to-Robot Pose Estimation via Conditioned 3D Flow Matching
Camera-to-robot pose estimation from a monocular image is significant for vision-based robot navigation and control. Existing camera-to-robot pose estimation methods either rely on time-consuming iterative rendering, PnP based on predicted 2D keypoints with limited robustness or 3D registration which suffers from monocular scale ambiguity and complex training configuration. In this work, we propose CtRoFlow, a flow matching-based framework for camera-to-robot pose estimation. From the predicted 2D heatmaps, we propose a Keypoint-centered Feature Extractor (KFE) to obtain positional features along with contextual representations from keypoints-centered multi-scale patches. Moreover, we introduce Graph-guided Condition Generator (GCG), which generates condition for flow matching based on correlation among keypoints. Besides, we introduce Bone Length Loss and Ray-based Loss for decoupled training and scale consistency, with joint-wise weighting to mitigate varied prediction difficulty across joints. The proposed method outperforms state-of-the-art methods for pose estimation across different types of robots with multiple cameras, as well as for zero-shot cross-scene robot pose estimation. Code will be available upon acceptance.
comment: 9 pages, 8 figures. Submitting
♻ ☆ Galilean Symmetry in Robotics
In this paper, we present a robotics-tailored exposition of Galilean symmetry that leverages the community's familiarity with and understanding of rigid-body transformations and pose representations. Our approach contrasts with common treatments in the physics literature that introduce Galilean symmetry as a stepping stone to Einstein's relativity. We show that the classical homogeneous matrix algebra known well to roboticist translates directly into the Galilean modeling framework and provides a powerful tool for tackling robotics problems involving inertial measurements and spatio-temporal variables. We demonstrate the potential of the approach by developing a powerful model for studying motion of vehicles in an Earth fixed rotating reference frame, and developing a fusion algorithm for measurements with uncertain time-stamps.
comment: accepted
♻ ☆ A Generalizable Approach to Safety-Aware Robotic Behavior Planning with an Explicit Constraint Space
Xianxu Wu, Shang-Ching Liu, Zhaocong Liang, Hengxiang Chen, Yujian Ma, Shenwen Deng, Qiang Li, Yung-Hui Li, Jianwei Zhang
Large language and vision-language models have substantially advanced robotic behavior planning from natural-language instructions and visual observations. However, safe manipulation in tabletop environments remains challenging because the safety constraints required for a task depend on the surrounding scene and may not be explicitly specified in the instruction. We propose a safety-aware behavior planning framework based on an explicit Constraint Space that stores reusable textual safety rules. Rather than retrieving rules solely by semantic similarity, we formulate constraint selection as task- and environment-conditioned applicability learning. A lightweight query-conditioned dual-tower model identifies the constraints applicable to the current context, which are then provided together with the original task and image to a multimodal behavior planner. To handle incorrect constraint selections and previously unseen hazards, we introduce a multimodal Safety Inspector that evaluates generated primitive plans, reconstructs the complete applicable constraint set, and provides corrective planning feedback. Validated inspection feedback is further used for test-time adaptation of constraint applicability and parameter-efficient adaptation of the Safety Inspector through LoRA, while newly identified constraints are incorporated into the Constraint Space. The resulting framework integrates explicit safety knowledge representation,context-aware constraint selection, corrective safety inspection, and test-time knowledge adaptation for safety-aware robotic behavior planning.
♻ ☆ VLALagBench: Understanding When and Why VLA Inference Latency Matters
Qi Wen, Borui Xu, Hussein Amro, Hongshi Tan, Hanmin Lin, Chaoyi Ruan, Shanghao Liu, Gang Yang, Yuejiang Liu, Jiafei Duan, Bingsheng He, Jason Clemons, Christos Kozyrakis, Wenqi Jiang
Vision-language-action (VLA) models offer a promising path toward general-purpose robot manipulation. While inference latency can affect task success in a constantly evolving physical world, the relationship between latency and task success rate remains poorly understood. In this paper, we introduce VLALagBench, a comprehensive benchmark for studying VLA inference latency sensitivity across task types, object speeds, execution modes (synchronous and asynchronous), and VLA policies. Using VLALagBench, we find that even a modest inference latency of 100 ms can nearly halve task success on moving targets, with sensitivity varying substantially across tasks and policies. These differences reflect task-specific requirements for precision and continuous motion. By overlapping inference with action execution, asynchronous execution can recover some of the lost success at the cost of more frequent inference requests. Building on these findings, we develop an empirical model that predicts task success from limited calibration data, achieving root-mean-square prediction errors of only 5.7-7.8% across various VLA policies. Given an inference system's latency and per-robot throughput, these predictions guide the selection of system-specific VLA algorithm configurations to maximize task success. We believe VLALagBench and these findings provide a solid foundation for future robotics algorithm-system co-design.
comment: code: https://github.com/codestrongestx/vla-lag-bench
♻ ☆ Grounding Driving VLA via Inverse Kinematics
Counterfactual obstacle insertion exposes blind planning in representative Driving VLAs: their trajectories remain largely unchanged despite a vehicle appearing directly ahead. An empirically calibrated bias-variance analysis identifies how ego-predictable training examples admit a shortcut through ego status and commands. We address this failure with an inverse-kinematics-inspired architecture that anchors trajectory decoding on current and predicted future visual states. First, a next visual state prediction objective that requires the LLM to predict the future visual scene provides dense visual supervision and suppresses shortcut paths. Second, a separate Inverse Kinematics Network (a cross-attention-based conditional diffusion model) that takes only the current and future visual states as input is designed to suppress reliance on ego status and textual shortcuts during trajectory decoding. Our 0.5B model achieves state-of-the-art trajectory planning performance on NAVSIM-v1, NAVSIM-v2, and nuScenes using cameras alone, without LiDAR or a BEV encoder. Against the matched OpenDriveVLA* baseline, it improves NAVSIM PDMS and EPDMS by 19.0 and 20.4 points, respectively. Controlled ablations and intervention analyses support improved use of visual features, with future-state prediction sharpening hazard-specific responses.
♻ ☆ Long-Term Multi-Session 3D Reconstruction Under Substantial Appearance Change
Long-term environmental monitoring requires the ability to reconstruct and align 3D models across repeated site visits separated by months or years to enable the detection of structural change and the assessment of ecosystem health. However, existing Structure-from-Motion (SfM) pipelines are generally designed for imagery captured within a limited time window and consequently fail in long-term monitoring scenarios such as repeated coral reef surveys with substantial visual and structural change. In this paper, we show that the primary limitation of current approaches lies in their inability to maintain visual correspondences under substantial temporal appearance change. We address this limitation by enforcing cross-session correspondences within a joint SfM reconstruction. Our approach combines handcrafted and learned visual features to establish correspondences across large temporal gaps, enabling the reconstruction of a single coherent 3D model from imagery captured years apart, where standard SfM fails to connect sessions. To ensure scalability and further improve alignment robustness on large datasets, we restrict expensive learned feature matching to a small set of likely cross-session image pairs identified via visual place recognition, which reduces computational cost while improving the reliability of cross-session correspondence establishment. We evaluate our method on long-term underwater datasets exhibiting significant real-world change, and demonstrate consistent joint reconstruction across sessions in cases where existing methods fail to connect the sessions or align them only coarsely.
♻ ☆ Learning Latent Volumetric Dynamics for Multimodal Goal-Conditioned Robot Manipulation of Hair ICRA 2026
Chengyang Zhao, Uksang Yoo, Arkadeep Narayan Chaudhury, Giljoo Nam, Nathan Dennler, Andreea Bobu, Jonathan Francis, Jeffrey Ichnowski, Jean Oh
Hair care is an essential daily activity for many, yet it remains inaccessible to individuals with limited mobility and challenging for robot systems due to the fine-grained physical structure and complex dynamics of hair. In this work, we present DYMO-Hair, a model-based robot hair care system for generalizable multimodal goal-conditioned hair styling. We introduce a volumetric dynamics learning method that models combing-induced hair deformations through action-conditioned editing in a pre-trained hierarchical 3D latent space. This latent space captures diverse hair states through volumetric occupancy and local strand orientations, while large-scale pre-training, supported by a physics-based hair-combing simulator, enables generalization to previously unseen hairstyles. Coupled with a Model Predictive Path Integral (MPPI) planner, the learned dynamics model enables closed-loop hair styling toward both visual references and natural language instructions as user input. In closed-loop experiments, DYMO-Hair achieves lower geometric error than baseline systems for visual goals and receives significantly higher ratings than VLM-based planners for language goals in a 58-participant user study. Real-world experiments further demonstrate zero-shot transfer to seven physical wigs varying in color, length, and texture, effectively styling hair toward both visual and language goals without real-world fine-tuning. Together, these results represent a step toward more generalizable, flexible, and accessible robot hair styling in the real world. More details are available on: https://dymohair.github.io/.
comment: Extended version of the previous work in ICRA 2026. Project page: https://dymohair.github.io/
♻ ☆ Toward Interaction Dynamics: A Predictive Framework for Safe Physical Human Robot Interaction
Physical human-robot interaction requires yielding transiently to contact yet recovering the commanded reference under sustained load. Finite-stiffness impedance control retains a static deflection there, while predictive alternatives typically optimize a nonlinear model online. Operational-space cancellation instead exposes a translational error double integrator with a fixed transition matrix and a configuration-scheduled input map, making interaction a predictive quantity. We build on it a compact offset-free interaction-error MPC for torque-controlled manipulators: a force-domain random-walk state estimates persistent interaction and model error, and a 30-variable convex QP maps the correction through the current task inertia while constraining the applied joint torque. Conditional results establish impedance equivalence of the unconstrained passive feedback, offset-free regulation at feasible frozen configurations, and quadratic stabilizability of the scheduled backbone. In a 1 kHz MuJoCo simulation of a 7-DOF Franka FR3, the estimator cuts steady-state error under a repeated 15 N step from 2.73 mm to 0.033 mm when added to the otherwise identical 100 Hz MPC, at unchanged peak joint power. A stiffness-and-damping-calibrated impedance baseline attains 2.63 mm at the same actuator cost, and 1.36 mm with ideal measured-force cancellation, so constant-load rejection is not unique to MPC; the sensorless controller still reaches 0.033 mm. The torque constraint holds the command inside a budget tightened to the lowest the task can meet, at no tracking cost. On a physical UR5e, five alternated pairs under a nominal 1.5 kg hanging load give 1.505 mm without the estimator; with it the controller is offset-free, its 0.023 mm error being the noise level of the encoder-based position measurement. A tuned anti-windup integral also removes the offset, with larger transients.
♻ ☆ ToM-2 Understanding Statements for Teacher-Guided Preference Learning IROS 2026
Preference learning is a powerful tool for reward learning, but treats the teacher as a passive oracle that answers queries driven by the learner. This learner-guided query selection ignores the teacher's knowledge of the solution. We reverse the typical preference query selection method by having the teacher create ordered preference pairs, since a teacher who knows the target can choose more informative pairs than a learner choosing its own queries. Exploiting this advantage requires the teacher to model what the learner knows, and that model goes stale when the learner's belief changes in ways the teacher cannot see, as when several teachers each teach without knowing what the others taught. We call this teacher-model drift. In this work, we use second-order Theory-of-Mind (ToM-2) to have the learner create understanding statements that correct the teacher's model of the learner's knowledge. In simulation, we show that a learner with a perfect teacher, one with perfect knowledge of the learner, needs about half as many rounds as learner-guided querying to learn the reward. Yet when teacher-model drift is induced by alternating between four co-teachers, the advantage erodes. We tested two types of understanding statement generation: the learner's mean belief and ToM-2 drift minimization. ToM-2 statements reduced the drift and recovered more of the lost advantage than mean statements. In a within-subjects study (n = 101) with a simulated ideal co-teacher, ToM-2 statements raised the learner's final alignment with the true reward above both uninformed and mean-informed teaching; on the regret participants removed in their own rounds, they outperformed uninformed teaching but not mean statements.
comment: v2: substantially revised and extended version with a human-subject study. v1 appeared at the IROS 2026 Workshop on Leveraging Behavioral Models in Human-Autonomy Teams (BEMHAT)
♻ ☆ Gait Generation Balancing Joint Load and Mobility for Legged Modular Robots with Easily Detachable Joints
Modular robots can adapt their morphology to diverse tasks, but locomotion can impose large loads on detachable joints. This paper presents an NSGA-III-based multi-objective gait optimization framework with three objectives: a locomotion speed, a static stability, and a normalized joint-load metric based on joint reaction forces. The framework yields nondominated gait candidates that reveal the trade-offs among these objectives. In a flat-terrain comparison using the 6-legged tetrapod gait, adding the joint-load objective reduced the mean maximum joint reaction force by 11.5% and increased the mean minimum stability by 41.2%, while decreasing the mean forward speed by 10.5%. Simulations under flat, slope, and step conditions showed that locomotion feasibility depended on the combination of robot configuration, gait, and terrain. In physical experiments, the 4-legged trot completed locomotion on flat terrain and a 10 degrees slope, whereas the 6-legged tetrapod and tripod completed locomotion on flat terrain. These results demonstrate the feasibility of joint-load-aware gait generation based on explicit trade-offs among the three evaluated quantities
comment: 8 pages, 9 figures
♻ ☆ HAVANA: A Hierarchical Video-Action Model for Vision-Language Navigation
Vision-Language Navigation (VLN) requires long-horizon instruction-consistent planning and short-horizon path planning grounded in the currently observed geometry. Anticipating how future observations evolve along the instructed route provides predictive guidance beyond the current field of view, motivating video-action models (VAMs) that learn representations of future visual states. However, many existing VAMs couple future visual prediction with action generation during inference, despite their distinct temporal requirements: instruction-conditioned route guidance can be reused across multiple control steps, whereas local trajectories must continuously adapt to fresh observations. To bridge this gap, we introduce HAVANA, a dual-system VAM that decouples predictive reasoning from reactive local planning. A slow system produces a compact language-conditioned latent plan from observation history, while a fast system combines the latest plan with the current observation to generate local trajectories. Their asynchronous execution enables rapid adaptation without repeated future prediction, and no future video is generated during inference. Experiments on OmniNavBench demonstrate state-of-the-art navigation performance over prior baselines, while zero-shot real-world deployments further show robust generalization to unseen environments.
comment: 10 pages,6 figures
♻ ☆ One from Infinity: Actualizing Futures from Pretrained World Models into Robot Actions
A pretrained video world model admits many plausible futures for a scene, but a robot must realize the exact task-conditioned one. To turn world models into executable robot policies, existing methods fine-tune the heavy world model backbone using large-scale robot data and computational resources. Challenging this status quo, we argue that the expensive part has already been paid in the world model pretraining since the representation space of a video world model encoder lays out the diverse potential futures. In this case, what remains is to determine the future that accomplishes the task and to read out the actions that realize it. We formalize this task as actualization, which learns a task-conditioned realization on top of a prior supplied by a frozen world model encoder. This can be solved by a tiny actualizer model. We implement RoboActualizer with as few as 60M parameters on top of a frozen world model encoder. The actualizer is composed of two lightweight DiT experts that jointly predict future latents and actions by flow matching. The model can be trained entirely on a single GPU with 32 GB peak memory. With up to 100x fewer trainable parameters than existing WAMs and VLAs, RoboActualizer reaches great performance on simulation benchmarks including LIBERO, LIBERO-Plus, RoboTwin 2.0 and five tasks on two real-world platforms, with a low latency of 39 ms that allows real-time control.
♻ ☆ Spatial Calibration of Diffuse LiDARs
Diffuse direct time-of-flight LiDARs report per-pixel depth histograms formed by aggregating photon returns over a wide instantaneous field of view, violating the single-ray assumption behind standard LiDAR-RGB calibration. We present a simple spatial calibration procedure that estimates, for each diffuse LiDAR pixel, its footprint (effective support region) and relative spatial sensitivity in a co-located RGB image plane. Using a scanned retroreflective patch with background subtraction, we recover per-pixel response maps that provide an explicit LiDAR-to-RGB correspondence for cross-modal alignment and fusion. We demonstrate the method on the ams OSRAM TMF8828.
♻ ☆ MorphQuad: Morphable Quadrotor for Superhuman Maneuverability, Manipulation, and Resiliency
Jose Diaz Peon Gonzalez Pacheco, Jiawei Xu, Andrew Zhao, Hongyu Zhou, Atharva Navsalkar, Andrew Scheffer, Amrith Malli Reddi, Sashreek Shankar, Yuqing Bao, Vasileios Tzoumas
Infrastructure maintenance, contact-based inspection, and emergency response can benefit from aerial vehicles that act as a flying human hand with extreme maneuverability, manipulation, and resiliency (MMR): maneuverability to fly in arbitrary orientations; manipulation to point sensors, turn valves, and press tools at arbitrary orientations; resiliency to maintain motion accuracy and force application despite disturbances from arbitrary directions, such as wind, ground effects, and friction. Realizing MMR requires more than omnidirectional flight; it also requires (I) vectoring of maximum thrust in any direction, (II) almost-everywhere stability, accounting for kinematic singularities, control singularities, and inter-rotor downwash interference, and (III) miniaturizable structural designs. Existing aerial vehicles satisfy a subset of I--III. We present MorphQuad, a morphable quadrotor that delivers MMR by realizing I--III on a quadrotor structure with independently articulated rotor systems through novel control and thrust allocation algorithms. On hardware, MorphQuad preserves a planar quadrotor structure, and independently articulates each of the four rotor systems, eliminating control singularities and enabling all rotors to align and vector the platform's maximum thrust in any direction; on control and thrust allocation, MorphQuad enjoys a closed-form null-space correction for gimbal lock and downwash interference that maintains almost-everywhere stability via coupling with a generalized geometric controller. With fully-onboard computation, localization, and trajectory execution, MorphQuad demonstrates continuous multi-revolution rotation while translating or hovering (maneuverability); valve turning, perching, and object pressing and pushing with human-wrist strength (manipulation); and push-pull recovery and wind rejection, even when directed to a single rotor (resiliency).