Whole-Body Teleoperation

Summary

Whole-body teleoperation is the practice of driving every controllable degree of freedom on a humanoid robot, arms, hands, torso and sometimes legs, from a human operator’s own movements in real time. It exists because a humanoid has too many joints and too many failure modes for scripted motion to cover, so a person’s motion becomes the source of both the training demonstration and the moment-to-moment control signal. It matters for Physical AI because the recording it produces is the raw material for imitation learning policies, and what that recording actually contains, versus what a corpus later assumes it contains, is a specification question with no single settled answer.

Whole-Body Teleoperation: Definition and Scope

Whole-body teleoperation captures a human operator’s body, hand and head motion through a wearable or camera-based interface, converts it into target poses or joint commands for a humanoid robot, and lets the robot’s own controller execute and stabilize that motion in real time. It differs from single-arm teleoperation, which drives a gripper or manipulator in isolation, because it must also keep the robot balanced while the operator’s motion is reproduced. The scope here is the pipeline from capture device through the recorded demonstration, not the balance controller as a control theory topic in its own right, which Robot Calibration and Coordinate Frames and future infrastructure nodes cover separately.

How Whole-Body Teleoperation Works in Practice

Take a warehouse pick task recorded with Apple Vision Pro tracking, as described for the Humanoid Everyday dataset. The headset’s downward-facing cameras extract the operator’s wrist and finger keypoints at 30 Hz. Two parallel transforms turn those keypoints into robot commands: a dexterous-hand retargeting system maps finger motion onto the robot’s own hand topology, and a Pinocchio-based inverse kinematics solver converts wrist poses into arm joint targets (arXiv:2510.08807). The robot’s low-level controller then executes those joint targets, adding whatever correction is needed to keep the robot from tipping. What gets written to disk, RGB video, depth, joint poses and joint actions, is the output of that whole chain, not a copy of the operator’s motion.

The failure this produces is specific. The corpus is usually tapped at the executed joint command level, after the controller has already blended in stabilization torques, so a policy trained on the recording cannot tell whether a knee bend it is imitating was the operator’s intended crouch or the controller’s save from a near stumble. AgiBot World’s own documentation illustrates the upstream half of this problem: its VR mode restricts the dexterous hand to a handful of predefined gestures because the controller only maps hand closure to end-effector rotation and translation, while its separate motion capture mode records individual finger joints and therefore captures intent that VR mode structurally cannot (arXiv:2503.06669). Two capture methods on the same robot platform produce demonstrations with different information content, and neither the video nor the joint log says which mode a given episode used unless that is logged explicitly.

Use Cases

• Imitation learning demonstration collection for humanoid manipulation policies: production practice at data collection scale, exemplified by AgiBot World’s more than one million trajectories (arXiv:2503.06669).

• Live remote operation of a humanoid for tasks too unstructured for an autonomous policy, such as disaster response reconnaissance or one-off manipulation: research stage, gated by the latency figures discussed below.

• Sim-to-real controller validation, where a teleoperated motion stream drives both a simulated and a physical robot to check that a policy trained in simulation transfers: research stage, demonstrated on the Unitree G1 with zero-modification sim-to-real deployment (arXiv:2605.12347).

Companies

• Unitree: Manufactures the G1 and H1 humanoid platforms used in the Human2Humanoid and Humanoid Everyday projects, as well as in the IMU-based sim-to-real study cited here. Production hardware vendor, not a teleoperation software company (Unitree G1).

• AgiBot: Runs teleoperation data collection at fleet scale across more than 100 humanoid robots, using both VR and motion capture modes on the same platform to trade off manipulation complexity against collection speed. Startup, production-scale collection (AgiBot World).

Datasets

• AgiBot World: 1,001,552 trajectories totaling 2,976.4 hours, collected across VR and motion capture teleoperation modes on more than 100 robots. The largest of the whole-body teleoperation corpora surveyed here, nearly a hundred times the scale of Humanoid Everyday below (project page).

• Humanoid Everyday: 10,300 trajectories and more than 3 million frames across 260 tasks, captured through Apple Vision Pro teleoperation on Unitree G1 and H1 platforms, with RGB, depth, LiDAR, tactile, IMU, joint pose and joint action streams recorded per timestep (arXiv:2510.08807, GitHub).

Papers

• Teleopit, 2026. Optimization-based hand retargeter combining finger direction, fingertip closure and thumb-frame alignment across different robotic hands, plus a history encoder and failure-aware rewind sampling. 96 successful demonstrations trained ACT to 90.0 percent and GR00T N1.7 to 95.0 percent task success (arXiv:2608.01834).

• Human2Humanoid, 2026. Morphology-invariant end-effector consistency loss and a skeleton-aware graph convolutional network, addressing skeletal topology, limb proportion and degree-of-freedom mismatches between a human demonstrator and a Unitree G1 (arXiv:2606.03476).

• ExtremControl, 2026. Maps human motion directly to SE(3) poses of the hands, feet, torso and pelvis rather than through full joint-space inverse kinematics, and adds velocity feedforward to the low-level controller. Reports roughly 50 milliseconds end-to-end latency against a prior baseline near 200 milliseconds (arXiv:2602.11321).

• Real-Time Whole-Body Teleoperation with IMU-Based Motion Capture, 2026. Drives a Unitree G1 from a wearable IMU suit, validated first in MuJoCo simulation and then deployed to the physical robot without modification (arXiv:2605.12347).

• Humanoid Everyday, 2025. Apple Vision Pro capture with dex-retargeting for hands and Pinocchio-based inverse kinematics for arms, released with the dataset described above (arXiv:2510.08807).

• AgiBot World, 2025. Documents the VR mode versus motion capture mode split described above and the collection pipeline behind it (arXiv:2503.06669).

Hardware

Three capture approaches recur across published systems, and each measures a different thing.

VR headsets with hand and head tracking, such as the Apple Vision Pro used in Humanoid Everyday, extract wrist and finger keypoints from onboard cameras without external infrastructure, but they track only the upper body that the headset’s own cameras can see, leaving torso and leg motion to be inferred or ignored (arXiv:2510.08807).

IMU-based motion capture suits, such as the Virdyn suit used to drive a Unitree G1 and the Noitom Perception Neuron Studio used in wearable legged-manipulation teleoperation, place inertial sensors across the body, 16 sensors tracking 19 body segments plus hand segments in the Noitom case, updating at 100 Hz with a stated resolution of 0.02 degrees. IMU suits capture full-body pose including legs and torso without needing line of sight, at the cost of accumulated inertial drift that a system has to correct for over a session (arXiv:2605.12347, Frontiers in Robotics and AI).

Optical motion capture, using external cameras tracking markers or unmarked features, is reported in prior teleoperation and biomechanics literature as more accurate than wearable IMU capture at the positions it can see, but it requires a fixed camera rig and line of sight to every tracked point, which is why it appears more often in controlled lab validation than in fleet-scale collection (STT Systems).

Software

• dex-retargeting: The open-source library behind the finger-retargeting approach used in Humanoid Everyday and related pipelines, mapping tracked hand keypoints onto a target hand’s own joint configuration (GitHub).

• Pinocchio: The rigid-body dynamics library used to solve inverse kinematics from tracked wrist poses to arm joint targets in Humanoid Everyday’s pipeline (documentation).

• MuJoCo: The physics simulator used to validate IMU-driven whole-body control before zero-modification deployment to a physical Unitree G1 (mujoco.org).

State of Practice

Published systems have converged on splitting the pipeline into a capture stage, a retargeting stage and a controller stage, and on treating retargeting as the place where morphology mismatch has to be resolved rather than left to the controller. Human2Humanoid’s morphology-invariant loss and Teleopit’s hand-agnostic retargeter both target that same stage, from opposite ends of the body (arXiv:2606.03476, arXiv:2608.01834).

Where practice has not converged is capture hardware and latency budget. AgiBot World runs two capture methods side by side on the same fleet because neither dominates the other, VR mode is faster to set up and constrains the hand to safe, predefined gestures, while motion capture mode is slower to don but records fine finger motion (arXiv:2503.06669). On latency, ExtremControl’s reported reduction from roughly 200 milliseconds to roughly 50 milliseconds by removing joint-space retargeting from the control loop is a result from one system under its own benchmark tasks, not yet a figure the field has adopted as a standard target. Of the five other systems surveyed here, all route through full-body inverse kinematics or an optimization-based retargeter rather than ExtremControl’s direct extremity mapping, so its latency reduction has not yet been reproduced outside the system that reported it (arXiv:2602.11321).

Pain Points

• Operator fatigue and availability bound collection throughput. A human has to physically perform every demonstrated motion, so scaling a corpus scales the number of operator hours required, and AgiBot World’s use of more than 100 robots in parallel is one response to that ceiling rather than a way around it (arXiv:2503.06669). Out of scope for Kotwel, which works downstream of collection.

• Latency changes what gets recorded, not just how it feels to operate. When the control loop lags, an operator compensates, overshooting a reach or pausing before a grasp, and that compensation gets recorded as if it were the intended motion. This is a concrete failure mechanism, not a general complaint: it is a direct consequence of the roughly 200 millisecond baseline latency ExtremControl reports for conventional pipelines (arXiv:2602.11321). Out of scope for Kotwel to fix in hardware, but the specification question of how to flag a likely latency-compensated segment sits squarely in annotation.

• Operator intent and controller correction share one joint stream with no default provenance field separating them. This is the failure named above in How Whole-Body Teleoperation Works in Practice, and it is where Kotwel’s work applies.

Kotwel’s Role

The specification question here is concrete: when a recorded episode shows a sudden joint correction, was that the operator adjusting a grip, or the balance controller catching a near stumble, and where in the pipeline should that distinction be logged so a downstream annotator does not have to guess from video alone. A pipeline that only stores the post-controller joint command has already discarded the answer. Two readings of what to do about it are both defensible: tag corrections after the fact by inspecting torque or velocity anomalies in the executed stream, which needs no change to the collection rig but is a lossy reconstruction, or instrument the retargeting and controller stages to emit their pre-correction and post-correction targets separately, which is exact but requires the collection team to change what a recording captures before a single demonstration is collected. Kotwel works on the data layer question of which fields a labeling specification needs to carry through a pipeline like this so that operator intent stays separable from controller correction after the fact, on engagements where that pipeline already exists. Kotwel does not build teleoperation rigs or balance controllers.