Physical AI
How learned intelligence moves into physical action, why state transitions make robot data difficult, and how the field shifted robot behavior out of engineered code and into datasets and learned parameters.
Robot Demonstration Data
What a single demonstration actually contains once it leaves the teleoperation rig, why a corpus assembled from many operators can teach a policy less than one assembled from few, and where the real cost of a trajectory sits.
Summary
Robot demonstration data is a recording of a person completing a task using a robot’s own body: what the robot saw at each moment, and what its joints and gripper were commanded to do at that same moment. It exists because text and ordinary video do not directly provide the commands that produced a robot’s motion, while a demonstration records observations and actions together in a form a policy can learn from. It is often among the most expensive sources of robot training data because every example consumes an operator’s time, physical hardware, and a real collection environment. It also has an unusually direct effect on whether a policy works, since the recorded behavior defines what the policy is being trained to reproduce.
Robot Demonstration Data: Definition and Scope
Robot demonstration data is a set of time-indexed episodes in which a human drives a robot through a task while the robot’s observations and its own commanded actions are recorded together. Each episode is a trajectory: timesteps carrying camera frames, proprioceptive state such as joint positions and gripper width, the action command issued at that step, and usually a language description of the task. The defining property is that actions sit in the robot’s action space rather than in human terms, which makes an episode a directly usable supervised target for Behavior Cloning and for the Vision-Language-Action Models built on top of it.
Scope here is data produced by teaching a robot, through teleoperation, kinesthetic guidance, or handheld devices that record in a robot-compatible frame. It excludes Human Video Data, which records people and carries no action labels, and Synthetic Data for Robotics, where the trajectory is generated rather than performed.
How Robot Demonstration Data Works in Practice
Take a bimanual insertion task on an ALOHA style rig. The operator holds two small leader arms; two larger follower arms track their joint positions while four webcams record at 480×640, two on the follower wrists and two positioned front and overhead. Teleoperation and recording both run at 50 Hz, and the station was built within a 20,000 USD budget (Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, 2023). One recorded attempt produces an episode of a few hundred timesteps, each holding four images, a 14-dimensional joint position vector for the two arms and their grippers, and the leader position that served as the action target. Fifty episodes, roughly ten to twenty minutes of recorded data, was the per-task budget in that work. Episodes are written into a format such as LeRobotDataset v3, which stores state and action as Parquet shards, camera frames as MP4 shards, and episode boundaries as metadata offsets.
Spatial Multimodality Versus Velocity Multimodality
Now suppose two operators recorded those fifty episodes. Both are competent and every episode succeeds. One completes the approach in about a second; the other, more cautious about the peg, takes several times longer along a nearly identical path. Pooled, the corpus pairs near-identical wrist images with per-step joint deltas that differ by a large factor, and an action head trained to predict the next chunk from that observation regresses toward the average of the two. On hardware the arm reaches the peg with less momentum than the insertion needs and stalls at contact. Nothing is mislabeled and no offline metric catches it, because the held-out split contains the same two speeds. This is the difference between spatial multimodality, where different paths are genuinely different valid strategies, and velocity multimodality, where the path is the same and only the speed differs. Recent work argues the second is noise rather than diversity, and reports that normalizing it away is worth about a 15% improvement, equivalent to 2.5 times more pre-training data (Is Diversity All You Need for Scalable Robotic Manipulation?, 2025).
Use Cases
Training a single-task policy. Production ready in narrow fixed settings. Fifty demonstrations per environment and object pair is the working figure, and returns saturate: with 32 such pairs, performance plateaus around 1,600 total demonstrations (Data Scaling Laws, 2024).
Post-training a pretrained policy. Production ready inside lab and pilot programs. Toyota Research Institute reports new tasks learned with 3 to 5 times less data than from scratch (TRI LBM, 2025).
Cross-embodiment pretraining. Research stage. Pooling many robots’ demonstrations produced positive transfer, with RT-2-X tripling performance on emergent skill evaluations (Open X-Embodiment).
Reference behavior for evaluation. Research stage, because scoring against demonstrations offline does not predict rollout behavior. That is why Closed-Loop Policy Evaluation is a separate discipline.
Feeding a Data Flywheel. Research stage outside a few fleets, and only workable if episodes are findable by condition rather than by filename.
Companies
Physical Intelligence: startup, research-stage deployment. Trained pi-0.5 on roughly 400 hours of mobile manipulator demonstrations across about 100 homes, with 97.6% of pre-training examples coming from elsewhere.
Figure AI: startup, pilot deployment. Helix used approximately 500 hours of multi-robot, multi-operator teleoperation, with language labels generated afterward by a vision-language model (20 February 2025).
Toyota Research Institute: corporate research lab. 468 hours of real bimanual teleoperation plus 45 in simulation, mixed with roughly 1,150 hours from Open X-Embodiment.
Google DeepMind: research lab. RT-1 used about 130,000 demonstrations across more than 700 instructions, collected over 17 months with 13 robots.
AgiBot and OpenDriveLab: startup and academic lab. Standardized collection with human-in-the-loop verification; policies trained on it averaged 30% better than on Open X-Embodiment.
Hugging Face: open tooling. LeRobot supplies the format and training stack most smaller-scale collection standardizes on.
Datasets
Open X-Embodiment: over 1 million trajectories pooled from 60 datasets, 22 embodiments and 21 institutions, covering 527 skills. The reason Action Space Standardization became an operational problem.
DROID: 76,000 trajectories and 350 hours across 564 scenes and 86 tasks, gathered by 50 collectors at 13 institutions over 12 months on Franka Panda arms teleoperated with an Oculus Quest 2.
AgiBot World: over 1 million trajectories across 217 tasks, notable for publishing its verification pipeline.
BridgeData V2: 60,096 trajectories across 24 environments on a low-cost platform, widely used as a generalization benchmark.
Humanoid Everyday: 10,300 trajectories across 260 tasks on Unitree G1 and H1 humanoids teleoperated through an Apple Vision Pro, recording RGB, depth, LiDAR, tactile and language together.
Papers
BifrostUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation (2026)
Is Diversity All You Need for Scalable Robotic Manipulation? (2025). Separates spatial from velocity multimodality.
A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation (2025). 1,800 real rollouts, and the finding that 50 rollouts give a confidence interval 20 to 30 points wide.
How to Train Your Robots? The Impact of Demonstration Modality on Imitation Learning (2025). Kinesthetic teaching gives the highest action consistency; mixing modalities beat either alone by about 20%.
Data Scaling Laws in Imitation Learning for Robotic Manipulation (2024)
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024)
Open X-Embodiment: Robotic Learning Datasets and RT-X Models (2023)
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023). The ALOHA rig.
Eliciting Compatible Demonstrations for Multi-Human Imitation Learning (2022)
Hardware
Leader and follower arm pairs are the workhorse of bimanual collection; ALOHA fits a full station inside a 20,000 USD budget by pairing a WidowX leader with a ViperX follower on each side. Virtual reality headsets dominate single-arm and humanoid collection: DROID standardized on an Oculus Quest 2, Humanoid Everyday on an Apple Vision Pro with hand keypoint retargeting. Handheld grippers such as UMI remove the robot from collection entirely, at the cost of latency matching before the data is usable. Kinesthetic teaching needs no interface hardware and yields the most action-consistent data, but its physical load rules it out at scale. Recording is not compute bound; storage and bus bandwidth for several cameras at 50 Hz are the binding constraint.
Software
LeRobot provides the dominant open dataset format and training stack, with v3 adding streaming so million-episode corpora train without a local copy. The Open X-Embodiment episode schema remains the interchange format for pooled data. robomimic supplies the reference offline learning implementations. ROS 2 is where raw streams are usually captured, which makes its time model the origin of most alignment problems downstream.
State of Practice
The field has converged on the shape of a demonstration. Teleoperation is the dominant modality, an episode is a synchronized observation and action stream at 20 to 50 Hz, actions live in the robot’s own space, and a handful of formats now carry almost everything published. What has not converged is anything about the corpus: how many episodes a task needs, who records them, and what an acceptable episode is.
The evidence is uncomfortable for anyone planning a collection budget. Diversity of environments and objects buys generalization on a power law while demonstrations per pair saturate, so recording more attempts at one station stops paying quickly (Data Scaling Laws, 2024). Meanwhile most published comparisons cannot resolve the differences they report: at 50 rollouts the confidence interval on a success rate is typically 20 to 30 percentage points wide, wider than most claimed improvements (TRI LBM, 2025).
Practice has responded by treating demonstrations as the scarce ingredient and everything else as filler. In pi-0.5, 97.6% of pre-training examples come from sources other than the target robot doing the target task (Physical Intelligence, 2025), and TRI sets 468 hours of its own teleoperation against roughly 1,150 hours from Open X-Embodiment. The demonstrations that decide whether a policy works are a minority of what it sees, which is what makes their quality expensive to get wrong.
Pain Points
Velocity multimodality across operators. Identical observations paired with per-step action magnitudes differing by a large factor, producing a policy that hedges toward an average speed and loses the momentum a dynamic phase needs. Invisible offline, because the validation split carries the same spread. Kotwel addresses this at the specification and acceptance stage.
Operator incompatibility more broadly. Two operators can adopt genuinely different strategies for one task, such as reaching down then across versus across then down, and pooling them degrades a policy that would learn either alone (Eliciting Compatible Demonstrations, 2022). Kotwel could address this by making strategy a recorded field rather than an accident of who was on shift.
Cost per trajectory, most of which is discard. DROID’s published totals divide out to roughly 16.6 seconds of retained interaction per trajectory, taking 350 hours across 76,000 trajectories as the denominator. Staging, resets, failed attempts and episodes dropped in review are not in that number. Cutting hardware cost is out of scope for Kotwel; cutting the fraction of collected episodes that never reach a training split is not.
Privacy and compliance on in-home collection. A corpus recorded across roughly 100 homes or 564 scenes captures bystanders, faces, documents and household contents alongside the task. Kotwel addresses this through anonymization and redaction, though the harder problem is that redaction after collection often removes context the policy needed.
Language annotation nobody audits. Automatic labeling with a vision-language model, as Figure uses for Helix, trades annotation cost for label noise with no established review protocol. Kotwel could address this, and the open question is what human verification rate is defensible rather than whether to verify.
Kotwel’s Role
The specification question arrives the first time a corpus contains episodes from more than one operator: when two successful episodes trace the same path at very different speeds, is the slower one valid data or is it noise. Both readings are defensible. Treated as valid, speed becomes variation the policy should be robust to, which is right when slowness is task-relevant, as around a fragile object or a tight tolerance. Treated as noise, the corpus is normalized to one velocity profile and the policy learns a crisper action distribution, which is right when the slowness is only operator caution, and there is published evidence this is usually the case (Is Diversity All You Need, 2025). Nothing in the recording distinguishes the two, because both look like a success, so the decision belongs in the collection protocol rather than in the post mortem after a rollout stalls at contact. Kotwel works at that stage: designing collection protocols, defining what an acceptable episode is, structuring the synchronized observation and action records, and running the quality checks that catch a corpus drifting from its own specification. Kotwel does not build robots, operate collection fleets, or choose the policy architecture the data is destined for.
At a glance
Key takeaways
Robot demonstration data provides one of the most direct training signals in the robot’s own action space, which makes it immediately usable for supervised policy learning and expensive to collect on physical hardware.
Diversity of environments and objects scales generalization on a power law while demonstrations per pair saturate, so more attempts at the same station stop paying.
Demonstrations are a small minority of what large policies train on, which raises rather than lowers the cost of quality problems in that minority.
Operator variation splits into spatial multimodality, which is useful, and velocity multimodality, which behaves as noise, and the two are indistinguishable in a raw episode.
Deciding which one you have belongs in the collection protocol, because no offline metric will surface it and a closed-loop failure will.
FAQ
How many demonstrations do we need for one task?
There is no universal demonstration count for one task. The requirement changes with task difficulty, embodiment, starting policy, environment variation, and the success rate a deployment must reach. In one 2024 scaling study, approximately 50 demonstrations were collected for each environment and object pair, and performance across 32 pairs showed little further improvement beyond roughly 1,600 demonstrations in total (Data Scaling Laws, 2024). The practical lesson is not that every task needs 50 demonstrations, but that increasing environment and object coverage may deliver more generalization than repeatedly recording the same task at one station.
Can we use human video instead of robot demonstrations?
Not as a direct substitute, because human video carries no action labels in the robot’s action space and imitation training needs those. Web and human video are useful as pretraining, and in pi-0.5 they are the large majority of the mixture, but the demonstrations teach the specific behavior. Handheld devices such as UMI are the middle path: a human performs the task, and the recording is already in a robot-compatible frame.
Does it matter that different people collected our data?
Yes, and it is one of the most underestimated variables in a collection program. Different operators adopt different strategies for the same task, and pooling incompatible ones degrades a policy that would learn either alone (Eliciting Compatible Demonstrations, 2022). The practical response is to record who collected each episode and under which protocol, so the question can be asked of the data later rather than guessed at.
Why does our policy look fine on the validation split and fail on the robot?
Because the validation split came from the same distribution as the training data, so any systematic property of the corpus sits in both. A velocity spread, a calibration change midway through collection, or a timing offset all survive the split intact and leave the validation number looking healthy. Only closed-loop rollouts surface them, and 50 rollouts is not enough trials to resolve a small difference.
