Physical AI
How learned intelligence moves into physical action, why state transitions make robot data difficult, and how the field shifted robot behavior out of engineered code and into datasets and learned parameters.
Summary
EgoMimic is a 2024 robot learning framework from Georgia Tech that trains one manipulation policy on two sources at once: egocentric human video with 3D hand tracking, and teleoperated robot demonstrations. It matters in Physical AI because it is a published test, on three real tasks, of treating human hand data as demonstration data in its own right instead of as pretraining or planning input, and because its headline result compares hours of data rather than reporting one success rate.
Overview
EgoMimic: Scaling Imitation Learning via Egocentric Video was written by Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman and Danfei Xu. Seven of the eight authors list the Georgia Institute of Technology and Chen Wang lists Stanford, according to the paper. It was submitted to arXiv on 31 October 2024, and later work cites it as an ICRA 2025 paper.
The status is research code. The public repository holds the data processing and training code for both Aria human data and robot teleoperation data, and its README lists human and robot data files for the three evaluated tasks hosted on Hugging Face under the gatech organization (checked 3 October 2026). The paper also says the robot’s bill of materials will be released. This note did not verify the repository license or the hardware documents.
What EgoMimic Does
The paper describes four components that together form what its authors call a full-stack framework.
Human capture. The system builds on Project Aria glasses, which the paper gives as 75 g. It uses the front RGB camera for observation and two side scene cameras for device pose and hand tracking, so hands stay tracked when they leave the RGB view. Aria’s cloud Machine Perception Services return device pose from SLAM, hand positions and other estimates, which the authors convert into a robomimic training file.
A low-cost robot. Two 6-DoF ViperX 300 S arms with RealSense D405 wrist cameras are mounted on a height-adjustable rig as a torso, and the rig costs under $1,000 excluding the arms, per the paper. A second pair of Aria glasses serves as the robot’s main camera, which removes the camera hardware difference between human and robot data.
Cross-domain alignment. Hand and end effector trajectories are transformed into the camera frame at the current timestep. Poses and actions are Gaussian normalized separately for each source. Both the hand and the robot arm are masked with SAM and overlaid with a red line to narrow the appearance gap. Human data is also slowed by a factor of 4, so a one second human horizon matches a four second robot horizon, with human data recorded at 30 Hz and robot data at 50 Hz.
A unified policy. The architecture extends Action Chunking with Transformers with a shared encoder and transformer plus shallow input and output heads for each source. A pose loss supervises both human and robot data, while a joint action loss supervises robot data only. The robot is driven in joint space because Cartesian control hit singularities on the 6-DoF arms.
The evaluation covers three long-horizon tasks: continuous object-in-bowl, laundry folding and grocery packing, with 45, 40 and 50 evaluation rollouts respectively. In the paper’s results, EgoMimic improved task score over Action Chunking with Transformers by 34 to 228%, with the largest gain on the bowl task (39 points to 128). Success rates rose by 8 to 33 points over that baseline. Against EgoMimic trained without human data, score gains were 10 to 88% and success rate gains were 2 to 15 points. On an unseen shirt color the baseline held 25% success while EgoMimic held 85%, and in a new scene with no new robot data EgoMimic scored 63 points against 4 for the MimicPlay baseline.
The scaling result comes from the bowl task alone. EgoMimic trained on 2 hours of robot data plus 1 hour of hand data scored 128, against 74 for the baseline trained on 3 hours of robot data. One hour of human collection yielded 1,400 demonstrations while one hour of robot collection yielded about 135, according to the paper. The authors also note that EgoMimic at 2 hours of robot data already outperformed the baseline at 2 hours, so part of the gap comes from architecture rather than from human data.
Physical AI Relevance
The paper’s argument is about where human data enters the pipeline. Earlier approaches the authors cite use human video to train a high level planner that conditions a low level policy trained on robot data alone. EgoMimic instead puts human and robot samples through one network, and the authors read the new scene result (63 points against 4 for their reimplementation of the hierarchical baseline) as evidence that a shared representation generalizes better than a planner and policy split. That is one comparison on one task, so it supports the design choice without settling it.
It also shows what the human side of the data consisted of in this one system. The action signal for human data is hand pose from a perception service, so the pipeline the paper describes carries no human written labels. In that pipeline, pose quality, SLAM accuracy, timestamp alignment and mask quality set the quality of the training signal directly, which is the practical content behind Human Video Data, Egocentric Data and Hand Pose Estimation for Robot Learning. Whether other egocentric capture stacks need the same four checks was not tested here.
The collection speed advantage depends on the task. The paper’s data collection table shows 23 human demonstrations per minute against 2 for the robot on the bowl task, but only 2 against 1 on groceries, where the authors attribute the slower human collection to resetting time.
Connections
EgoMimic uses Project Aria for human capture and as the robot’s camera. It uses Human Video Data, Egocentric Data and Robot Demonstration Data as its two training sources, and uses Behavior Cloning in the form of an action chunking policy. It uses Hand Pose Estimation for Robot Learning outputs as its human action signal. It is related to Teleoperation Interfaces for Data Collection because its robot data comes from a leader and follower rig, to Data Scaling Laws in Imitation Learning because both study how data composition changes policy performance, and to Ego4D, which the authors cite as the kind of large scale egocentric collection their framework is meant to merge with.
Limitations
Every result here comes from one lab, three tasks and one purpose built robot platform with human like arm kinematics, with 40 to 50 rollouts per task. The tables read for this note report single values without confidence intervals, so small gaps, such as 30% against 28% success on groceries, cannot be separated from noise from the reported data alone. The policies are single task, and the authors list generalizing to new robot embodiments and to behaviors shown only in human data, such as folding pants, as future work, so those cases are untested.
The scaling trend was measured on the bowl task at one data mix. Whether one hour of hand data outweighs one hour of robot data on other tasks, or at larger robot data volumes, was not tested in the paper. The comparison baseline for MimicPlay was reimplemented by the authors on the same transformer backbone without goal conditioning, so it differs from the original system. The ablations cover masking, action normalization and hand data, and do not isolate the effect of matching the camera hardware.
Human data carries no grasp information, because Aria records hand pose only. In the appendix, grasping is supervised only through the robot joint loss. The authors’ failure examples include missing the bag handle in the groceries task.
The 400% figure needs separate care. Meta’s blog post on the work says the lead author “achieved a 400% increase in his robot’s performance across various tasks vs previous methods” with “just 90 minutes of Aria recordings,” and Meta’s Aria Gen 2 pilot dataset paper repeats it. In the arXiv text read for this note, including the appendix, the largest relative score gain is 228%, the introduction says up to 200%, and the human data collected per task is 60, 80 and 100 minutes, none of which is 90. A follow-up check on 2026-10-11 found no protocol behind the figure in the other places named. The arXiv listing shows a single version (v1, 31 October 2024), so there is no later arXiv revision to differ. The project page contains no percentages and no 90 minute figure, only the scaling comparison of 2 hours of robot data plus 1 hour of hand data against 3 hours of robot data. The ICRA 2025 camera ready was not opened, so it remains unchecked. The blog’s wording compares against unnamed “previous methods,” not the paper’s action chunking baseline, so it may rest on a different experiment or metric. This note does not establish which, and the figure is best not repeated as a result of the paper.
Later work such as EgoScale cites EgoMimic as prior work on scaling imitation with egocentric human data and positions its own contribution around scaling laws and a larger data volume.
Links
Paper abstract: https://arxiv.org/abs/2410.24221
Paper PDF: https://arxiv.org/pdf/2410.24221
Project page: https://egomimic.github.io/
Meta blog post: https://ai.meta.com/blog/egomimic-project-aria-georgia-tech-ego4d-robotics-embodied-ai/
Code repository: https://github.com/SimarKareer/EgoMimic
Kotwel’s Role
Kotwel is the data-quality and data-operations partner for Physical AI. We turn multimodal physical-world and synthetic data into synchronized, structured, validated, and traceable datasets for training, evaluation, and continuous improvement. We make consequential data choices explicit and develop measurable quality criteria that enable organizations to benchmark and improve the data behind physical AI systems.
Scope and responsibility
Kotwel works at the data layer and remains independent of hardware, sensing architecture, and model family. We define data-quality criteria, audit datasets, and assess conformance against documented requirements. Clients retain control over program-specific conventions, intended model behavior, and operating requirements, provided those choices are explicitly documented and consistently applied. We do not build robots or autonomy stacks, develop control systems, or train foundation models. Decisions about system safety, legal compliance, and deployment remain with the client and its qualified advisers.
EgoMimic’s human data has no gripper state, so grasp is supervised by robot data alone. A team extending the format faces a specification choice the paper does not test. One reading derives a binary grasp signal from finger pose and supervises it on the human side, which risks teaching the policy a closure threshold that belongs to a hand rather than a gripper. The other leaves grasp to robot hours, which keeps the human signal clean but leaves the stage where the paper’s own failure examples sit, grasping a bag handle, supervised by robot data only. Kotwel puts that choice in front of the team that owns the policy before labeling starts, and does not decide the grasp definition on their behalf.
At a glance
Key takeaways
EgoMimic co-trains one policy on egocentric human hand data and teleoperated robot data, and reports gains over an action chunking baseline on three real tasks.
The reported relative score gains over that baseline are 34 to 228%, and the scaling claim rests on one task at one data mix.
The 400% and 90 minute figures in Meta’s coverage could not be traced to the paper’s text.
Human data in this pipeline carries machine estimated hand pose and no grasp signal, so tracking quality and grasp definition are the open data questions.
FAQ
Is one hour of human video worth more than one hour of robot data?
In this paper, on the bowl task, 2 hours of robot data plus 1 hour of hand data scored 128 against 74 for a baseline trained on 3 hours of robot data. The authors note that part of the gap comes from architecture, and the comparison covers one task and one data mix, so it is a measured result under those conditions and not an exchange rate.
Did EgoMimic improve robot performance by 400%?
Meta’s blog post says so, but the paper text read for this note reports a largest relative score improvement of 228% over its action chunking baseline, on the bowl task. This note could not trace the 400% figure to a protocol in the paper.
Does EgoMimic require Project Aria glasses?
The paper’s system is built on them for hand tracking, device pose and as the robot’s main camera. It does not test other capture devices, so whether the method transfers to them is unknown.
