Physical AI
How learned intelligence moves into physical action, why state transitions make robot data difficult, and how the field shifted robot behavior out of engineered code and into datasets and learned parameters.
Summary
ABC-130k is an open source teleoperation dataset for bimanual manipulation, released by XDOF in June 2026 alongside the paper “Scalable Behavior Cloning with Open Data, Training, and Evaluation” from researchers at UC Berkeley, MIT, Amazon FAR and Carnegie Mellon University (arXiv:2606.27375). It matters in Physical AI as the graph’s first example of a large teleoperation corpus distributed in its native recording container, MCAP, rather than converted out of it into a training-ready format. XDOF’s launch announcement calls it the largest open source teleoperation dataset released to date, a comparison the announcement does not itemize against named alternatives.
Overview
ABC-130k is maintained by XDOF and hosted on Hugging Face, gated behind acceptance of usage conditions. The Hugging Face dataset card, checked 2 September 2026, lists 130,703 episodes totaling 3,590.7 hours across 195 bimanual manipulation tasks, with 42,980 episodes (about a third of the total) carrying subtask annotations, at a combined size of 22.6 TB. It is released under the Apache 2.0 license. The accompanying arXiv manuscript describes a related snapshot of the same collection effort at 134,806 episodes and 3,553 hours, with a 1,552 hour subset annotated, a difference consistent with the paper describing an earlier cut of a dataset that XDOF continued to grow after submission. The paper organizes the 195 tasks into seven primitive categories: pick and place, fine pick and place, folding, insertion and ejection, tool use, sorting, and tying and untying.
What ABC-130k Does
Each episode in ABC-130k is collected on a bimanual station built from two I2RT YAM arms, six degree of freedom manipulators with parallel jaw grippers, teleoperated by a human operator. Each station carries one overhead camera plus one wrist camera per arm. Two station hardware variants exist in the released data: RealSense stations, where the overhead camera is monochrome at 640×480 with every stream in that configuration encoded in H.264, and ZED-X stations, where the overhead camera is a stereo pair at 1920×1200 encoded in H.265 while the wrist cameras stay H.264. A station’s kinematic and collision model is distributed as a MuJoCo MJCF, which lets a downstream user simulate the exact station a given episode was recorded on rather than an approximation of it.
Each episode is stored as a directory holding episode.mcap, present for every episode, and annotation.mcap, present only for annotated episodes, both sharing one absolute time base. episode.mcap carries per-arm robot state on topics such as /left-arm-state and /right-arm-state, six joint angles in radians plus a flattened 4×4 end-effector pose, per-arm gripper aperture as a 0 to 1 value on paired state topics, one compressed video stream per camera, and camera calibration on sibling -info topics. annotation.mcap carries the task name on /instruction and subtask segment boundaries on /subtask-annotation, kept in a separate file on the same time base rather than merged into the recording it describes. The dataset card states that the video and state streams run on independent clocks and instructs nearest neighbor timestamp matching rather than index alignment when reconstructing a synchronized frame.
Physical AI Relevance
ABC-130k is the first large scale example in this graph of a teleoperation corpus released in its recording container rather than converted into a training-ready format such as RLDS or a LeRobot dataset. That choice makes the separable annotation file, rather than a merged record, the corpus’s provenance pattern: a consumer can tell from the file listing alone which episodes carry subtask labels without opening any file, and can add a new annotation pass as a new sibling file instead of rewriting the recording it describes. The mixed station hardware also makes ABC-130k a concrete case of what pooling teleoperation data across camera configurations costs: the RealSense and ZED-X halves of the corpus differ in overhead camera resolution, color depth, and codec, a difference a training pipeline has to handle explicitly, whether by resampling to one common resolution and codec or by treating the two station types as separate visual domains, rather than one it can pool without a decision either way.
Connections
ABC-130k uses MCAP as its per-episode container format. ABC-130k is_a Robot Demonstration Data. ABC-130k related Open X-Embodiment, the pooled cross-embodiment corpus that ABC-130k is not part of but is comparable to in scale and intent. ABC-130k related RLDS, the episode-and-step training format that a consumer would convert this corpus into rather than the format it ships in. ABC-130k generated_by Trajectory Data concepts recorded through the per-arm robot state topics.
Limitations
The dataset card reports annotation coverage of about a third of episodes by count (42,980 of 130,703, checked 2 September 2026); the remaining episodes carry no subtask labels, so they support training on the full episode but not on objectives that condition on subtask segmentation. The station hardware is not uniform across the corpus: RealSense and ZED-X stations differ in overhead camera resolution, color mode, and codec, a split a pooled training run has to notice and decide how to handle rather than one it can assume away. The manuscript notes its own collection was conducted inside a white walled, three sided enclosure to reduce visual variance during evaluation; transfer to uncaged, in-the-wild settings was untested at the time of writing. The paper also reports that validation loss did not correlate with real-world policy performance in its own experiments, evidence against treating validation loss as a reliable offline proxy in that setting, though whether the same holds for every training run on this dataset is untested.
Links
Kotwel’s Role
The two station hardware variants in ABC-130k raise a labeling specification question before a single frame is annotated: does a subtask boundary drawn against a 640×480 monochrome RealSense view hold at the same timestamp against the corresponding 1920×1200 stereo ZED-X view of a similar task, or does the higher resolution and color stereo pair change what an annotator can actually see at the moment of contact. Two readings are defensible: treat the annotation as a property of the task regardless of which station recorded it, or treat each station type as its own annotation context with its own review pass. Which one a team picks changes what a policy trained across both station types actually learns to condition on. Kotwel surfaces this kind of split early, before an annotation spec is written against one station type and silently applied to the other, and puts the decision in front of the team that owns the training run.
At a glance
Key takeaways
ABC-130k is a 130,703 episode, 3,590.7 hour bimanual teleoperation corpus (dataset card, checked 2 September 2026), released by XDOF in June 2026 under Apache 2.0 and distributed as per-episode MCAP files rather than a converted training format.
Two station hardware variants, RealSense and ZED-X, differ in overhead camera resolution, color mode, and codec, a split a pooled training run has to notice and decide how to handle rather than assume away.
Annotations live in a separate sibling file,
annotation.mcap, on the same time base as the recording, making the annotation pass a provenance pattern rather than a merged edit to the original data.About a third of episodes carry subtask annotations as of the dataset card’s September 2026 check; the remainder supports training on the full episode but not on subtask-conditioned objectives.
FAQ
What robot arms does ABC-130k use?
Each station uses two I2RT YAM arms, six degree of freedom manipulators with parallel jaw grippers, arranged as a bimanual pair.
Why does ABC-130k ship as MCAP files instead of a training-ready format like RLDS?
The dataset card distributes each episode as episode.mcap and, when annotated, a sibling annotation.mcap, both on the same absolute time base, so a consumer converts it into a training format such as RLDS or a LeRobot dataset rather than receiving one directly.
Are all episodes in ABC-130k annotated?
No. The dataset card, checked 2 September 2026, reports 42,980 of 130,703 episodes carry subtask annotations, about a third of the corpus by episode count.
