What a generated episode contains once its pixels or its actions come from a model rather than a sensor, why the mixing ratio against real data decides whether co-training helps, and where a corpus stops recording which episodes were measured.
Synthetic data for robotics is training data a machine produced rather than a sensor recorded: trajectories replayed and adapted inside a physics simulator, camera frames rendered or restyled by a generative model, and whole episodes predicted by a video model that never touched hardware. It exists because recorded robot data is bounded by how many hours people can spend driving robots, while a generation pipeline is bounded mainly by compute. It matters for Physical AI because a generated episode and a recorded one are indistinguishable once both sit in the same training format.
Synthetic data for robotics covers any training example whose observations, actions, or both were computed rather than measured. Three provenance classes sit under that one label, and they differ in which part of the episode is trustworthy.
Simulated execution. Actions come from a planner or a transform of a recorded demonstration and a physics engine executes them, so both channels are computed but stay consistent with the simulator’s model of contact. MimicGen and RoboCasa work this way.
Generative augmentation. An existing recording keeps its measured actions while its frames are restyled by a generative model. NVIDIA’s Cosmos Transfer step does this.
Neural trajectories. A video model generates the observation stream and the actions are recovered afterwards by an inverse dynamics or latent action model, so no action in the episode was ever commanded (DreamGen, May 2025).
Whether a policy trained on any of this survives contact with hardware is the transfer question, and it belongs to Sim2Real. This node is about the artifact: what a generated episode contains, how it is mixed with recorded data, and what a corpus records about where each episode came from.
NVIDIA’s synthetic manipulation motion generation pipeline runs the full sequence. An operator teleoperates a robot’s digital twin in Isaac Lab through an Apple Vision Pro hand tracking stream, GR00T-Mimic extrapolates that seed set by annotating keypoints and interpolating between them, Cosmos Transfer restyles the resulting frames, and behavior cloning trains on the augmented set. NVIDIA reports 780,000 synthetic trajectories generated in 11 hours, which it puts at the equivalent of 6,500 hours of human demonstration, and 84 percent success averaged over 50 trials on a stacking task trained from 1,000 of them, about 40 percent better than the same task trained on its real data alone (NVIDIA Technical Blog, 18 March 2025).
What comes out is an episode file with observations, actions and timestamps, in the schema a teleoperated recording would occupy. What does not come out is any statement of which stage produced which channel. In an augmented episode the action stream is what the simulator executed and the pixels are model output, and nothing in the record separates it from the pure simulation episodes beside it or from real episodes imported later. The consequence lands at training time: the mixing ratio the team believes it set is not the ratio it set, and that ratio is not a free parameter. A mechanistic study of sim-and-real co-training found a balanced band of roughly 0.016 to 0.3 across the manipulation tasks it examined, with fully overlapping representations producing negative transfer rather than a smaller gain (Lei et al., arXiv:2604.13645, April 2026).
Multiplying manipulation demonstrations from a small human seed. Production tooling: MimicGen generated more than 50,000 demonstrations across 18 tasks from roughly 200 human ones (Mandlekar et al., 2023), and now ships as an NVIDIA reference workflow.
Co-training a real-world policy on mostly simulated batches. Research stage: across six tasks on a Panda arm and a GR-1 humanoid, real data alone averaged 45.3 percent success against 81.1 percent when co-trained with simulated data (Maddukuri et al., arXiv:2503.24361, April 2025).
Generating driving scenarios and sensor streams. Deployed research program: the Waymo World Model generates camera and lidar output under driving action, scene layout and language control (Waymo, February 2026). Retrieving the real counterparts is Scenario Mining.
Teaching behaviors nobody demonstrated. Research stage: DreamGen reported a humanoid performing 22 new behaviors in seen and unseen environments from teleoperation data covering a single pick-and-place task in one environment (Jang et al., 2025).
NVIDIA. Builds Isaac Lab, the synthetic manipulation motion generation blueprint, and the Cosmos world foundation models. Production tooling (Cosmos 3 release, 31 May 2026).
Waymo. Runs generative driving simulation built on Google DeepMind’s Genie 3 inside a deployed autonomy program rather than as a released product (Waymo, February 2026).
Skild AI, Agile Robots, Doosan Robotics, LG Electronics, Samsung Electronics. Named as robotics adopters in the Cosmos 3 release, which reports no deployment stage for any of the five (NVIDIA, 2026).
UT Austin and NVIDIA research groups. Authored MimicGen, RoboCasa and the co-training recipe cited here. Research origin now inside shipped tooling (RoboCasa).
RoboCasa: 100 tasks in kitchen scenes with thousands of 3D assets across more than 150 object categories, meshes from text-to-3D models and textures from text-to-image models (June 2024). The assets are generated, not only the trajectories.
MimicGen datasets: more than 50,000 generated demonstrations across 18 tasks, spanning several scene configurations, object instances and robot arms, from roughly 200 source human demonstrations.
Neither is a recording in the sense Robot Demonstration Data describes. Both are the output of a generator whose configuration, rather than an operator, decided what each episode contains.
A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies (Lei et al., 2026)
DreamGen: Unlocking Generalization in Robot Learning through Video World Models (Jang et al., 2025)
Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation (Maddukuri et al., 2025)
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (Nasiriany et al., 2024)
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (Mandlekar et al., 2023)
Isaac Lab and Isaac Sim: the generation stack behind the blueprint above, which pins Isaac Lab 2.0.2 and Isaac Sim 4.5.0 and lists an RTX A6000 with 48 GB for local generation.
NVIDIA Cosmos: open world foundation models used for video augmentation, neural scene reconstruction and defect image generation.
LeRobotDataset v3.0: the format many of these corpora land in. Its documentation describes per-episode records as lengths, tasks and offsets, and names no field for whether an episode was recorded or generated. Format detail sits in LeRobot.
Multiplying manipulation demonstrations from a small human seed has converged into standard tooling: what was published as a research system in 2023 (MimicGen) is now a supported NVIDIA reference workflow with pinned simulator versions and a hardware specification. What that convergence rests on is narrow. The published results here concentrate on tabletop and kitchen manipulation with grippers, and the blueprint’s own figures come from a stacking task.
How much generated data to mix has not converged, and the two studies surveyed here that report a mixing ratio do not point the same way. The co-training recipe found that for one humanoid cup pick-and-place task the best sampling ratio put 99 percent of each training batch on simulation data (Maddukuri et al., 2025), while the later mechanistic analysis found a balanced band of roughly 0.016 to 0.3 across its own tasks and identified representation alignment rather than the ratio itself as the larger driver (Lei et al., 2026). Both are single-study results on different task sets, which together suggest the ratio is worth measuring per task family rather than carrying over.
Validation of generated corpora against real data is where published method runs out: none of the sources surveyed for this node sets out a quantitative protocol for it. Waymo’s February 2026 description of its world model sets out generation and control capability in detail and describes no procedure for checking generated sensor output against recorded output (Waymo, 2026). The navigation finding covered in Sim2Real, that a lower fidelity and more heavily randomized simulator transferred better than a higher fidelity one, is a reminder that visual realism and training value are separate axes.
Provenance is a naming convention rather than a field. In the LeRobot v3.0 schema above, whether an episode was recorded or generated is carried by the dataset name and the team’s memory, so a corpus assembled from three sources cannot be re-split by provenance later without re-deriving it from file paths. Kotwel works on this where a corpus is specified and accepted.
Simulator ground truth is the engine’s convention, not an annotation guideline. A physics engine can emit an exact box for an almost entirely occluded object because it knows where the object is, while a human annotator working to a visibility rule would drop it. Merging the two shifts the label distribution for occluded objects with nothing reporting it. Kotwel could address this by writing both conventions into one specification before the merge, adjacent to Point Cloud Annotation.
Inferred actions are consumed as if they were measured. In a neural trajectory the action labels come from an inverse dynamics model, so the training target carries that model’s error and no field records the substitution (DreamGen, 2025). That is a measurement question rather than a labeling one and sits outside what Kotwel does.
The specification stops giving one answer the moment a real teleoperated episode is passed through a generative augmentation step and written back into the same corpus. Its action stream is measured, recorded from the operator’s commands at the rig, and its pixels are model output. A team typing that episode as real preserves the fact that its actions were commanded rather than inferred, then reports a mixing ratio that undercounts how much of what the policy sees is generated. A team typing it as synthetic gets an honest visual accounting and loses the distinction between an action that was executed and one recovered by a model, which is the distinction the evaluation later depends on. Kotwel works on this as a specification question: naming the provenance classes an episode can belong to before generation starts, writing the acceptance rule for each into the collection standard alongside the rules in Demonstration Collection Protocol, and checking that the field survives into the delivered corpus. Kotwel does not generate synthetic data, tune simulators, or select mixing ratios; those belong to the team training the policy.
The ratio in a training config and the actual composition of a batch are distinct quantities; reconciling them requires episode-level provenance. A per-episode provenance schema with a stated acceptance rule for each class, applied at ingest and checked at delivery, is a data layer artifact close to the corpus specification work Kotwel already does on recorded data. Judging whether generated data is realistic is a separate job needing the simulator and the hardware.
Synthetic data for robotics spans three different artifacts: simulated execution, generative augmentation of real recordings, and neural trajectories whose actions were inferred rather than commanded.
Generation ratios in published pipelines are large: NVIDIA reports 780,000 trajectories in 11 hours, and MimicGen more than 50,000 demonstrations from roughly 200 human ones.
Mixing generated with recorded data raised average success from 45.3 percent to 81.1 percent across the six tasks in one co-training study, and the useful ratio differed by task in both studies covered here.
Common training formats, including LeRobotDataset v3.0, document no per-episode field for whether an episode was recorded or generated.
Among the sources surveyed for this node, none sets out a quantitative protocol for validating a generated corpus against real data, and visual realism is not a substitute for one.
For the tasks in the studies covered here, no: all of them keep real demonstrations in the mixture. The co-training recipe used 20 to 100 real demonstrations per task alongside 1,000 to 10,000 simulated ones, and reported real data alone reaching 45.3 percent average success against 81.1 percent for the mixture across six tasks (Maddukuri et al., 2025). Whether a purely generated corpus suffices outside tabletop and kitchen manipulation is not settled by the work surveyed here.
Published ratios in these two studies run from roughly 10 to 500 simulated episodes per real one, and they disagree on where the useful band sits. One reports 20 to 100 real demonstrations against 1,000 to 10,000 simulated per task, with a 99 percent simulation sampling ratio best for one humanoid task (Maddukuri et al., 2025); the other finds a balanced band of roughly 0.016 to 0.3 across its tasks (Lei et al., 2026). It depends on how closely the simulated scene matches the real one, so measure it per task.
In the formats covered here you cannot tell from the episode itself and have to ask the supplier. LeRobotDataset v3.0 documents per-episode records as lengths, tasks and offsets with no provenance field, so the information lives in dataset naming or a README rather than in the data. The fix is to require a provenance value per episode in the delivery specification before collection or generation starts.
