What has to hold before two modalities agree on which of their parts refer to the same thing, why a robot policy scores highly on instructions it is actively contradicting, and where that correspondence comes from the labeling rather than the architecture.
Cross-modal alignment is the problem of working out which parts of one signal correspond to which parts of another: which words of an instruction refer to which region of an image, which moment of a video matches which sentence of a narration, which contact event in a force trace belongs to which frame. It exists because a system taking in several kinds of data at once has to know what refers to what before it can use any of them together. In Physical AI it decides whether a policy is following the instruction it was given or executing the action the scene makes visually plausible. Those two behaviors produce the same offline score, which is why alignment is usually discovered late.
The canonical definition comes from the multimodal machine learning survey of Baltrušaitis, Ahuja and Morency (2017), which names alignment as finding relationships and correspondences between sub-components of instances from two or more modalities. That survey places alignment as one of five core challenges alongside representation, translation, fusion and co-learning, and it separates two forms. Explicit alignment is when the correspondence itself is the output, such as matching recipe steps to segments of an instructional video. Implicit alignment is when correspondence is a latent intermediate step inside some other task, which is what happens inside every attention layer of a Vision-Language-Action Models policy.
Cross-modal alignment is not Multimodal Dataset Synchronization, and conflating the two is the most common scoping error. Synchronization is the time axis at the recording layer: several devices, several clocks, one agreed timeline. Alignment is the semantic correspondence a model learns or an annotator states on top of that timeline. Synchronization is a precondition, because correspondence learned over a misaligned corpus is learned against the wrong pairs and is still learned confidently.
Take a language-conditioned manipulation policy. In go RGB frames, proprioceptive state, and an instruction tokenized into a short sequence. Inside, cross-attention layers score every instruction token against every image patch, and those attention weights are the alignment: nothing in training supervises them, because the only loss is action reconstruction against the recorded trajectory. Out comes an action chunk in the robot’s action space, typically several future timesteps of joint or end effector commands plus a gripper value.
The mechanical failure is that attention mass collapses onto sink tokens, the sequence-initial and punctuation positions that carry no content, so the action is predicted from pixels and proprioception while the instruction contributes nothing. Held-out loss does not move, because the recorded action is still the action the scene implies. The measurement is direct: under semantically impossible instructions, an attention diagnostic published in March 2026 found pi-zero still completing the visually plausible action in 76 to 96 percent of trials, pi-0.5 in 88 to 99 percent, and OpenVLA-OFT in 82 to 99 percent, with linguistic grounding scores near zero. A policy that scores 90 percent on a contradicted instruction is not partially grounded. It is not reading the instruction at all.
The corpus is why. If every trajectory containing a red block is captioned “pick up the red block” and no trajectory in the corpus is captioned to contradict it, the word “red” is statistically inert. The model has no gradient telling it that the word does work, so it learns it as decoration.
Language-conditioned manipulation. Research stage moving into pilots. Alignment between instruction spans and scene entities is what makes an instruction changeable at run time rather than a label on a fixed skill.
Vision and touch in contact-rich manipulation. Research stage. ViTacFormer couples a cross-attention encoder over high-resolution vision and touch with an autoregressive tactile prediction head, reporting roughly 50 percent higher success than prior systems and autonomous execution of up to 11 sequential stages over 2.5 minutes on an anthropomorphic hand.
Learning from human video. Research stage. Aligning an egocentric view to a simultaneously recorded exocentric view is what lets a demonstration recorded from a human’s head be reused as supervision for a robot that sees the scene from elsewhere.
Cross-embodiment pooling. Research stage with early production use. The Motion Transfer mechanism in Gemini Robotics 1.5 exists to learn from heterogeneous multi-embodiment robot data, which is an alignment problem across bodies rather than across senses.
Corpus search and curation. Production ready as tooling. Joint embeddings are what make “find every episode where someone opens a drawer” a query rather than a labeling project.
OpenAI. Research release. CLIP, published in February 2021, trained on 400 million image and text pairs and established contrastive pairing as the default way to align two modalities.
Meta AI. Research. ImageBind binds images, text, audio, depth, thermal and IMU into one space and showed that only image-paired data is required to bind all six, which removes the need for every pairwise combination to exist.
Physical Intelligence. Research to deployment pilots. pi-0.5, published April 2025, co-trains robot data with web data, object detections and semantic subtask prediction, and reports that this transfer is essential for generalization. Its policies are among those measured above as ignoring contradictory instructions.
Google DeepMind. Research to early production. Gemini Robotics 1.5, October 2025, is a multi-embodiment VLA with an explicit embodied reasoning model that plans in natural language before acting.
Open X-Embodiment. 60 pooled datasets from 21 institutions covering 22 embodiments, over 1 million trajectories, 527 skills and 160,266 tasks. Instructions were contributed per dataset and analyzed afterwards with PaLM to extract objects and behaviors. The paper states that coordinate frames were not aligned across datasets, so the standardization is of format rather than of meaning.
Ego-Exo4D. 1,286 hours from 740 participants across 13 cities, with simultaneously captured egocentric and exocentric video, eye gaze, IMU, camera poses, multiple paired language descriptions and expert commentary. It is the closest thing available to explicit alignment supervision at scale, because the correspondence between views is recorded rather than inferred.
CLIP’s web image and text corpus. 400 million pairs, described in the CLIP paper. Not released, but it remains the reference point for what scale of weak pairing the contrastive approach assumed.
RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models, June 2026. A benchmark built specifically because action-accuracy scores cannot separate grounded success from ungrounded success.
Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration, March 2026. Source of the contradiction success rates above, and of an inference-time fix that cuts success under contradiction to 36 to 76 percent while leaving normal task performance within half a percentage point.
Gemini Robotics 1.5, October 2025.
CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models, August 2025. Queries a VLM for instructions that could have been executed at other points in existing trajectories, and reports roughly double the instruction-following success of the same policy trained on standard labels, with no new data collected.
ViTacFormer, June 2025, revised May 2026.
pi-0.5: a Vision-Language-Action Model with Open-World Generalization, April 2025.
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning, 2022.
Learning Transferable Visual Models From Natural Language Supervision, 2021.
ImageBind, the reference implementation for binding additional modalities to an image anchor.
RoboSemanticBench, released with the benchmark above for running grounding diagnostics against a policy.
The field has converged on contrastive pairing as the default mechanism and on the finding that a single anchor modality is enough to bind the rest. What it has not converged on is whether the resulting space is aligned in any measurable sense. Liang et al. (2022) showed that image and text embeddings in models like CLIP occupy separate cone-shaped regions rather than interleaving, held apart both by random initialization and by the contrastive objective itself, and that deliberately changing the size of that gap changes downstream zero-shot accuracy and fairness. “Shared embedding space” therefore names a training objective rather than a verified property, and practitioners inherit an alignment whose quality nobody measured.
In robotics the distance between published results and deployed practice is sharper still. Published instruction-following numbers are reported on evaluations where the instruction and the scene agree. Diagnostics that hold the scene fixed and vary the instruction find that leading policies score above 76 percent on instructions they are contradicting (March 2026), and RoboSemanticBench exists in June 2026 precisely because no standard benchmark separates the two cases. Contradiction testing is not yet part of standard published VLA evaluation, so reported task-success results alone do not establish whether a policy is following its instruction or merely executing the action implied by the scene.
On the data side, the unsettled question is how to produce labels that discriminate. Open X-Embodiment standardized storage format across 60 datasets and explicitly left coordinate frames and action semantics to the contributor, which means the pooled corpus inherits whatever labeling convention each lab used. The CAST result, that relabeling existing trajectories with counterfactual instructions roughly doubles instruction following without collecting anything new, points at where the ceiling actually sat. It was in the labels.
Faithful captions that never contrast. The concrete failure mechanism: every episode is described correctly, no episode’s description distinguishes it from its neighbors, and attributes such as color, side and object identity carry no gradient. The corpus is accurate and teaches nothing about the words in it. Kotwel addresses this at specification time.
No alignment ground truth. The 2017 survey named scarce annotated alignment data as the first difficulty and it remains true. Frame-level or span-level correspondence is expensive to produce and almost never budgeted for. Kotwel could address this.
The defect is invisible offline. Detecting it requires closed-loop rollouts with deliberately contradicted instructions. Building and running that evaluation harness is out of Kotwel’s scope, although constructing the contradiction set is data work.
Synchronization errors that look like alignment errors. If a force channel lags video, the correspondence the model learns is genuine and wrong, and no amount of relabeling fixes it. Kotwel addresses this through Multimodal Dataset Synchronization checks before annotation begins.
Anonymization that covers only one channel. Human video corpora carry faces, voices and home interiors, and the paired language description frequently names the person, the place or the employer. De-identifying the video while leaving the caption intact re-identifies the episode through the very channel alignment work depends on. Kotwel addresses this.
The guideline stops giving one answer whenever a recorded episode admits more than one true description. An operator picks up the only block on the table and the instruction log says “pick up the red block”. The annotator can caption what happened, which is the most specific true statement about that episode, or caption to discriminate, which means writing the description that separates this episode from its neighbors and therefore requires knowing what the rest of the corpus contains. Both readings are defensible and they teach different things: faithful captioning yields a corpus where the word “red” is inert because nothing contradicts it, while discriminative captioning yields labels whose correctness is relative to a batch and which stop being right when the corpus is resampled or merged with another lab’s. Public practice has not settled this, and CAST is a bet on the second reading strong enough to double instruction-following success on unchanged trajectories. Kotwel’s work here is to put the question in front of the team before collection rather than after training: which attributes the corpus is supposed to teach, and which contrasting episodes or counterfactual labels have to exist for each one to carry signal. Kotwel builds and audits the label sets and the collection specification. It does not train policies and it does not run the closed-loop evaluation that confirms the fix landed.
Cross-modal alignment is the correspondence between sub-components of two signals, and it is distinct from synchronization, which only guarantees they share a clock.
In language-conditioned robot policies the alignment is implicit, unsupervised and easy to skip: measured success rates above 76 percent on contradicted instructions mean the policy is reading pixels and ignoring words.
The defect is invisible in offline metrics because the recorded action is still the action the scene implies, so it survives every held-out evaluation and appears on hardware.
Contrastive training does not guarantee an aligned space: image and text embeddings sit in separate regions held apart by the objective itself, and nobody measures the gap by default.
The strongest reported fix is a labeling change rather than a modeling change, which puts the ceiling on instruction following in the annotation specification rather than in the architecture.
No. Synchronization puts every stream on one agreed clock, and alignment decides which parts of those streams refer to the same thing. Synchronization is a precondition: a corpus whose force channel lags its video by tens of milliseconds will still train an alignment, and that alignment will be confidently wrong. Fix the clocks first, then ask what corresponds to what.
Run the policy on paired trials that hold the scene and initial state constant while changing only the instruction. Include satisfiable controls alongside contradictions involving the requested object, attribute, spatial relation, or action, then compare whether the policy’s behavior changes with the language. If a policy continues completing the visually obvious action under semantically impossible instructions, the language pathway is not controlling the action. In the March 2026 diagnostic, pi-zero, pi-0.5, and OpenVLA-OFT completed the visually plausible action in more than 76 percent of contradicted trials.
Relabel first. CAST roughly doubled instruction-following success by generating counterfactual instructions over trajectories that already existed, collecting nothing new. Collection is the expensive lever and it is often the wrong one, because more episodes captioned the same undiscriminating way add volume without adding contrast.
Scale has not removed the problem. ImageBind showed that binding six modalities through a single image anchor works, and the modality gap result showed that the spaces those models learn keep the modalities separated by construction, with the separation partly set by initialization and the loss temperature rather than by the data. Larger models inherit the same objective, so what changes with scale is capability, not whether the correspondence was ever verified.
