Two identical robot arms receive different abstract instruction signals but both reach for the same blue cube, while a gold correspondence marks the intended target in the right-hand trial.

Cross-modal Alignment

What has to hold before two modalities agree on which of their parts refer to the same thing, why a robot policy scores highly on instructions it is actively contradicting, and where that correspondence comes from the labeling rather than the architecture.

Cross-modal Alignment Read More »