Reinforcement Learning for Robotics
How rewards turn robot experience into a control policy, why offline data and simulation reduce costly hardware interaction, and where reinforcement learning still breaks on ambiguous success signals.
Missing Modality Robustness
How models learn to operate when one sensor disappears, why absence must remain distinct from delay or corruption, and what evaluation exposes dependence on a dominant channel.
Summary
Missing modality robustness is the ability of a multimodal model to keep producing useful, bounded outputs when an expected input is absent or unusable. It matters in Physical AI because cameras, lidar, radar, force signals, audio, and proprioception do not fail in the same way or at the same time. Robustness comes from representing availability explicitly, training on realistic failure patterns, and measuring each failure condition separately rather than hiding it inside one average score.
Missing Modality Robustness: Definition and Scope
Missing modality robustness covers model behavior when one or more expected channels are absent at training time, inference time, or both. The broader scope also distinguishes complete absence from stale, delayed, corrupted, occluded, or contradictory observations, because each condition carries different information and calls for a different response. It complements [[Sensor Fusion]] and [[Multimodal Evaluation]]: fusion combines signals, while robustness asks what remains trustworthy when that combination breaks.
How Missing Modality Robustness Works in Practice
Consider a mobile robot whose perception model consumes an RGB frame, a lidar sweep, and wheel odometry for obstacle detection. Each sample includes the three observations, their timestamps, and an availability mask. During training, the pipeline sometimes withholds one complete modality, masks selected regions, and injects failure patterns that resemble what the deployed sensors can produce. Separate encoders create representations for the available inputs, and a gated fusion module chooses which representations contribute to the prediction. The output contains obstacle classes, locations, and confidence values, plus a record of which modalities were present.
In an illustrative failure case, suppose the lidar packet arrives 120 milliseconds after the camera frame but is marked merely as available. The interval is a premise of the example, not a recommended rejection threshold. The fusion layer treats two different moments as one scene and may place a moving obstacle behind its current position. This is not the same as missing lidar. A properly designed pipeline preserves the timestamp and quality state, then either aligns the packet, rejects it, or routes the sample through a camera and odometry path. The downstream safety controller can then apply a policy tied to the declared degradation state instead of trusting an apparently complete observation.
Training approaches include modality dropout, masked reconstruction, quality aware routing, distillation from a full modality teacher, and generation of a representation for the absent input. The original ModDrop study randomly dropped channels while preserving modality specific representations. MultiMAE later masked patches across modalities and trained the model to reconstruct multiple outputs. These methods address different goals: dropout teaches useful partial paths, while reconstruction encourages cross modal predictive structure. Neither by itself proves safe behavior under an unseen hardware failure.
Use Cases
Autonomous driving perception is a research active use case. Camera and lidar fusion models are tested against complete sensor loss, restricted fields of view, occlusion, and degraded lidar beams, but public benchmark performance does not establish production safety.
Warehouse and mobile robot navigation is deployable when a degraded mode has explicit operating limits. A robot can slow, stop, or fall back to a narrower sensor path when a channel fails, but the fallback must be validated in the environment where it will run.
Manipulation is mainly research stage for learned policies that combine vision, force, tactile sensing, and proprioception. Missing contact signals can be confused with no contact unless the data contract carries a separate availability state.
Human robot interaction has research and limited production uses. Gesture, audio, gaze, and pose inputs are often incomplete because of occlusion, noise, or privacy controls. ModDrop was evaluated on multimodal gesture recognition and reported meaningful predictions with one or more missing signals under that study’s conditions.
Companies
Waymo studies multimodal end to end driving models. Its 2024 EMMA disclosure describes camera and text inputs as research and identifies adding lidar and radar encoders, long video context, inference efficiency, and verification as open challenges.
Google Research released LANISTR, a research codebase that uses unimodal and multimodal masking objectives to learn from tabular, text, and image data with missing modalities.
EPFL VILAB released MultiMAE as an academic research project with pretrained models and code for single modality and multimodal transfer.
These examples are research systems or research releases. They show active methods, not a common production standard for degraded sensor operation.
Datasets
nuScenes supports camera, lidar, radar, inertial, and positioning research. Its official site lists 1,000 scenes of 20 seconds each, one lidar, five radars, six cameras, an inertial measurement unit, and GPS as of the full 2019 release. Robustness studies can remove or corrupt selected streams while retaining synchronized ground truth.
MultiBench provides a research benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas in its published repository description. Its breadth helps compare robustness methods beyond one sensor suite, though some included tasks fall outside Physical AI.
Multimodal AMR Dataset records RGB, thermal, radar, ultrasonic, time of flight, and laser scanner data from an autonomous mobile robot. It is useful for studying availability and quality across different sensing technologies, subject to the failure labels provided with each sequence.
Papers
Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion, 2025. The CVPR paper routes object queries among camera, lidar, and fused experts under camera drop, lidar drop, occlusion, limited field of view, and lidar degradation on nuScenes-R.
Deep Multimodal Learning with Missing Modality: A Survey, 2024. The survey separates missing modalities during training and testing and reviews methods, applications, datasets, and open challenges.
MetaBEV: Solving Sensor Failures for 3D Detection and Map Segmentation, 2023. The ICCV study evaluates six corruption conditions and two complete missing sensor conditions on nuScenes.
MultiMAE: Multi-modal Multi-task Masked Autoencoders, 2022. The ECCV work uses sparse visible patches from several modalities to learn reconstruction and cross modal transfer.
ModDrop: Adaptive Multi-Modal Gesture Recognition, 2015. The paper introduces random channel dropping during gradual fusion to preserve partial input paths.
Hardware
Robustness begins with hardware observability. Sensor drivers need timestamps, sequence counters, health indicators, and calibration identifiers so the model can distinguish no reading from a valid zero. Edge compute must keep the fallback path within its latency and memory budget. A model that can route among several experts in a data center may be infeasible on a mobile controller unless the routing reduces rather than multiplies active computation.
Software
PyTorch supports custom masks, stochastic channel removal, and separate modality encoders. MMDetection3D provides open source components for camera and point cloud detection research. The nuScenes devkit exposes synchronized sensor records for controlled corruption and dropout experiments. None of these tools defines the availability contract, failure taxonomy, or acceptance threshold for a deployment.
State of Practice
Across the methods examined here, four recurring patterns are visible: separate modality encoders before controlled fusion, exposure to incomplete inputs during training, explicit routing among available representations, and reporting for named failure states. ModDrop established randomized channel removal, MultiMAE showed masking across inputs can support cross modal prediction, and MetaBEV and MoME introduced routing structures designed for camera and lidar failure cases. This small set spans gesture recognition, general visual representation learning, and autonomous driving perception; it does not establish adoption across all multimodal systems.
The reviewed literature has not established one failure distribution, one missingness representation, or one acceptable degradation curve. The 2024 survey identifies absence caused by sensor limits, cost, privacy, and data loss, which are operationally different mechanisms. The 2025 MoME paper tests several adverse sensor failures, but its results remain tied to autonomous driving perception benchmarks. Deployed systems also need health monitoring, fallback rules, and [[Closed-Loop Policy Evaluation]], because an accurate perception score does not say how a robot behaves after confidence falls.
Pain Points
Encoding missingness as a zero tensor collapses a failed channel, a padded record, and a physically valid zero into one value. Kotwel can address the data specification and annotation distinction, while model architecture remains out of scope.
Training dropout may not match deployed failures. Independent random masks do not reproduce burst loss, clock drift, occlusion, thermal saturation, or shared power faults. Kotwel could help define and label the observable failure states; hardware fault diagnosis remains out of scope.
Contradictory modalities lack a universal ground truth rule. A camera may show an obstacle while lidar returns no points because of reflectance, field of view, or timing. Deciding which observation is invalid requires context rather than majority voting. Kotwel can surface these ambiguous cases for owner decisions.
Privacy controls can create structured absence. Audio, faces, locations, or operator identifiers may be removed for compliance, leaving missingness correlated with people, sites, or events. Kotwel can make redaction provenance explicit, but legal interpretation belongs to the data owner and counsel.
Aggregate metrics hide brittle dependence. A model can perform well with all inputs while failing when its dominant modality disappears. [[Multimodal Evaluation]] should report each subset, corruption class, and contradiction case separately.
Kotwel’s Role
Kotwel is the data-quality and data-operations partner for Physical AI. We turn multimodal physical-world and synthetic data into synchronized, structured, validated, and traceable datasets for training, evaluation, and continuous improvement. We make consequential data choices explicit and develop measurable quality criteria that enable organizations to benchmark and improve the data behind physical AI systems.
Scope and responsibility
Kotwel works at the data layer and remains independent of hardware, sensing architecture, and model family. We define data-quality criteria, audit datasets, and assess conformance against documented requirements. Clients retain control over program-specific conventions, intended model behavior, and operating requirements, provided those choices are explicitly documented and consistently applied. We do not build robots or autonomy stacks, develop control systems, or train foundation models. Decisions about system safety, legal compliance, and deployment remain with the client and its qualified advisers.
One annotation edge case occurs when a camera frame shows an obstacle but the aligned lidar sweep contains no return in that region. One defensible reading labels lidar as missing or invalid because timing, occlusion, or reflectance may explain the absence; another labels the empty return as valid negative evidence because the sensor record passed its health checks. The first reading teaches the model to rely on the camera during a declared lidar failure, while the second teaches it to treat cross sensor disagreement as information. Kotwel can surface the trigger, preserve the timestamps and health metadata, and route the two readings to the program owner before labeling scales. Kotwel does not choose the sensor truth rule, diagnose the hardware, or build the fusion model.
Opportunities
A reusable modality availability schema could separate absent, delayed, stale, corrupted, redacted, and contradictory inputs across collection tools. Failure focused review sets could sample by duration, co occurrence, environment, and downstream consequence rather than by random frame count. Both opportunities depend on a clear [[Robot Failure Mode Taxonomy]] and reliable [[Multimodal Dataset Synchronization]].
At a glance
Key takeaways
Missing modality robustness is a family of explicit degraded input behaviors, not one score.
Availability, delay, corruption, and contradiction must remain separate in the data contract.
Training with dropout or masking helps only for the failure patterns and tasks that the evaluation actually covers.
Production use requires model robustness, sensor health evidence, and a bounded fallback policy.
FAQ
Is modality dropout enough to make a robot robust?
No, modality dropout covers only the missingness patterns represented during training. Burst failures, stale packets, contradictions, and correlated sensor faults require separate tests and may require routing or fallback logic.
Should a missing sensor value be replaced with zero?
No, a zero value should not stand for absence unless zero is impossible in that field and the contract states the convention. A separate availability mask and reason code preserve the difference between missing data and a valid measurement.
How should robustness be measured?
Measure the task metric for each declared modality subset and each corruption condition, then report the degradation from the full input condition. Also measure calibration, latency, abstention, and closed loop behavior where they affect deployment.
Can one modality be reconstructed from another?
Sometimes a model can predict a representation of an absent modality from the inputs that remain. The prediction is an inference, not a recovered measurement, so downstream systems should retain its provenance and uncertainty.
