Autonomous Systems

Summary

Autonomous systems are machines that pursue a goal in the physical world with limited or no human intervention, deciding for themselves what to do as conditions change. They span road vehicles, delivery aircraft, mining and yard equipment, marine vessels, and robots, and what unites them is not the form factor but how much of the decision and fallback responsibility the machine holds. The field has moved from research demonstrations to metered commercial service, with millions of paid driverless rides and hundreds of millions of autonomous miles now on record. The central tension is that autonomy is judged on rare events, and rare events are the thing a fleet collects least of and labels worst.

Autonomous Systems: Definition and Scope

An autonomous system perceives its environment, decides on a course of action toward a goal, and executes that action without a human approving each decision. The definitional weight sits on the fallback rather than on the perception: what matters is who is responsible when conditions leave the envelope the system was designed for.

The most precise public grading of that boundary is SAE J3016, revised April 2021, which grades driving features rather than vehicles across six levels, and which NHTSA summarizes for the public. At Level 2 the human remains responsible for monitoring the environment even while the feature controls speed and steering. At Level 3 the system takes over that monitoring and the human intervenes only when asked. At Level 4 the system handles the fallback itself inside a defined Operational Design Domain, and Level 5 removes the domain boundary. The step from Level 2 to Level 3 is where engineering responsibility and legal responsibility both move, which is why it is the only level distinction most practitioners need to argue about, and Philip Koopman’s practitioner guide to J3016 is blunt that this is the point the public discussion most often gets wrong.

In scope for this node: road vehicles, uncrewed aircraft, mining, agricultural and construction equipment, warehouse and yard vehicles, marine vessels, defense platforms, and any robot that acts without a human in the loop. Out of scope: teleoperated machines, where a person supplies the decisions continuously, and fixed sequence automation such as a caged industrial arm replaying a taught path, where there is no environmental decision to make. Teleoperation Data is nonetheless central to the field, because supervised operation is how most autonomous programs collect the experience they later learn from.

Industry usage and standards usage differ, and the gap causes real confusion in procurement. Industry commonly calls anything with a perception stack “autonomous”. The standards usage ties the word to a stated operating domain and a stated fallback owner. When a claim of autonomy arrives without both, it is a capability claim rather than an autonomy claim.

How the Field Arrived Here

The pattern the table reveals is a repeated substitution: at every step that mattered, a rule someone wrote was replaced by a distribution someone recorded, and each substitution raised the volume and quality of data needed to know the system actually worked.

Why Now

Sensing and compute are no longer the only binding constraints. Waymo’s sixth generation Driver, which began fully autonomous operations in February 2026, uses fewer than half the cameras of the previous generation while adding 17 megapixel imaging, lidar with better weather penetration and custom silicon, in a configuration Waymo describes as driving costs down. As sensing and compute become more capable and cost-efficient, a larger share of the deployment challenge moves to what the fleet has encountered, how reliably those events were recorded, and whether they can be turned into trustworthy training and evaluation data.

One trained model now spans perception, reasoning and control. A survey of vision-language-action models for autonomous driving published in June 2025 compares more than twenty representative models that integrate visual perception, language understanding and control in a single policy. The practical consequence is that capability now tracks training data rather than module engineering, which is the same shift Vision-Language-Action Models produced in manipulation.

Deployed fleets became the primary source of rare events. Waymo reported driving more than 4 million autonomous miles every week as of June 2026, and Baidu reported more than 220 million fully driverless autonomous kilometers accumulated as of May 2026. At that exposure the fleet, rather than a dedicated collection campaign, is what encounters the situations a model has not seen.

Regulation turned into a described process. In its AV framework update of 31 July 2026, NHTSA extended its Automated Vehicle Exemption Program to domestically manufactured vehicles, streamlined the Standing General Order on crash reporting effective June 2026, and proposed modifications to Federal Motor Vehicle Safety Standards 102, 103, 104, 110 and 135 for vehicles without conventional manual controls. The same notice records 517 ADS crash reports received between June and December 2025.

Commercial pull arrived from outside passenger transport. Aurora reported nearly 440,000 driverless miles across ten routes as of the end of June 2026, and Zipline’s 125 million autonomous miles, reported in January 2026, were flown in bounded aviation corridors. Freight and logistics do not need a general solution, only a reliable bounded one, so their economics closed earlier.

None of this solved the problem RAND named in 2016. Cheaper sensing, a single model and a large fleet all improve the average case. Deployment is decided by the tail, and the tail is not primarily a modeling problem. It is a question of whether the rare event was captured intact, aligned across sensors, typed consistently, and still retrievable months later.

Autonomous Systems Compared to Robotics

The row that causes the others is the environment. Because an autonomous system operates in a space its operator does not control and cannot enumerate in advance, the people at risk are often third parties, the regulatory surface extends beyond machine safety, and deployment evidence increasingly depends on rates measured over real-world exposure. Learned robotics is also moving toward statistical and closed-loop evaluation, but autonomous systems make that evidence problem especially consequential because failures occur in open environments shared with people who are not part of the operation. The two categories are perspectives on the same machine rather than disjoint sets: a warehouse mobile robot is studied as robotics and governed as an autonomous system, and the closer a robot gets to open environments the more the second framing decides what it takes to deploy it.

Why Autonomous Systems Matters

Take a driverless truck on the Dallas to Houston lane Aurora opened in May 2025. On a night run the truck encounters a stationary vehicle on the shoulder with a person walking around its rear toward the travel lane, in light rain, on a stretch where the lane markings are worn. The system slows, moves over and passes wide. Nothing goes wrong. The event is logged, and it is exactly the kind of event the next model should learn from.

For that to happen, several things have to be true of the recording. Every stream captured at that moment, meaning the camera set, the lidar sweeps, radar returns, vehicle state, the planned trajectory, the executed control, map context and weather, has to sit on one shared time base. The person and the stationary vehicle have to be labeled the way every comparable instance in the corpus was labeled. The situation has to be typed under a scenario taxonomy someone can query months later. And the outcome has to be recorded as a judgment, meaning “handled correctly”, not merely “no collision occurred”.

Here is what goes wrong. The camera stream and the lidar sweeps carry timestamps from different clock domains that have drifted roughly 60 milliseconds over the shift. At highway speed that is close to two meters of travel. The 3D box drawn from the lidar is therefore associated with a camera frame taken two meters further down the road, so the labeled pedestrian sits slightly off the pixels that show them. The offset is small enough to pass frame by frame review and consistent enough to survive a training run without looking like noise. The files are complete, the loss falls, and offline detection metrics improve. What the model actually absorbs is a small systematic bias between where an object appears and where it is, and that bias is worst precisely where geometry changes fastest, at the moment of closest approach. This is the failure mode Multimodal Dataset Synchronization exists to prevent, and no amount of additional mileage corrects it, because every additional mile carries the same offset.

The second failure costs even more. Months later a safety case requires every night time encounter with a pedestrian near the shoulder. Two annotation vendors typed this scenario differently, one as “pedestrian, roadside” and the other as “vulnerable road user, ingress”. The query returns a fraction of the events that exist, the team concludes the situation is rarer than it is, and the next training mix underweights it. The event was captured, paid for, and stored, and it still did not reach the model.

That is the practical version of the RAND result. A fleet driving more than 4 million autonomous miles a week, as Waymo reported in June 2026, does not have a data volume problem. It has a data trustworthiness problem, and the two failures above are what it looks like from the inside.

Industry Perspective

Leading programs disagree in public about almost every architectural question. Waymo builds multi-sensor stacks and, in its sixth generation Driver, invested specifically in lidar and radar performance in rain and snow. Tesla runs camera-only perception and, according to CNBC reporting from 22 January 2026, began removing in-car safety supervisors from some Austin robotaxis. They disagree on how large a domain to claim: Aurora deliberately bounded itself to ten Sun Belt freight routes as of June 2026, Waymo runs geofenced metropolitan service areas and reported safety results across five operating territories as of June 2026, and Zipline flies defined aviation corridors. They disagree on how supervision is withdrawn, whether through an in-vehicle monitor, remote assistance, or a trailing vehicle.

What they converge on is less discussed and more revealing. Every one of these organizations publishes an exposure denominator rather than a capability claim, and each built the measurement apparatus before it built the market. Waymo publishes location matched and time matched human benchmarks and, in research published on 7 July 2026, argues that not all miles are equal, reporting that its fleet drives overnight roughly four times more than the average driver and that 53% of the injury crashes it avoided fell in those overnight hours. Baidu reports fully driverless ride counts and driverless kilometers inside financial disclosure. Aurora reports driverless miles per route in investor releases. Zipline reports cumulative autonomous miles alongside delivery counts. NHTSA converged from the opposite direction, running a standing crash reporting order that produced 517 ADS incident reports between June and December 2025.

The inference those independent behaviors support is that the field has settled on recorded exposure and its analysis, rather than architecture, as the thing that demonstrates autonomy. Whether a stack is camera-only or multi-sensor, modular or end-to-end, the artifact that gets a program approved and keeps it running is the dataset behind the claim: sensor records that are time aligned, events typed consistently enough to be counted, and outcomes labeled well enough that a rate can be computed and defended to a regulator. Waymo’s own argument that a benchmark has to match when and where the miles were driven is a data curation problem before it is a statistical one. Organizations that disagree about everything else are spending on the same underlying infrastructure.

Kotwel’s Role

Kotwel does not build autonomy stacks, operate fleets, or hold a position on sensing architecture. That boundary is deliberate. The data failures described above occur identically in a camera-only program and a lidar program, and a data partner tied to one architectural bet would be less useful to teams that made a different one.

Kotwel works at the data layer where raw fleet records become datasets that autonomy teams can reliably train on, evaluate against, and audit. The operation begins by preserving the context and provenance of incoming multimodal records, then validating synchronization across related sensor and state streams before annotation compounds an upstream error. Scenarios, objects, actions, outcomes, and failures are structured under consistent taxonomies, with quality assurance applied across sequences rather than isolated frames. The resulting records are versioned and delivered with enough provenance to trace a training or evaluation example back to the source event and the decisions made during its preparation.

The objective is to ensure that recorded experience, especially rare events, edge cases, and failures, remains findable, comparable, and trustworthy as models and datasets change.

Key Concepts

Physical AI is the broader move of learned models into perception, reasoning and action on physical machines, of which autonomy is the deployment case with the highest evidentiary bar.

Robotics is the engineering discipline that supplies the machines, sensing and control that autonomous systems run on.

Embodied AI is the academic framing of intelligence developing through a body acting in an environment and observing the consequence.

Multimodal AI covers joining camera, lidar, radar, state and language signals into a single interpretation, which every autonomy stack does by necessity.

Operational Design Domain is the stated set of conditions a system is designed to handle, and the boundary that makes a Level 4 claim meaningful or empty.

LiDAR provides direct range measurement whose accuracy in poor visibility is the crux of the sensing disagreement between programs.

Multimodal Dataset Synchronization aligns sensor, state, action and annotation streams on a shared time base, and is the failure this article’s worked example turns on.

Trajectory Data records the planned and executed paths that make an autonomous decision reviewable after the fact.

Scenario Mining retrieves rare situations from fleet logs, which only works when the taxonomy behind it was fixed before the labeling started.

Closed-Loop Policy Evaluation tests a system through repeated perception, action and feedback, exposing compounding errors that offline metrics conceal.

Sim to Real Transfer studies how behavior learned in simulation carries to hardware, and where the reality gap breaks the transfer.

Synthetic Data for Robotics extends scenario coverage beyond what physical exposure can reach, on the condition that its coverage claims are validated against real events.

Data Flywheel is the loop by which deployment events become the next training corpus, which requires outcome labels and failure typing to be instrumented at collection.

Vision-Language-Action Models condition action on visual observation and language in a single trained model, and are the architecture pulling driving and manipulation toward the same data requirements.

Further Reading

Primary sources

1. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?, Nidhi Kalra and Susan M. Paddock, RAND RR1478, 2016.

2. ALVINN: An Autonomous Land Vehicle in a Neural Network, Dean A. Pomerleau, NeurIPS 1988.

3. A Survey on Vision-Language-Action Models for Autonomous Driving, arXiv:2506.24044, 30 June 2025.

4. Automated Vehicles for Safety, NHTSA, carrying the agency’s public description of the SAE J3016 levels of driving automation.

5. AV Framework Updates and Request for Comments on Interim Guidance, NHTSA, Federal Register, 31 July 2026.

6. Waymo safety data update, Waymo, 24 June 2026, covering data through 31 March 2026.

7. Not All Miles are Equal: Why Time and Location Matter When Benchmarking Autonomous Safety, Waymo, 7 July 2026.

8. Beginning fully autonomous operations with the 6th-generation Waymo Driver, Waymo, 12 February 2026.

9. Baidu Announces First Quarter 2026 Results, Baidu, 18 May 2026.

10. Aurora Begins Commercial Driverless Trucking in Texas, Aurora Innovation, 1 May 2025.

11. Aurora Launches Second-Generation Driverless Trucks in U.S., Aurora Innovation, 22 July 2026.

12. The DARPA Grand Challenge: Ten Years Later, DARPA, 2014.

13. Zipline Surpasses 2 Million Deliveries, Zipline, 21 January 2026.

14. Waymo One: The next step on our self-driving journey, Waymo, 5 December 2018.

Industry and strategy

1. The Prometheus project: The story behind one of AV’s greatest developments, ADAS and Autonomous Vehicle International, covering the 1986 project start and the 1995 Munich to Copenhagen drive.

2. Musk says Tesla taking safety supervisors out of some Robotaxi vehicles in Austin, CNBC, 22 January 2026.

3. SAE J3016 User Guide, Philip Koopman, Carnegie Mellon University, a practitioner reading of the level definitions. The standard itself, SAE J3016 revised April 2021, is sold by SAE rather than published openly.