Scenario Mining
How rare situations are retrieved from fleet logs long after they were recorded, why the recall of a scenario query is almost never measured, and where two defensible readings of one scenario type quietly split a corpus.
Offboard Auto-labeling
What running a labeling model after the drive is over buys over doing it in real time, why two published pipelines report matching or beating average human accuracy on the same benchmark, and where that claim depends on how independently it was checked.
Summary
Offboard auto-labeling is the practice of generating 3D bounding box labels for a recorded LiDAR sequence after the drive is over, using a model that can look at every frame in the sequence at once rather than only the frames a vehicle has already driven past. It exists because a fully recorded sequence carries information a real-time onboard detector never has access to: the same object seen from many distances and angles, both before and after the moment being labeled. It matters for Physical AI because the pipelines built this way are now reported to match or exceed average human labeling accuracy on at least one public benchmark, which changes where the human labeling budget on a large driving corpus actually goes.
Offboard Auto-labeling: Definition and Scope
Offboard auto-labeling is a non-causal 3D detection pipeline: a model that generates bounding box labels for a recorded point cloud sequence with access to the entire sequence, past and future relative to any given frame, rather than to a causal window ending at the current frame. That is what the name refers to: the “board” is the vehicle, and the labeling model runs offboard, on a server, after the drive, rather than onboard on hardware that has to decide within a fixed latency budget while driving. Running offboard removes both constraints that define an onboard perception model: there is no latency budget to meet, and the model can use frames that have not happened yet relative to the frame being labeled.
The scope of this node is the labeling pipeline itself: the architecture that turns a raw point cloud sequence into a set of temporally consistent boxes, and the evidence for how those boxes compare to boxes a human draws. What a point actually measures belongs to [[LiDAR]]; the specification questions in drawing any single box, onboard or offboard, belong to [[Point Cloud Annotation]]; retrieving which parts of a long log are worth labeling at all belongs to [[Scenario Mining]].
How Offboard Auto-labeling Works in Practice
Take a roughly 20 second segment of about 200 LiDAR frames from the Waymo Open Dataset, run through the detect, track, refine design that 3DAL and DetZero share. First, a multi-frame 3D detector runs over every frame in the sequence and proposes a box for each object it sees in that frame, the same kind of detector an onboard system would run, applied to every frame in the log rather than only the most recent one. Second, an offline multi-object tracker links those per-frame boxes into tracks, one track per physical object across the whole segment; because this stage is not causal, DetZero couples it directly with the detector and evaluates it by how complete the resulting tracks are, since the paper’s authors found a standard onboard-style tracker still left many tracks fragmented even when running on a full recorded sequence. Third, every point that fell inside one track, across every frame the object appeared in, is gathered into a single object-centric point cloud in the object’s own moving reference frame; a refinement network runs over that aggregated cloud once, using this object-centric 4D representation to jointly correct the track’s box size, position, orientation and per-frame confidence in one pass, borrowing precision from the frames where the object was close and its return dense to correct the frames where it was far away or partly occluded. The output is a full track of refined boxes and a confidence score per box, rather than the disconnected frame-by-frame boxes a real-time detector alone would produce.
The failure worth naming sits in the second stage. DetZero’s authors report that an onboard-style tracker leaves many tracks fragmented even on a full recorded sequence, and one way to see how that happens is an illustrative case, not one taken from a paper: a car stopped behind a bus at an intersection is fully occluded for an extended stretch, and the offline tracker can split it into two disconnected track identities on either side of the occlusion, because nothing in the recorded points links them. The refinement stage then receives two short, low-context fragments instead of one long, well-observed track, and each fragment is refined using only the frames it was actually given, so the cross-frame context that offboard processing exists to supply never reaches the fragment that needed it most.
Use Cases
Generating training labels for onboard 3D detectors at fleet scale, where a small human-labeled seed set trains an offboard model that labels the rest of the fleet’s recorded sequences. Production in AV programs with the sensor fleet and offline compute to run it, research stage elsewhere.
Producing the released ground truth for a public dataset outright, rather than seeding a separate detector. The [[Waymo Open Motion Dataset]] states its object tracks are generated by an offboard auto-labeling model rather than by human annotators, which makes the auto-labeler’s output the dataset’s ground truth rather than an intermediate step toward it. Production.
Semi-supervised training of a smaller, real-time student detector on a mix of human labels and auto labels, cutting the volume of sequences that need a human pass before a new detector can be trained. Research stage, reported inside individual papers rather than as a standardized workflow.
Auditing an existing human-labeled corpus, by running the offboard pipeline over already-labeled sequences and surfacing the cases where the two disagree for review. Research stage; the disagreement is evidence, not automatically a correction.
Companies
[[Waymo]] built the original 3DAL pipeline and used it to annotate the Open Motion Dataset, publishing the auto-labeled output itself as the dataset’s ground truth. Production AV program.
Shanghai AI Laboratory, through its PJLab-ADG group, published DetZero, the pipeline reporting the first offboard result to outperform average human labels on the Waymo benchmark. Research lab, with a released open-source implementation.
Google Research applied the same offboard-to-onboard transfer to a semi-supervised setting, training a real-time student detector on a mix of human-labeled and pseudo-labeled Waymo run segments. Research.
Datasets
[[Waymo]] Open Dataset, Perception: the benchmark both 3DAL and DetZero train and evaluate on, and the corpus their human label comparison studies are drawn from.
Waymo Open Motion Dataset: its released object tracks and box positions are the direct output of an offboard auto-labeling pipeline rather than of human annotation, making it one of the clearest public examples of auto labels shipped as ground truth rather than as a training aid.
Papers
DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds, 2023. Reports the first offboard model to outperform average human labels on the Waymo benchmark. arXiv:2306.06023
Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection, 2023. Proposes a track-centric alternative to object-centric refinement, modeled on how experienced human annotators label a track. arXiv:2304.12315
Offboard 3D Object Detection from Point Cloud Sequences, 2021. Introduces the 3DAL pipeline and the human label study it was verified against. arXiv:2103.05073
Pseudo-labeling for Scalable 3D Object Detection, 2021. The teacher-student formulation for training a student detector on human labels plus offboard pseudo-labels. arXiv:2103.02093
Software
DetZero (PJLab-ADG): the reference implementation released alongside the paper. No public code for the other pipelines named here turned up in the sources read for this node.
State of Practice
Figures below are as read from the arXiv versions of each paper on 2026-10-09; the papers do not all state the same metric level, and each is noted where it matters.
Two claims about beating human labels are not the same claim, and the field’s own papers keep them distinct. [[Waymo]] describes 3DAL’s output as on par with human labels, verified through a human label study, a parity claim. DetZero goes further, reporting gains of 3.79 and 4.87 points of 3D AP at an IoU threshold of 0.7 for the vehicle class (L1), against human labels (86.45) and against 3DAL (85.37) respectively, on 5 sequences selected from the Waymo Open Dataset validation set (Table 3), which the paper describes as the first offboard result to outperform average human labels. A third paper, CTRL, working from a track-centric rather than object-centric design, reports surpassing human labels by 5.79 3D AP for vehicles at the stricter 0.8 IoU threshold, 66.28 against 60.49, on the same kind of 5-sequence validation subset (Table 6). In a separate study it asked annotators to relabel 1,000 hard vehicles in the validation split and compared the IoU distributions of the official boxes, the relabeled boxes and its own predictions, to check which was actually right; that study reports IoU distributions rather than AP. That last step answers an objection any of these results invites: a model trained on human labels and then scored against a human-labeled test set cannot straightforwardly be said to beat the thing it was trained to imitate, unless the comparison is against an independently checked reference rather than the same labels used for training and tuning. Relabeling hard cases and comparing, rather than trusting either side by default, is one way that circularity gets addressed, and it appears in one of the four papers surveyed here rather than as a standard step.
In the papers surveyed here, offboard output is used to reduce, rather than eliminate, human labeling volume; whether that holds across the wider field is not established here. DetZero’s own semi-supervised experiment trains a CenterPoint student on 10 percent human-labeled data (79 sequences) plus its own auto labels for the remaining 90 percent (719 sequences), reaching 74.44 vehicle AP and 78.92 pedestrian AP against a 100 percent human-labeled baseline of 75.41 vehicle AP and 77.51 pedestrian AP, where the 10 percent human-labeled data alone gives 66.88 and 67.13: a small shortfall on vehicles and a gain on pedestrians, close enough to call the auto-labeled mix competitive with the fully human-labeled set rather than a strict parity in both classes (DetZero, Table 4; the paper does not state the difficulty level). A separate Google Research pipeline reports gains of 5.4 AP for vehicles and 1.9 AP for pedestrians (74.0 against 68.6 and 69.8 against 67.9), as Level 1 AP on the Waymo Open Dataset test set with a PointPillars student, when a real-time student detector is trained on human labels plus about 75,000 offboard pseudo-labeled run segments, over the authors’ own baseline trained on the human labels alone. Both results start from a labeled seed set chosen at the beginning of a data collection round; how that seed set should be sampled to cover rare classes and edge cases is not something either paper addresses.
Pain Points
Track fragmentation under occlusion. The refinement stage is only as good as the track it is given, and an offline tracker can still split one object into two disconnected tracks across a long occlusion, which is the specific mechanism named above.
The circularity in every “beats human” claim. A model trained on human labels, evaluated against a human-labeled test set, needs an independently checked reference before “outperforms human labels” means what it sounds like it means, and only one of the papers surveyed here describes running that check.
Confidence scores that decide what becomes ground truth. The refinement stage assigns a confidence score to every box, and a downstream user has to pick a threshold that decides which boxes are kept; that threshold is tuned on whichever distribution of objects the validation set happened to contain.
Extended data retention. An offboard pipeline needs the full recorded sequence, including everything that happens after the moment being labeled, so raw fleet data has to be stored and replayed well past the point an onboard, real-time-only system would need to keep it, extending how long a program is exposed on privacy and retention grounds.
No shared disclosure standard for what “verified against human labels” means. 3DAL’s parity claim and DetZero’s superiority claim both rest on human label studies whose sampling and adjudication procedure is not standardized across papers, so two corpora both described as verified are not necessarily verified the same way.
Kotwel’s Role
Kotwel is the data-quality and data-operations partner for Physical AI. We turn multimodal physical-world and synthetic data into synchronized, structured, validated, and traceable datasets for training, evaluation, and continuous improvement. We make consequential data choices explicit and develop measurable quality criteria that enable organizations to benchmark and improve the data behind physical AI systems.
Scope and responsibility
Kotwel works at the data layer and remains independent of hardware, sensing architecture, and model family. We define data-quality criteria, audit datasets, and assess conformance against documented requirements. Clients retain control over program-specific conventions, intended model behavior, and operating requirements, provided those choices are explicitly documented and consistently applied. We do not build robots or autonomy stacks, develop control systems, or train foundation models. Decisions about system safety, legal compliance, and deployment remain with the client and its qualified advisers.
The edge case is what a corpus does with a track the offline tracker fragmented under occlusion, the specific failure named above. A fragmented pair left as two ground-truth objects asserts a break in identity that the physical object never had, so the corpus needs a rule for when two tracks separated by an occlusion gap are merged back into one object before it ships. That rule requires deciding how much positional and appearance continuity across the gap counts as the same object, and neither choice is visible in the box coordinates alone. Kotwel works this at the specification and audit layer: sampling fragmented tracks before a corpus ships, writing down the merge-or-split rule the program chooses for a reviewer to apply, and recording which rule a given release used, so a later user of the corpus knows what a track boundary actually asserts. Kotwel does not build the tracker or the refinement model that produces the auto labels.
Opportunities
Independent audit sample design for programs adopting an offboard pipeline for the first time, sized and stratified against the object classes and conditions the pipeline’s own published human label study did not cover.
A disagreement adjudication protocol as a deliverable: a written rule, applied consistently, for what happens when an auto label and a human reviewer disagree, so a corpus can state what its verification actually checked rather than only that verification happened.
At a glance
Key takeaways
Offboard auto-labeling runs after a sequence is recorded, using the whole sequence, trading an onboard model’s latency budget for full temporal context that lets it correct a box drawn early in a sequence using frames that come later.
The detect, track, refine design that 3DAL and DetZero share is multi-frame detection, offline tracking, then track-level refinement, and the weak link is the track: an offline tracker that fragments a long occlusion into two tracks starves the refinement stage of the context it exists to supply.
Reported comparisons to human labels range from parity to a measured advantage, but every comparison surveyed on this node is scored against a human-labeled reference, so a claim to outperform human labels is only as strong as how independently that reference was checked.
Every semi-supervised result surveyed here still starts from a human-labeled seed set; auto labels have been shown to reduce how much of a training set needs that seed, not to remove the need for one.
Only one of the four papers surveyed here describes having annotators relabel its hard cases and comparing the result against the official human labels, which is currently the exception rather than a standard step in publishing a “beats human” result.
FAQ
What does “offboard” actually mean here?
It means the labeling model runs after the drive is over, on a server, with access to the entire recorded sequence rather than only the frames a vehicle has already seen. An onboard detector has a fixed latency budget and can only use past frames; an offboard one has neither constraint, which is what lets it use later frames to correct a box drawn earlier in the sequence.
If the model is trained on human labels, how can it outperform them?
By comparing against an independently checked reference rather than the same labels used to train and tune it. One paper surveyed here reports outperforming average human labels on the standard benchmark test set, and a separate paper reports manually relabeling the specific cases where its predictions and the official human labels disagreed, finding some of its own predictions were the more accurate ones. Both are real results under their own evaluation conditions; neither means the model has stopped depending on human-labeled data somewhere in its pipeline.
Can an auto-labeler train a detector with no human labels at all?
Not in the results surveyed here. Every semi-supervised result cited on this node still starts from a human-labeled seed set, typically a fraction of the full fleet, before the auto-labeler’s output is added; the reported gains are against a baseline trained on that same seed set alone, not against zero human labels.
How is an auto-labeled corpus checked before anyone trusts it as ground truth?
By comparing a sample of its output against human labels and, in at least one published pipeline, manually re-examining the cases where the two disagree rather than assuming either side is automatically right. How large that sample is and how it is chosen is not standardized across the papers surveyed here, so two corpora both described as verified against human labels are not necessarily verified to the same standard.
