Ingest
The declared rig is checked against the actual session before a frame is decoded.
PRODUCT / ACTUATERaw capture in.
Raw capture in.
Certified episodes out.
Actuate turns egocentric manipulation capture into a provenance-carrying canonical episode, a quality certificate, and — when every gate passes — a training dataset. It is deliberately fail-closed: an absent measurement stays absent, an uncalibrated model score is never shown as a probability, and a failed gate blocks the export rather than being replaced with a plausible value.
EGOCENTRIC · REAL-DATA VALIDATED / LEROBOT V3 · RLDSTHE PIPELINE
Eight stages.Every skip on the record.
Each stage writes its own status and, when it does not run, the exact reason. A stage that skips says so in the output rather than leaving a gap that looks like a result.
Perceive
3D hand pose, metric depth, and object detection across the frames of the source.
Canonical
One frozen, versioned schema. Every populated field carries its provenance.
Label actions
Action intervals per hand, each carrying a confidence rather than a boolean.
Retarget
Wrist trajectory solved to robot joints, then replayed against a physics model.
Certify
Per-component quality certificate, including the components that were not measured.
Language
Task and subtask annotation grounded in frames sampled from the episode itself.
Package
LeRobot v3 or RLDS, written only once consent, PII, and task gates all pass.
REAL OUTPUT
An actual run,opened in the viewer.
The pipeline's final stage writes a Rerun recording containing every modality it produced. This is that file — from a real processed session, not a reconstruction and not a mock.
Scrub the timeline and the source video, the reconstructed 3D hand skeletons, the depth point cloud, the per-finger contact traces, and the episode's own quality certificate all move together — because they were all logged on one clock by the same run.
Episode 579231cb…_ep0045 framesQuality 3/5
Rendered by Rerun 0.26.2 · recording written by the same versionDownload the .rrd ↓
Honest about what changed to make it servable: the recording the pipeline wrote is 520 MB. For this page the video was downscaled to 960×540 and JPEG-encoded, the depth raster to 320×180, and the point cloud decimated by half. Every geometry value, timeline value, certificate number, and annotation is the run's own — nothing was added, and nothing was substituted. The session is an internal capture used to exercise the pipeline end-to-end, not customer data.
CAPTURE RIGS
Two rigs declared.One selectable today.
A rig entry declares which channels it hardware-measures and which it infers. That declaration is what the schema validates against, so a rig cannot emit a signal no sensor on it ever produced — and being registered is not the same as being offered.
Egocentric, monocular
Head-mounted RGB with an optional head IMU, bare hands. This rig measures nothing directly — hand pose, depth, contact, and grasp are all inferred from vision, and the canonical record says so on every field rather than presenting inference as observation. This is the input the pipeline is real-data validated against.
Depth — model-estimatedStereo pair
Two calibrated cameras, where depth would come from the disparity between them — a physical measurement rather than a model's estimate. The rig is declared, and stereo sessions are already verified at ingestion, but it is not selectable until calibrated stereo is validated end to end. The stereo depth model is deliberately left unimplemented rather than shipped unvalidated: an absent depth path is better than a wrong one.
Depth — measured, once validatedTHE DECLARATION CHECK
Most pipelines trust the declaration. This one verifies it against the data.
One eye of a stereo rig is pixel-identical to a monocular capture — both are 1920×1080, both look like ordinary first-person video — so no single-frame test can separate them. Drop one eye into a session, declare it monocular, and it ingests cleanly and produces depth that is plausible-looking and physically meaningless. Actuate refuses that session at ingestion, before the wrong contract can be applied to it, in three layers ordered by cost:
L1Provenance
A file named take0003_L.mp4 with no take0003_R.mp4 beside it is one eye of a stereo pair. Caught by name, before any pixel is read, and refused with the exact missing filename.
L2Layout
A side-by-side stereo composite declared as monocular is refused on geometry — and a stereo declaration with only one view is refused rather than having the second invented.
L3Correspondence
Two declared eyes must hold a constant, sign-stable horizontal baseline with near-zero vertical disparity. Two unrelated cameras do not pass as a pair, and footage too textureless to judge is recorded as unverifiable — never upgraded to verified.
The thresholds behind the correspondence check were measured on real stereo footage, not guessed, and sit several times clear of both classes. They should be re-derived against a second stereo rig before being treated as general.
PERCEPTION
What the pipelineactually recovers.
Live models, running on every frame of the source by default. Sampling is an explicit mode, and the resulting coverage is recorded either way.
Hands
Real 3D hand pose
WiLoR produces per-frame hand keypoints. Because its output is root-relative, the root depth is solved against the depth map at the hand keypoints and each keypoint is back-projected, so the hand sits inside the scene at metric distance rather than at an arbitrary origin. Where the detector reports no confidence, confidence stays unknown instead of defaulting to 1.0.
Depth
Metric depth, with its source named
Depth on the egocentric path comes from UniDepth V2 — a model estimate, recorded as one. The schema refuses to mark depth hardware-measured on a monocular session, which is why the stereo rig's measured-depth claim cannot simply be borrowed: that distinction has to survive into the delivered dataset rather than being flattened into a single number. It is also why the stereo depth path stays unimplemented until it can be validated.
Objects
Detection and tracking
Grounding DINO proposes objects from text prompts and SAM2 segments and tracks them through the episode. Six-degree-of-freedom object pose is not claimed: a bare-hand egocentric capture does not support it, so it is absent rather than estimated into the output.
CERTIFICATION
Every episode carriesits own evidence.
A composite nobody can decompose is a score nobody can dispute. Each component is published alongside the total — including the ones that were never measured.
sync_integrity0.88Worst cross-stream timestamp drift, as a fraction of one frame period. Streams further apart than a full frame cannot be paired frame-to-frame at all, and score zero.
calibration_completeness0.30Capped at 0.30 because this rig's intrinsics are approximated rather than measured — a guess that was once a real bug, scaling every back-projection by 1.7×. A calibrated rig lifts the cap to 1.00.
perception_confidencenot measuredOnly explicitly calibrated probabilities may contribute. WiLoR's detector score and UniDepth's confidence raster are genuine signals, but neither has been calibrated as a probability on this capture domain — so publishing a number here would be inventing a measurement.
contact_consistencynot measuredThis is a bare-hand rig. There is no contact sensor to agree or disagree with vision, and a vision-versus-vision comparison would be a tautology dressed up as a check.
ik_convergence_ratenot measuredRetargeting skipped on this episode — no trained arm estimator was present — so there is no convergence figure to report.
quality3 / 5A weighted mean over the components that were measured, renormalised so an unmeasured channel neither helps nor hurts. Writing zero instead of “not measured” would claim a catastrophic reading on a channel nothing ever observed.
Thresholds
The component weights and bands are engineering priors, not results calibrated against downstream policy success. Every certificate carries that state explicitly, as eis-v1-uncalibrated, and it stays there until a real calibration exists. We are tuning these against real data over time; we are not presenting them as finished science.
GATES
Fail closed,every time.
Consent, privacy, task presence, and physics eligibility are export-time checks enforced in code — not metadata attached for decoration.
Nothing ships unconfirmed
Delivery writes pass through a fail-closed guard: an episode is deliverable only when consent is explicitly granted and the PII pass has passed. The test is an explicit allow, never “not denied”, so any status added later defaults to blocked. There is no override flag — the setting that would enable one exists purely so configuration validation can reject it.
Visible, not implied
Detected face regions are blurred before anything leaves the pipeline, and the privacy report is recall-bounded rather than presented as complete. Redaction is never implied merely because processing finished; if the classifier asset is unavailable, the stage fails with a preflight error instead of passing silently.
A trajectory that fails is withheld
Retargeted joint trajectories are replayed through the embodiment's MuJoCo model and checked for joint-limit violations, self-collision, and solver teleports. The verdict is kinematic — it means executable, not that a grasp will hold. On our A100 acceptance run the Franka candidate failed, the robot trajectory was withheld, and human-space data shipped instead. No trained root-frame model ships yet, so robot trajectories skip by default rather than being fabricated.
EXPORT
Loaded back bythe loader that matters.
An export is not finished because it was written. It is finished when the target framework reads it.
LeRobot v3
Verified end-to-end against LeRobot's own dataset loader: on the acceptance run all 619 exported frames were read back by LeRobotDataset, and one real CUDA ACT optimizer step ran on them.
RLDS
Written from a separate environment so TensorFlow never shares the GPU worker process with perception. The same run's 619 steps were loaded back by TFDS.
Canonical
A frozen, versioned JSON schema with per-field provenance, plus a run manifest carrying code and dependency lineage, frame coverage, and every stage's skip reason.
Export re-checks consent, PII status, task presence, successor integrity, grasp availability, and any requested embodiment's physics verdict. When the acceptance run produced 619 training records from 1,770 canonical frames, that gap was explicit filtering — the exporter requires usable measured successors — recorded in the manifest rather than hidden as a quiet processing cap.