northstar-robotics-ai/failure-recovery

Failure Recovery is a set of pi0.5 vision-language-action policies for a bimanual YAM robot, released with our post on failure recovery. For each task, a base policy is trained on 100 teleoperated demonstrations. It is then trained for another 6000 steps in one of two ways, each with 50 new episodes:

• 50-clean: 50 more clean teleoperated demonstrations.
• dagger: 50 runs of the base policy where it failed, an operator took over in VR, and finished the task (HG-DAgger). Only the human-controlled part, from takeover to completion, is trained on.

Policies

Folder	Task	Starts from	Training data	New frames	GPU
basket/base/	basket	lerobot/pi05_base	100 demos (94 train, 6 val)	86,825	A100
basket/50-clean/	basket	basket/base	50 new demos only (47 train, 3 val)	23,643	H100
basket/dagger/	basket	basket/base	50 interventions (47 train x3, 3 val) + the 93 base demos	25,953	H100
cubes/base/	cubes	lerobot/pi05_base	100 demos (94 train, 6 val)	45,967	A100
cubes/50-clean/	cubes	cubes/base	50 new demos only (47 train, 3 val)	16,796	H100
cubes/dagger/	cubes	cubes/base	49 interventions (46 train x3, 3 val) + the 94 base demos	22,362	B200

All data is collected by Northstar Robotics with VR teleoperation at 15 fps. The dagger policy also trains on the base demonstrations again (HG-DAgger keeps the aggregate dataset), and each intervention is repeated 3 times; the 50-clean policy trains on the new demonstrations only.

Task	Prompt
basket	Place the mouthwash, dishwashing liquid, and wipes into the white basket
cubes	Stack the cubes into a tower with the blue cube in the middle

Training

Attribute	Value
Framework	LeRobot 0.6.1, pi05
Steps	6,000 per stage
Batch size	32
Optimizer	AdamW, peak lr 2.5e-5, 200 warmup steps, cosine decay to 2.5e-6 at step 6000
Precision	bfloat16, full fine-tune
Robot	I2RT YAM, two 6-DoF arms with parallel grippers

Each folder's train_config.json is the exact config used; its local paths refer to our training volume.

Inputs and Outputs

Key	Shape	Description
observation.images.top	3x360x640	external scene camera
observation.images.left	3x360x640	left wrist camera
observation.images.right	3x360x640	right wrist camera
observation.state	14	joint positions: left joints 0-5, left gripper, right joints 0-5, right gripper
action	14	absolute joint position targets, same order as the state

Images are resized to 224x224 inside the policy. Each call returns a 30-step action chunk, 2 s at 15 Hz. State and action use quantile normalization; the statistics are in each folder's processor files.

Repository Structure

basket/base/
basket/50-clean/
basket/dagger/
cubes/base/
cubes/50-clean/
cubes/dagger/
data/<task folder>/<folder>/<session>/

Each policy folder holds:

model.safetensors
config.json
train_config.json
policy_preprocessor.json
policy_preprocessor_step_3_normalizer_processor.safetensors
policy_postprocessor.json
policy_postprocessor_step_0_unnormalizer_processor.safetensors

Usage

Loading a Policy

from huggingface_hub import snapshot_download
from lerobot.policies.pi05.modeling_pi05 import PI05Policy
from lerobot.policies.factory import make_pre_post_processors

root = snapshot_download("northstar-robotics-ai/failure-recovery", allow_patterns=["basket/dagger/*"])
path = f"{root}/basket/dagger"
policy = PI05Policy.from_pretrained(path)
pre, post = make_pre_post_processors(policy.config, pretrained_path=path)

Fine-tuning

lerobot-train --policy.path=<downloaded folder> --dataset.repo_id=<your dataset>

Data

data/ holds the recordings behind these policies: the demonstrations and interventions they were trained on, and 100 evaluation runs of each policy. 1,000 sessions, 35 GB, all from one YAM robot. The task folders are named after the prompt:

• basket: Place_the_mouthwash__dishwashing_liquid__and_wipes_into_the_white_basket
• cubes: Stack_the_cubes_into_a_tower_with_the_blue_cube_in_the_middle

Task	Folder	Sessions	Hours	Contents
basket	100-base	100	1.73	teleoperated demonstrations, trained basket/base
basket	50-clean	50	0.47	teleoperated demonstrations, trained basket/50-clean
basket	50-intervention	50	1.35	policy runs with an operator takeover, trained basket/dagger
basket	100-policy-run	100	3.68	evaluation runs of basket/base
basket	100-eval(50clean)	100	2.91	evaluation runs of basket/50-clean
basket	100-eval(50-intervention)	100	3.33	evaluation runs of basket/dagger
cubes	100-base	100	0.92	teleoperated demonstrations, trained cubes/base
cubes	50-clean	50	0.34	teleoperated demonstrations, trained cubes/50-clean
cubes	50-interventions	50	1.21	policy runs with an operator takeover; 47 of them trained cubes/dagger
cubes	100-base-eval	100	2.85	evaluation runs of cubes/base
cubes	100-clean-eval	100	2.00	evaluation runs of cubes/50-clean
cubes	100-recovery-eval	100	3.73	evaluation runs of cubes/dagger

data/sessions.csv has one row per session: task, folder, session, path, episode id, trial number (R01 to R100 for the cubes evaluations), whether it was used in training, who was driving when the recording started, number of operator takeovers, and duration.

Each session holds:

metadata.json            recording info and control timeline; interventions[] has each takeover's start and end
control_events.jsonl     control state changes: POLICY, TELEOP, HOLD (missing in older basket/100-base sessions)
recording/
  exo.mp4                external scene camera, 640x360, about 30 fps
  wristL.mp4             left wrist camera
  wristR.mp4             right wrist camera
  frames_<camera>.json   per-frame timestamps; idx is the mp4 frame index
  joints_left.json       measured joint positions (6, radians), velocities and gripper
  joints_right.json
  actions_left.json      commanded joint positions and gripper, from the operator or the policy
  actions_right.json
  tag.json               operator tag set at recording time

All timestamps (ts_us) are microseconds on one shared clock, so cameras, joints and actions line up by timestamp. The policies were trained on these recordings resampled to 15 fps. Exo depth (RealSense D435) was recorded for most sessions but is not included.

License

Derived from pi0.5 by Physical Intelligence (openpi) through lerobot/pi05_base. The PaliGemma backbone is subject to the Gemma Terms of Use.

Citation

@misc{northstar2026failurerecovery,
  author = {Northstar Robotics},
  title = {Failure Recovery},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/northstar-robotics-ai/failure-recovery}
}