IDEAS-Lab-Northwestern/pi05-base-dusty-transfer-joint-2cam-lora

pi0.5 LoRA — Dusty-Transfer (JointController, 2-cam)

LoRA fine-tune of pi0.5 (pi05_base) for a simulated household manipulation task: wipe a dusty container clean with a sponge, then transfer a potato into it. Trained with the ManiGuard pipeline (NU IDEAS Lab) on top of a pristine openpi trainer.

This is the JointController track: collection → render → SFT → eval are all joint-space (no end-effector / IK anywhere).

Tasks & prompts

The instruction is pulled per-episode from the dataset's task field (prompt_from_task=True). Three variants:

• "Wipe the dusty stockpot clean with the sponge, then transfer the potato from the chopping board into the stockpot."
• "Wipe the dusty mixing bowl clean with the sponge, then transfer the potato from the platter into the mixing bowl."
• "Wipe the dusty saucepan clean with the sponge, then transfer the potato from the platter into the saucepan."

Model

• Base: pi0.5 — gs://openpi-assets/checkpoints/pi05_base/params (warm-start).
• LoRA: PaliGemma gemma_2b_lora + action expert gemma_300m_lora; base frozen, only the LoRA adapters trained (openpi freeze_filter).
• action_dim=32 (padded), action_horizon=16, pi05=True, discrete_state_input default (True).
• Controller — JointController: 8-D joint state + 8-D absolute-joint action. The 7 arm joints are consumed as per-step deltas (gripper kept absolute) and reconstructed to absolute joint targets at inference, fed straight to a JointController — no eef→joint IK.
• Cameras (LIBERO 2-cam): base_0_rgb ← image_left (third-person overview), left_wrist_0_rgb ← wrist_image; the third slot (right_wrist_0_rgb) is zero-filled and masked off.

Training data

IDEAS-Lab-Northwestern/sim-dusty-transfer-30-joint-3cam (LeRobot v2.1, private): 30 episodes / 20,265 frames, sim teleop rendered joint + 3-cam (image_left / image_right / wrist_image); consumed 2-cam (image_right dropped). Episode split across the 3 task variants: 12 / 10 / 8.

Training setup

Hardware	4× RTX A5000 (24 GB), FSDP (fsdp_devices=4)
Precision	float32 (see note)
Global batch	12 (3 / GPU)
Steps	3,380 (~2 epochs)
LR schedule	cosine, warmup 340, peak 3e-5, decay → 3e-6
Optimizer	AdamW (b1 0.9, b2 0.95), global-norm grad clip 1.0, EMA off
Wall-clock	6.75 h; final train loss **0.014** (from ~0.069), no divergence

Note: why float32 (and when bf16 is fine)

On 24 GB cards the full model does not fit replicated, so training uses FSDP parameter sharding. In bf16, FSDP's gather/scatter was numerically unstable on this hardware and diverged to NaN within ~25–100 steps — the same config trains fine in bf16 on 80 GB A100/H100 in pure data-parallel (no sharding). Setting the model dtype="float32" removed the instability. If you fine-tune on larger-memory GPUs (data-parallel, no FSDP), bf16 is fine and faster.

Intended use & limitations

• Research artifact for sim joint-control evaluation in the ManiGuard pipeline. Success-rate eval is a separate effort and is not reported here.
• Sim-only; not validated on physical hardware.
• Small dataset (30 episodes, single object/scene family) — expect limited generalization beyond the trained variants.

License & provenance

Base weights are Gemma-derived (pi0.5); use is governed by the Gemma terms. LoRA deltas released for research. Trained via openpi's JAX trainer, configured by ManiGuard's maniguard.openpi_sft module (openpi itself unmodified).