luuuulinnnn/egohoi-layer-ablation-hot3d-h2o

EgoHOI Layer-Injection Ablation — HOT3D + H2O (Wan 2.1 I2V-14B)

Six training runs that vary which Wan DiT blocks receive camera and object conditioning, on a frozen Wan 2.1 I2V-14B-720P base. Everything else — data, schedule, optimizer, seed — is held fixed, so the runs are directly comparable to each other.

Code: edisiondyli/egohoi_extension, branch obj_rope, commit 6ea62eb ("obj layer ablations", 2026-08-13). Every run directory carries a PROVENANCE.txt with the exact branch and commit it was launched from.

This is a separate suite from the earlier egohoi-control-hot3d-h2o repo, which trained at commit 563354c for 13000 steps to ask a different question (does conditioning help at all, vs. a frozen-Wan baseline). The two repos' numbers are not comparable: this suite runs 15000 steps, on refreshed captions, and evaluates on a held-out test split rather than the training-set validation grid.

The six runs

hand_encoder is disabled in all six. Wan has 40 self-attention blocks; "shallow" means blocks 0–19, "deep" means 20–39.

run	camera blocks	object blocks	val loss	train cum-avg
train_cam_all_obj_all_layers	0–39 (all)	0–39 (all)	0.04656	0.12229
train_cam_0_19_obj_0_19_layers	0–19 shallow	0–19 shallow	0.04774	0.12342
train_cam_0_19_obj_20_39_layers	0–19 shallow	20–39 deep	0.04782	0.12326
train_camera	0–19 shallow	— (disabled)	0.04890	0.12350
train_cam_20_39_obj_20_39_layers	20–39 deep	20–39 deep	0.05473	0.12616
train_obj	— (disabled)	20–39 deep	0.07448	0.13318

Validation uses fixed timesteps [357, 682, 833, 921, 978] with deterministic noise over 106 samples, so these are directly comparable.

What the losses say

• Camera conditioning carries the result. train_obj (object-only) is far worse than everything else; the gap to camera-only is much larger than any gap among the camera-enabled runs.
• Shallow beats deep for camera. Holding the object branch at 20–39, moving camera from 0–19 to 20–39 is the largest effect of any placement change.
• Where the object branch goes barely matters. Shallow vs deep differ by well under a thousandth — indistinguishable.
• All-layers wins, but it is not a fair comparison — see the caveat below.

Caveat on cam_all_obj_all. Injecting camera residuals into 40 blocks requires camera_encoder.num_layers=40, because the trainer refuses to inject into more Wan blocks than the ControlNet has residual blocks. That makes this run's camera branch twice the size of every other run's (25.1 GB vs 12.6 GB checkpoints). Its win therefore confounds placement with capacity, and cannot be read as "all-layer injection is better" on its own.

Results — PSNR / SSIM on the held-out test split

Generation is 81 frames at 40 sampling steps, unipc, guide_scale 5.0, seed 23, compared to ground truth over the same interval at 480×480. Each HOT3D clip is generated at two start frames (0-80 and 69-149), so a full run is ~192 HOT3D videos plus ~45 H2O videos.

All six runs are scored.

run	camera / object blocks	PSNR (dB)	SSIM	n
train_cam_all_obj_all_layers	0–39 / 0–39	21.124	0.6779	235
train_camera	0–19 / —	20.888	0.6654	237
train_cam_0_19_obj_20_39_layers	0–19 / 20–39	20.779	0.6651	235
train_cam_0_19_obj_0_19_layers	0–19 / 0–19	20.755	0.6637	235
train_cam_20_39_obj_20_39_layers	20–39 / 20–39	19.890	0.6434	235
train_obj	— / 20–39	13.796	0.4010	238

The object-only run collapses — roughly seven dB below anything with camera conditioning. Camera placement is worth about 1 dB, shallow over deep, the same direction the validation losses point. Note that adding the object branch does not improve pixel metrics: camera-only sits at the top of this table even though the object branch did lower validation loss. The two metrics disagree, and PSNR/SSIM is the one measured against held-out ground truth.

Per-video numbers are in metrics/<run>_per_video.csv; metrics/psnr_ssim_summary.json carries the overall and per-dataset aggregates.

Repository layout

<run>/                              # the six training runs, named as above
  checkpoints/final.pt              # step 15000; full training checkpoint
  train.log                         # per-step loss + every validation block
  resolved_config.yaml              # the fully resolved config actually used
  submit.sbatch                     # the exact job script
  PROVENANCE.txt                    # branch + commit + submission time
  slurm-*.out                       # scheduler logs, one per leg
  tensorboard/                      # training curves
  wandb/                            # offline W&B run

infer_<run>/
  hot3d/<clip>/<clip>_<span>_*.mp4  # generated videos
  h2o/<clip>/<clip>_<span>_*.mp4
  inference_summaries.jsonl         # GT ↔ generated pairing, caption, seed, checkpoint
  slurm-*.out, submit.sbatch

metrics/
  psnr_ssim_summary.json            # overall + per-dataset aggregates
  <run>_per_video.csv               # per-video PSNR/SSIM
  eval_psnr_ssim.py                 # the driver that produced them

Compare all six training curves at once:

tensorboard --logdir .

On the checkpoints

Each run's checkpoints/final.pt is the step-15000 full training checkpoint (model + optimizer state), not merged weights. The trainer also keeps the last three rotating step_*.pt snapshots on disk; those are resume points only — step_00015000.pt holds the same training state as final.pt — and are not mirrored here.

train_obj's checkpoint is much smaller (0.63 GB) than the rest because it carries only the Stand-In object LoRA, with no camera ControlNet branch. cam_all_obj_all's is twice the usual size for the reason given in the caveat above.

Training setup

Identical across all six runs except the injection layers and the enable flags.

Base model	Wan 2.1 I2V-14B-720P (frozen)
Steps	15000
Optimizer	AdamW, lr 1e-5 decayed to 0, weight_decay 0
Precision	bf16 + gradient checkpointing
Datasets	HOT3D + H2O interleaved — 19 HOT3D steps : 1 H2O step
Batch size	1 per GPU, 8 GPUs
Video	81 frames @ 480×480
View renders	hand_view_render for both datasets
Validation	every 500 steps, 106 samples, fixed timesteps
Checkpointing	every 1000 steps, last 3 kept
obj rope	tss-rope (temporal −1, shifted spatial grid)
Object stages	A 1500 / B 1500 / C remainder
Seed	23
Hardware	8× NVIDIA H200, 2 CPUs per GPU

Conditioning modules

module	configuration
camera_encoder	ControlNet-style residuals; Plücker rays (256 hidden, 6 ch) + masks; control_dim 1024, 20 residual blocks (40 for cam_all), 16 heads, ffn 4096, rms_norm
obj_encoder	Stand-In image-branch LoRA r128 / α128 on self_attn.{q,k,v}, zero init
hand_encoder	disabled in every run

Data

Preprocessed HOT3D + H2O from EndeavourDD/Jichen.

• Train: outputs/hot3d_preprocessed (HOT3D clips outside the test grid) + H2O subject1_ego, subject2_ego
• Test: outputs/hot3d_preprocessed_test — 96 clips, exactly the config's val_clips grid, packaged separately. H2O test is subject3_ego.
• Captions: refreshed hot3d_qwen_captions.json (4548 entries; 207 differ from the previous revision, same key set) + h2o_action_label_captions.json

Reproducing this

Two things in the upstream README do not work as written:

1. The "camera all, obj all" command crashes. It sets 40 camera injection layers but leaves camera_encoder.num_layers at its default of 20, and the trainer raises ValueError: Cannot inject more Wan layers than CameraEncoder residual blocks. Add --set camera_encoder.num_layers=40. This is what doubles that run's camera branch.
2. The inference commands silently use the training set. Section 4 is titled "use hot3d_preprocessed_test", but the commands omit --hot3d-root, which defaults to outputs/hot3d_preprocessed. Pass --hot3d-root outputs/hot3d_preprocessed_test to actually evaluate on the held-out split.

Further environment notes:

3. H2O object_render_metadata.json stores absolute /home/jc/... paths. Only frames[].image_path is read by the training and inference loaders, so that one field must be rewritten to outputs/h2o_preprocessed/...; mask_path, sequence_dir and object_root are read only by scripts/preprocess/objects/render_canonical_views.py and can be left alone.
4. CUDA_HOME must point at a real toolkit (module load CUDA/12.6.0), or transformers → deepspeed raises MissingCUDAException at startup.
5. scripts/eval.py is empty (0 bytes). The working metric tool is scripts/compute_video_psnr_ssim.py, which handles one pair at a time.

Run notes

• 15000 steps exceeds the 48 h scheduler limit, so each run was submitted as chained legs with auto-resume from the newest step checkpoint; the slurm-*.out files cover the separate legs.
• train_cam_0_19_obj_20_39_layers lost a leg to a node fault on dgx073 (CUDA reported num_gpus= empty, NVML unavailable) and was resubmitted from its step-13000 checkpoint. Its inference was then run as a six-way clip shard to catch up.