songyiren/ltx-v2v-reference-conditioning

LTX-V2V Reference-Conditioned Video-to-Video

Video-to-video generation on top of LTX-2 (Lightricks' DiT-based video+audio diffusion model): given a source video (structure/motion signal) and a first-frame reference image (target identity/appearance), generate an output video that follows the source video's motion while adopting the reference image's look.

This repo is a trimmed-down, sharable fork containing:

• packages/ltx-core, packages/ltx-pipelines — the LTX-2 inference stack, with a small local patch to the reference-conditioning position logic (see below) that is not in upstream Lightricks/LTX-2.
• packages/ltx-trainer — LTX-2's LoRA/IC-LoRA training stack, with a matching patch on the training side, plus a reference_image dataset column so you can train on (video, reference_image, caption) triples.
• scripts/infer_v2v.py — new inference script for the video+ref-image → video workflow described above.
• configs/*.yaml, scripts/train_ic_lora.sh — training configs and launcher for the two custom IC-LoRA variants this workflow needs.
• data/ — dataset format documentation and example JSONL rows.

Not included: pretrained IC-LoRA checkpoints. The ones we trained internally were trained on licensed movie footage, so we're not redistributing the weights — train your own with scripts/train_ic_lora.sh on your own data (see data/README.md).

Quick start

./setup.sh                     # uv sync + download ~90GB of base-model weights
source .venv/bin/activate

# 1. Prepare your data (see data/README.md for the JSONL schema)
scripts/preprocess_dataset.sh data/dataset.jsonl "1920x1024x233"

# 2. Train an IC-LoRA (defaults to configs/v2v_reference_ic_lora.yaml)
scripts/train_ic_lora.sh --config configs/v2v_reference_ic_lora.yaml

# 3. Run inference with your trained checkpoint
python scripts/infer_v2v.py \
  --input-video my_source_video.mp4 \
  --ref-image my_reference.png \
  --prompt "A woman in a red dress walks through a garden, camera tracking steadily." \
  --structure-lora packages/ltx-trainer/outputs/v2v_reference_ic_lora/checkpoints/lora_weights_step_08000.safetensors \
  --output out.mp4

setup.sh downloads gated/large weights from HuggingFace — if a download 401s, run huggingface-cli login (or set HF_TOKEN) first.

How it works

Input video → output video

The source video is never fed to the transformer as raw pixels through a special "V2V" code path — there isn't one. It goes through the exact same video_conditioning mechanism IC-LoRA uses for any reference signal (ICLoraPipeline.__call__'s video_conditioning: list[tuple[path, strength]] parameter, in packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py): it's VAE-encoded, patchified into tokens exactly like the noisy video latent, and appended to the same self-attention token sequence the denoiser attends over (VideoConditionByReferenceLatent, packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py). A LoRA (--structure-lora) is what actually teaches the model to use those extra tokens meaningfully — the base model has no opinion on what an arbitrary appended video means.

Important: reference_video/--input-video is not necessarily raw source footage — it's whatever structure/motion signal your LoRA was trained to condition on (pose render, edge map, or the raw video itself, if that's what you trained with). See data/README.md → "About reference_video".

Two ways to position-encode the reference image

This is the part that's easy to get subtly wrong, so it's exposed as an explicit --pos-mode flag in scripts/infer_v2v.py rather than being baked in silently:

--pos-mode first-frame (default) — the reference image reuses the denoised video's own frame-0 RoPE position. Implemented via ICLoraPipeline's native images: list[(path, frame_idx, strength)] parameter → this is a base-model capability, no LoRA required. Code path: combined_image_conditionings() in packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py → VideoConditionByLatentIndex / VideoConditionByKeyframeIndex in packages/ltx-core/src/ltx_core/conditioning/types/. Because the reference image is placed at the exact same absolute time position as frame 0, RoPE's relative-rotation attention naturally treats it as "this is what frame 0 looks like" rather than as a separate signal.

--pos-mode reference — the reference image gets its own, disjoint RoPE position range instead of sharing frame 0's. Implemented via video_conditioning → VideoConditionByReferenceLatent (packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py). This is the path our custom IC-LoRAs (configs/ref_image_ic_lora.yaml, configs/v2v_reference_ic_lora.yaml) are actually trained against, and is generally the better-performing option once you have a matching LoRA — the base model alone doesn't know what to do with a disjoint-position reference block; it has to be taught.

Why "disjoint" instead of just also using t=0: giving the reference image the same absolute position as the target's frame 0 means RoPE's relative rotation between them is zero — the attention mechanism can't tell the reference token and the target's own frame-0 token apart by position alone. The fix (originally a local patch on top of upstream LTX-2, in reference_video_cond.py):

# packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py
REF_POSITION_TICK_SECONDS = 1e-3
current_min_time = latent_state.positions[:, 0, ...].min()
ref_max_time = positions[:, 0, ...].max()
positions[:, 0, ...] = positions[:, 0, ...] - ref_max_time + current_min_time - REF_POSITION_TICK_SECONDS

This shifts the whole reference block to end 1ms before whatever's already the earliest time position in the sequence — a strictly ordered, non-overlapping range. The training side (packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py, _apply_reference_condition) applies the identical tick-offset logic, so a LoRA trained with this code sees the same position layout at train and inference time. If you ever modify one side, modify the other — they must stay in parity.

When two reference-type conditions are stacked (as in configs/v2v_reference_ic_lora.yaml, which uses both reference_video and reference_image), each additional block is slotted immediately before whatever range is already occupied, so every block — target frames, structure reference, appearance reference — ends up in its own strictly-ordered slice of time, none of them aliasing.

Conditioning strength (separate from position mode)

Two independent knobs control how strongly a reference item affects the output, orthogonal to which position mode you use:

• --ref-strength / --structure-strength (denoise strength, strength in ConditioningItem): controls how much noise the conditioning latent itself gets mixed with during denoising (GaussianNoiser.torch.lerp, in packages/ltx-core/src/ltx_core/components/noisers.py). 1.0 = the reference stays exactly at its clean VAE-encoded value; lower values let the model deviate from it.
• --conditioning-attention-strength: scales how strongly conditioning tokens attend to/from the noisy tokens in self-attention, independent of the noise blending above.

Repository layout

setup.sh                        one-click env + weights setup
pyproject.toml, uv.lock         workspace definition (uv)
packages/
  ltx-core/                     LTX-2 model + conditioning + RoPE (patched)
  ltx-pipelines/                ICLoraPipeline and friends
  ltx-trainer/                  LoRA / IC-LoRA training stack (patched)
configs/
  v2v_reference_ic_lora.yaml    train on (video, reference_video, reference_image, caption)
  ref_image_ic_lora.yaml        simpler: train on (video, reference_image, caption) only
  accelerate/                   accelerate configs referenced by the trainer docs
scripts/
  train_ic_lora.sh              resolves __REPO_ROOT__ in a config, launches training
  preprocess_dataset.sh         wraps packages/ltx-trainer/scripts/process_dataset.py
  infer_v2v.py                  video + reference image -> video, --pos-mode switch
data/
  README.md                     dataset JSONL schema, resolution buckets, etc.
  dataset.jsonl                 example rows: video + reference_video + reference_image + caption
  dataset_image_only.jsonl      example rows: video + reference_image + caption
weights/                        populated by setup.sh (gitignored — large binaries)

Which config to train

• Have paired (source video, structure signal, reference image) data? Use configs/v2v_reference_ic_lora.yaml — trains one LoRA that uses both --input-video and --ref-image at inference.
• Only have (video, reference image) pairs, no structure video? Use configs/ref_image_ic_lora.yaml — simpler, doesn't need --input-video at all (pass an empty/no-op structure conditioning, or adapt scripts/infer_v2v.py to drop --structure-lora/--input-video for this case).

See data/README.md for the exact JSONL schema, column-to-encoder mapping, and resolution-bucket rules.

Known limitations / things we couldn't fully verify

• We could not confirm from the LTX-2 repo alone whether the public LTX-2.3-22b-IC-LoRA-Union-Control LoRA expects raw source video or a preprocessed control map (depth/pose/edge) for its video_conditioning input — its model card (not vendored here) would be the source of truth if you want to use it instead of training your own structure-control LoRA.
• reference_video/--input-video preprocessing (pose/edge/depth extraction) is out of scope for this repo — bring your own, or use the basic Canny-edge compute_reference() in packages/ltx-trainer/scripts/process_videos.py as a starting point.