docs/LIVE_VIEWER.md

# Interactive live viewer

![TRAIN LIVE domain-randomization mode](../media/dropbear_train_live_domain_randomization.png)

`scripts/play_dropbear_live.py` is a rendering/playback adapter around Isaac
Lab's RSL-RL player. Isaac Sim renders RGB on the selected simulation GPU. A
small Tk/Pillow helper in `scripts/dropbear_live_window.py` presents frames on
the desktop and sends user input back to the simulator over line-oriented
standard I/O.

This separation permits an A100 to run simulation and rendering while a small
display-attached GPU presents the window. The title and upper-right HUD query
the actual devices; hardware names are not hard-coded.

## Controls

- Drag the circular lower-left pad for body-frame planar velocity, up to
  1.0 m/s by default.
- Up is robot-forward (`+vx`).
- Left is robot-left (`+vy`).
- Right is robot-right (`-vy`).
- Down is robot-backward (`-vx`).
- Diagonals combine both axes and are radially clamped.
- Near the rim, the left/right 90-degree sectors progressively add yaw toward
  that side while preserving lateral translation. The forward/back
  90-degree sectors remain straight, forming a cross-shaped drive/turn map.
- Release springs the command to zero.
- Use the rectangular button to the right of the pad to switch between
  `MANUAL CONTROL` and `TRAIN LIVE`.
- Click-drag elsewhere to orbit azimuth/elevation around the robot.
- Resize the desktop window freely; the frame remains aspect-correct and all
  UI anchors follow the new size.

The first pad interaction switches from the command passed on the CLI to
manual joystick mode. Release intentionally keeps manual ownership at zero,
rather than silently restoring the scripted command.

## Live training passthrough

The `TRAIN LIVE` button displays a real environment sampled from the active
vectorized trainer. It is not a rollout made independently from the newest
checkpoint. The trainer publishes the sampled environment's root pose, all
joint positions and velocities, command, terrain type/level, episode clock,
randomized mass, and randomized contact material:

```bash
python -u scripts/train_dropbear.py \
  ... \
  --live-preview-state /tmp/dropbear_training_live.json
```

Connect the rendering process to the same atomic state file:

```bash
python -u scripts/play_dropbear_live.py \
  ... \
  --training-live-state /tmp/dropbear_training_live.json
```

The trainer remains headless. State publication copies only one environment
and adds no camera or rendering work to the training GPU. The separate viewer
mirrors the selected terrain tile and articulation on its render GPU.
Successive trainer samples are interpolated for a smooth display because one
environment in a large vectorized training batch advances more slowly than
wall-clock real time.

Manual state is saved when entering `TRAIN LIVE` and restored when returning
to `MANUAL CONTROL`. Camera orbit remains available in both modes; the
joystick is dimmed and ignored during passthrough. The left HUD identifies the
source environment, sample age, terrain and difficulty level, randomized
total mass, and static/dynamic friction.

The optional `--live-preview-env <index>` trainer flag selects a particular
source environment. Without it, the bridge selects the midpoint environment,
which lies in the obstacle portion of the current 35/65 flat/box terrain mix.

## Simulator geometry

The footprint ghosts and COM trail are USD scene geometry created at playback
time, not pixels painted over the captured frame.

- Blue cuboids mark historical left-foot placements.
- Orange cuboids mark historical right-foot placements.
- Opacity/brightness bins fade old contacts.
- Thin cyan cylinder segments trace the world-space COM path.

Because these are scene objects, they respond correctly to camera orbit and
perspective.

## HUD

The left stack reports command, COM target/actual velocity and height, body
velocity, foot clearance/order, reward, episode horizon, tracking score, and
fall rate. The upper-right stack identifies checkpoint, iteration, simulation
GPU, display GPU, measured simulation/wall-time ratio, presentation rate, and
value loss. The lower-right plot shows raw and exponentially averaged
value-loss history.

## Rendering path

The viewer uses native 960×540 renders at the 50 Hz policy cadence. Physics
remains at 200 Hz and policy/input remain at 50 Hz. The window estimates
bidirectional optical flow only between two consecutive genuine RTX frames;
it never feeds an interpolated result back into the next estimate. Large scene
changes are presented as hard cuts, and HUD/plot regions remain unwarped.

For the smooth, physically valid manual path, keep both the one-environment
simulation and RTX render on the selected CUDA device:

```bash
CUDA_VISIBLE_DEVICES=1 env_isaaclab/bin/python -u \
  scripts/play_dropbear_live.py \
  --actor-base-lin-vel \
  --viewer-com-control \
  --viewer-gait-period 0.55 \
  --viewer-com-stand-height 1.052 \
  --viewer-com-height-delta 0 \
  --viewer-directional-obstacle-terrain \
  --viewer-disable-pushes \
  --viewer-reset-policy-joints-only \
  --viewer-reset-joint-position-range 1 1 \
  --viewer-reset-joint-velocity-range 0 0 \
  --task Isaac-Dropbear-Velocity-Play-v0 \
  --checkpoint checkpoints/dropbear_locomotion_v0.2.0/model_10479.pt \
  --follow-checkpoints logs/rsl_rl/dropbear_velocity/STAGE66_RUN \
  --pin-manual-policy \
  --training-live-state /tmp/dropbear_training_live.json \
  --viewer-command 0.20 \
  --viewer-joystick-max-speed 1 \
  --viewer-joystick-max-yaw-rate 1 \
  --viewer-fps 50 \
  --viewer-display-fps 50 \
  --viewer-render-width 960 \
  --viewer-render-height 540 \
  --device cuda:0 \
  --num_envs 1 \
  --viz none
```

`--pin-manual-policy` keeps manual control on the explicitly selected,
validated checkpoint. `--follow-checkpoints` still updates the epoch, loss,
and current-training-checkpoint HUD, and `TRAIN LIVE` still mirrors the real
trainer. This separation avoids hot-swapping an incompletely evaluated actor
into the manually controlled episode.

The compact viewer atlas contains one column per semantic terrain label.
Incoming trainer snapshots are mapped by labels such as `forward_box` and
`left_box`, not by the trainer's numeric column index. This keeps obstacle
direction correct even though the full trainer and compact viewer have
different atlas dimensions.

The title and HUD report the runtime-derived simulation, render, and display
hardware. `sim realtime` and `rendered` are measured achieved rates; the
display number is the requested desktop presentation rate.

## Checkpoint following

Pass `--follow-checkpoints <run-directory>` to adopt new `model_*.pt` files as
training saves them. Adoption occurs only after a file is complete and the
actor state loads successfully.

Add `--pin-manual-policy` when new checkpoints should update telemetry without
replacing the manual-control actor. Omit it only when deliberate checkpoint
hot-reload testing is desired.