Edge Refiner v15: 32 channels, new default

README.md

@@ -19,55 +19,56 @@ datasets:
 
 # CorridorKey Real-Time Edge Refiner
 
-**A 23k-parameter CNN that brings back native-resolution hair and edge detail when
+A small CNN (40k parameters in v15) that restores native-resolution hair and edge detail when
 [CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) runs at reduced resolution for live
-green-screen keying, e.g. a 4K or 1080p facecam in OBS at 30–60 fps.**
+green-screen keying, for example a 4K or 1080p facecam in OBS at 30 to 60 fps.
 
 ![Input and real-time result on real 4K green-screen footage](images/hero_druid_4k.png)
 
 <sub>Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time
-pipeline: CorridorKey 2.0 Stage 1 on a 512×512 person crop + this refiner.</sub>
+pipeline: CorridorKey 2.0 Stage 1 on a 512x512 person crop, then Edge Refiner v15.</sub>
 
-## What problem it solves
+## What it is for
 
-CorridorKey 2.0 produces excellent mattes, but at full 4K it is an offline model. To key live video, the
-network has to run on a small grid: for example a 512×512 crop around the person, or 512×256 for the whole
-frame. Its matte then has to be scaled back up to the camera resolution, and that step destroys the edge:
+At full 4K, CorridorKey 2.0 is an offline model. For live video the network has to run on a small grid, such as a
+512x512 crop around the person or 512x256 for the whole frame, and its matte then has to be scaled back up to the
+camera resolution. Bilinear upsampling makes the edge blurry and blocky, with a green or dark halo. A colour guided
+filter recovers some structure, but it smears hair, leaves fringes and fails where foreground and screen have
+similar brightness.
 
-- **Bilinear upsampling** gives a blurry, blocky edge with a green or dark halo.
-- **A colour guided filter** recovers some structure, but it smears hair, leaves fringes and fails where foreground
-  and screen have similar brightness.
-- **This refiner** looks at the native-resolution pixels and learns how CorridorKey 2.0 would have keyed them at full
-  resolution. It predicts corrections for alpha *and* foreground colour (despill / unmixing) at every native pixel.
+The refiner looks at the native-resolution pixels and has learned how CorridorKey 2.0 keys them at full
+resolution. It predicts corrections for alpha and for the foreground colour (despill and unmixing) at every native
+pixel near the edge.
 
 ![1:1 native-pixel crops on real 4K footage](images/compare_druid_4k.png)
 
 Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest
-neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, the target the refiner was trained on.
+neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, which is what the refiner was trained
+to match.
 
 ## How it works
 
-The refiner is a fully convolutional network: 11 input channels, a 3×3 input convolution, four residual dilated
-3×3 blocks (dilations 1, 2, 4, 1), GELU, and a 1×1 output convolution. It has 24 channels in v12 and v10w and 16
-in v9b.
-Its receptive field is ±9 pixels. Per native pixel it receives:
+The refiner is fully convolutional: 11 input channels, a 3x3 input convolution, four residual dilated 3x3 blocks
+(dilations 1, 2, 4, 1) with GELU, and a 1x1 output convolution. v15 has 32 channels, v12 and v10w have 24 and v9b
+has 16. The receptive field is 9 pixels in each direction. Per native pixel it receives:
 
 | Channels | Input |
 |---|---|
-| 0–2 | `I`: native RGB frame (sRGB) |
-| 3–5 | `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background |
-| 6–8 | `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low − I_low)`) |
+| 0 to 2 | `I`: native RGB frame (sRGB) |
+| 3 to 5 | `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background |
+| 6 to 8 | `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low - I_low)`) |
 | 9 | `a_up`: CorridorKey alpha, bilinear upsampled |
 | 10 | `a_gf`: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha |
 
-It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is used only inside an
-uncertainty band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of
-CorridorKey's matte, which in solid foreground and clean background is CorridorKey's own 0 or 1. The band is weighted
-by nearby foreground, which keeps the refiner out of dark or noisy screen areas.
+It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is only used inside an uncertainty
+band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of CorridorKey's matte,
+which is CorridorKey's own 0 or 1 in solid foreground and clean background. The band is weighted by nearby
+foreground, so the refiner stays out of dark or noisy screen areas.
+
+In the real-time engine the refiner only runs on 32x32 tiles that touch the band, up to a fixed tile budget, most
+uncertain tiles first. The tile count decides the cost, not the frame resolution. In the facecam profile, raising
+the budget from 256 to 512 tiles adds about 1.8 ms per frame on an RTX 5070 Ti.
 
-In the real-time engine the refiner only runs on 32×32 tiles that intersect the band, with a fixed tile budget
-(most uncertain tiles first). In fp16 that costs about 5–6 ms per frame with 256 tiles and 10–11 ms with 512 tiles
-on an RTX 5070 Ti.
 `corridorkey_refiner.py` in this repository is a standalone full-frame reference implementation of the same maths.
 On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha.
 
@@ -75,12 +76,15 @@ On a test frame it matches the tiled engine to a mean absolute difference of 0.0
 
 | File | Description |
 |---|---|
-| `corridorkey_edge_refiner_v12.safetensors` | **recommended** (October 2026), 24 channels, same architecture and cost as v10w, lower edge error and flicker |
-| `corridorkey_edge_refiner_v10w.safetensors` | previous version, 24 channels |
-| `corridorkey_edge_refiner_v9b.safetensors` | 16 channels, 10,948 parameters, slightly cheaper, slightly worse |
+| `corridorkey_edge_refiner_v15.safetensors` | recommended (October 2026), 32 channels, 40,324 parameters |
+| `corridorkey_edge_refiner_v12.safetensors` | previous version, 24 channels, 23,332 parameters, about 0.1 to 0.4 ms cheaper per frame |
+| `corridorkey_edge_refiner_v10w.safetensors` | older version, 24 channels |
+| `corridorkey_edge_refiner_v9b.safetensors` | 16 channels, 10,948 parameters |
 | `corridorkey_refiner.py` | model definition, loader, input construction and full-frame `refine()` |
 | `LICENSE` | Corridor Key Licence, Version 2.0 |
 
+The channel count is stored in the safetensors metadata, so `load_refiner` reads every version.
+
 ## Usage
 
 You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included
@@ -90,7 +94,7 @@ here) to produce the low-resolution matte. Then:
 import torch
 from corridorkey_refiner import load_refiner, refine
 
-net = load_refiner("corridorkey_edge_refiner_v12.safetensors")    # fp16, CUDA
+net = load_refiner("corridorkey_edge_refiner_v15.safetensors")    # fp16, CUDA
 
 # rgb:       (1, 3, H, W)   native frame, sRGB in [0, 1]
 # rgb_low:   (1, 3, h, w)   the (crop of the) frame as CorridorKey saw it
@@ -102,114 +106,111 @@ alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low)           # native reso
 If you key a crop, pass the native-resolution crop as `rgb` and paste the result back. Apply despill and alpha
 contrast after the refiner, as you would for CorridorKey's own output.
 
+ONNX graphs of the refiner (fixed tile batches of 256, 384, 512 and 768) are in
+[mantelmoritz/CorridorKey-ONNX-Graphs](https://huggingface.co/mantelmoritz/CorridorKey-ONNX-Graphs).
+
 ## Results
 
-Edge error = mean absolute alpha error inside the edge band (×1000), measured against CorridorKey 2.0 offline at
-native resolution; flicker = temporal error of the frame-to-frame alpha change in the edge band (×1000). Lower is
-better. 32 consecutive frames per segment. None of these clips were used for training.
-
-- **Real stock footage:** 6 Pexels / Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p), curly / wavy / long hair,
-  headphones, headset, glasses.
-- **Real 4K, hair** and **Real 4K, transparency:** the two clips of the CorridorKey 1.0 Quality Benchmark
-  (2 segments each). The druid has curly hair, a beard and fur; the clown has tulle and water glasses.
-- **Generated streamers:** the 8 series-3 test clips of version 2 of the dataset. They are deliberately hard (props
-  in front of the person, partially covered screens, people leaving the frame).
-
-**Person crop, 512×512** (CorridorKey 2.0 Stage 1 on a tracked square around the person):
-
-| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame |
-|---|---|---|---|---|---|---|
-| Bilinear | 121.3 | 91.2 | 80.1 | 134.4 | 75.3 | — |
-| Colour guided filter | 51.2 | 45.3 | 64.1 | 103.8 | 33.1 | 14.4 |
-| Edge Refiner v9b | 32.5 | 32.7 | 61.9 | 88.9 | 22.0 | 24.4 |
-| Edge Refiner v10w | 30.9 | 30.6 | 62.3 | 84.9 | 20.5 | 25.8 |
-| **Edge Refiner v12** | **29.3** | **28.6** | **61.8** | **83.0** | **19.7** | 25.8 |
-
-**Full frame, 512×256** (cheaper, for 60 fps):
-
-| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame |
-|---|---|---|---|---|---|---|
-| Bilinear | 174.7 | 117.8 | 106.7 | 187.8 | 91.5 | — |
-| Colour guided filter | 61.3 | 53.4 | 76.4 | 137.5 | 34.9 | 11.8 |
-| Edge Refiner v9b | 41.7 | 46.3 | 73.6 | 127.7 | 26.4 | 16.5 |
-| Edge Refiner v10w | 39.4 | 45.2 | 73.5 | 125.0 | 25.0 | 17.6 |
-| **Edge Refiner v12** | **38.0** | **43.9** | **72.5** | **123.5** | **24.4** | 17.6 |
-
-GPU time is end-to-end per frame, CorridorKey 2.0 Stage 1 included, on an RTX 5070 Ti. It is the median over the
-6 real stock clips, measured without video I/O. v12 has the same architecture as v10w and therefore the same cost.
-The refiner itself costs about 5–6 ms with a budget of 256 tiles and 10–11 ms with 512 tiles. The tile count, not
-the frame resolution, dominates the cost.
-
-- Compared with the guided filter, the refiner (v12) reduces the edge error by 38–43 % on the real stock clips and
-  by 18–37 % on the 4K druid clip. Flicker drops by 30–40 %.
-- v12 against v10w, at the same cost: lower edge error on every test set, by 3–7 % on hair (real stock, druid)
-  and 1 % on the semi-transparent clown, and 2–4 % less flicker on the real clips.
-- On hair (druid) the gain is largest. On semi-transparent material (clown) it is small: there the result is limited
-  by what the low-resolution network sees.
+Edge error is the mean absolute alpha error inside the edge band (x1000), measured against CorridorKey 2.0 offline
+at native resolution. Flicker is the error of the frame-to-frame alpha change in the edge band (x1000). Lower is
+better. Each segment has 32 consecutive frames, and none of these clips were used for training.
+
+- Real stock footage: 6 Pexels and Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p) with curly, wavy and long
+  hair, headphones, a headset and glasses.
+- Real 4K, hair and Real 4K, transparency: the two clips of the CorridorKey 1.0 Quality Benchmark, 2 segments
+  each. The druid has curly hair, a beard and fur; the clown has tulle and water glasses.
+- Generated streamers: the 8 series-3 test clips of version 2 of the dataset. They are deliberately hard, with
+  props in front of the person, partly covered screens and people leaving the frame.
+
+Person crop, 512x512 (CorridorKey 2.0 Stage 1 on a tracked square around the person, 512 tiles):
+
+| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) |
+|---|---|---|---|---|---|
+| Bilinear | 121.3 | 91.2 | 80.1 | 134.4 | 75.3 |
+| Colour guided filter | 51.2 | 45.3 | 64.1 | 103.8 | 33.1 |
+| Edge Refiner v12 | 29.2 | 28.5 | 61.7 | 82.9 | 19.7 |
+| Edge Refiner v15 | 28.5 | 27.8 | 58.7 | 82.7 | 19.2 |
+
+Full frame, 512x256 (the facecam profile for 60 fps, 384 tiles):
+
+| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) |
+|---|---|---|---|---|---|
+| Bilinear | 174.7 | 117.8 | 106.7 | 187.8 | 91.5 |
+| Colour guided filter | 61.3 | 53.4 | 76.4 | 137.5 | 34.9 |
+| Edge Refiner v12 | 34.8 | 38.7 | 70.9 | 115.9 | 22.2 |
+| Edge Refiner v15 | 34.2 | 38.4 | 68.7 | 116.3 | 21.7 |
+
+End-to-end GPU time per frame on an RTX 5070 Ti, CorridorKey 2.0 Stage 1 included, without video I/O: about
+16.5 ms for the person crop and 12 to 13 ms for the full frame. v15 costs 0.1 to 0.4 ms more than v12.
+
+Against the guided filter, v15 has 44 % less edge error on the real stock clips with the person crop, 39 % less on
+the druid and 42 % less flicker. Against v12, v15 has 1 to 5 % less edge error on the real clips and about 2 % less
+flicker. On the generated streamers both are level within 0.4 %. The largest gains are on hair. On
+semi-transparent material such as the clown's tulle the result is limited by what the low-resolution network sees.
+
+In a matched comparison (same patches and steps, only the width changed), going from 24 to 32 channels lowered the
+edge error on all six of our test sets in both profiles. 48 channels gained nothing more and cost 6 to 7 ms per frame.
 
 ![Generated streamer clip from the test split, 1:1 crops](images/compare_generated_streamer.png)
 
 ## Training
 
-- **Target:** CorridorKey 2.0 offline at native resolution (alpha and foreground). The refiner is a student of
-  CorridorKey: it cannot be sharper than its teacher, and it inherits the teacher's mistakes.
-- **Inputs during training:** exactly what the real-time engine produces at run time. Training patches were cut
-  by running the real-time pipeline (512×512 person crop and 512×256 full frame, Stage 1 only) over every
-  training clip and computing the 11 input channels, mostly from the edge band.
-- **Data:**
+- Target: CorridorKey 2.0 offline at native resolution, alpha and foreground. The refiner is a student of
+  CorridorKey. It cannot be sharper than its teacher and it inherits the teacher's mistakes.
+- Inputs during training are exactly what the real-time engine produces at run time. The training patches were cut
+  by running the real-time pipeline (512x512 person crop and 512x256 full frame, Stage 1 only) over every training
+  clip and computing the 11 input channels, mostly from the edge band.
+- Data:
   - 32 segments of real stock green-screen footage (Pexels, Mixkit)
-  - the same frames re-composited from the teacher's foreground and alpha over procedural green/teal screens, also
-    passed through a "bad camera" model (sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG, exposure errors)
+  - the same frames re-composited from the teacher's foreground and alpha over procedural green and teal screens,
+    also passed through a camera model with sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG and exposure errors
   - the 207 training clips of version 2 of the
     [CorridorKey Synthetic Streamers](https://huggingface.co/datasets/mantelmoritz/CorridorKey-Synthetic-Streamers)
-    dataset: every clip reviewed by eye, broken labels removed, everything in front of the screen kept as
-    foreground (mugs, microphones, a second person)
+    dataset. Every clip was reviewed by eye, broken labels were removed, and everything in front of the screen
+    (mugs, microphones, a second person) counts as foreground.
 
-  Real footage is sampled 3× more often than its share. 110,000 patches, loss on alpha, alpha gradients, composite
-  and foreground. v12: 96,000 steps (v10w: 24,000 steps on the unreviewed version 1 of the dataset).
-- **Tried and dropped for v12:** removing screen colour from the foreground before re-compositing (makes the
-  synthetic frames cleaner than real camera frames, edge error +3–5 %) and 25 % blue synthetic screens (worse on
-  every test set).
-- **Evaluation data was never used for training.**
+  Real footage is sampled 3 times as often as its share. Loss on alpha, alpha gradients, composite and foreground.
+  v15: a random 90,000 patches held in memory, 96,000 steps. v12: 110,000 patches, 96,000 steps.
+- Tried and dropped: removing screen colour from the foreground before re-compositing (it makes the synthetic frames
+  cleaner than real camera frames; edge error rose by 3 to 5 %), 25 % blue synthetic screens (worse on every test
+  set) and 48 channels (see above).
+- The evaluation clips were never used for training.
 
 ## Limitations
 
 ![Limitation: free-flying strands far from the head](images/compare_facecam_1080.png)
 
-- **The teacher is the ceiling.** Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too.
-- **It can only refine what the low-resolution network found.** Single strands far outside CorridorKey's
-  low-resolution matte are outside the refiner band and are lost, as in the image above. Semi-transparent objects
+- The teacher is the ceiling. Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too.
+- It can only refine what the low-resolution network found. Single strands far outside CorridorKey's
+  low-resolution matte lie outside the refiner band and are lost, as in the image above. Semi-transparent objects
   (glass, tulle, water) depend almost entirely on the low-resolution network.
-- Trained for **green (and teal / blue-green) screens** with people in front, mostly facecam and upper-body framing.
-  It is not a general background-removal model.
+- It was trained for green, teal and blue-green screens with people in front, mostly facecam and upper-body
+  framing. It is not a general background-removal model.
 - The numbers above compare against CorridorKey 2.0, not against hand-made ground truth.
 
-## License
+## Licence
 
-**Corridor Key Licence, Version 2.0** (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full
-text is in [`LICENSE`](LICENSE).
+Corridor Key Licence, Version 2.0 (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full text is
+in [`LICENSE`](LICENSE).
 
-This refiner was trained on the outputs of CorridorKey 2.0 and only works together with it. It is therefore
-released as a CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4,
-§5). In short:
+The refiner was trained on the outputs of CorridorKey 2.0 and only works together with it, so it is released as a
+CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4, §5). In short:
 
-- **Free of charge.** You may not sell it, repackage it as a standalone or competing product, or offer it as a paid
-  API or inference service.
-- **Share-alike:** derivatives must be released under the same licence, free of charge, and keep the
-  "CorridorKey" name.
+- It is free of charge. You may not sell it, repackage it as a standalone or competing product, or offer it as a
+  paid API or inference service.
+- Derivatives must be released under the same licence, free of charge, and keep the "CorridorKey" name.
 - Images and video you produce with it are yours.
-- Commercial software integrations require a separate agreement with Corridor Digital
-  (contact@corridordigital.com). Organisations above US $20M revenue need CorridorKey plugin seats
-  (https://corridorkey.com).
+- Commercial software integrations need a separate agreement with Corridor Digital (contact@corridordigital.com).
+  Organisations above US $20M revenue need CorridorKey plugin seats (https://corridorkey.com).
 
 ### Attribution and modifications
 
-- **CorridorKey 2.0** © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration
-  with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0 ·
+- CorridorKey 2.0 © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration
+  with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0,
   https://corridorkey.com
 - CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google
   DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository.
-- **Changes relative to CorridorKey 2.0:** this repository adds a new, separately trained edge-refinement network and
+- Changes relative to CorridorKey 2.0: this repository adds a new, separately trained edge-refinement network and
   its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed.
 - Example footage: CorridorKey 1.0 Quality Benchmark
   (https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels

corridorkey_edge_refiner_v15.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b38746544edbdaa207723ffe1dd6e799e1fa7f176bd74ad4c7c91d5dd84abf74
+size 162320