Re-export with transformers-compatible RampNetModel wrapper + corrected card (fixes #19); weights unchanged from v1.0-paper

README.md

@@ -1,82 +1,141 @@
----
-tags:
-- curb ramp detection
-- accessibility
-license: mit
-datasets:
-- projectsidewalk/rampnet-dataset
-base_model:
-- timm/convnextv2_base.fcmae_ft_in22k_in1k_384
-pipeline_tag: object-detection
----
-
-**RampNet** is a two-stage pipeline that addresses the scarcity of curb ramp detection datasets by using government location data to automatically generate over 210,000 annotated Google Street View panoramas. This new dataset is then used to train a state-of-the-art curb ramp detection model that significantly outperforms previous efforts. In this repo, we provide the checkpoint for our curb ramp detection model.
-
-
-
-**Usage:**
-
-*For a step-by-step walkthrough, see our [Google Colab notebook](https://colab.research.google.com/drive/1TOtScud5ac2McXJmg1n_YkOoZBchdn3w?usp=sharing), which includes a visualization in addition to the code below.*
-```py
-import torch
-from transformers import AutoModel
-from PIL import Image
-import numpy as np
-from torchvision import transforms
-from skimage.feature import peak_local_max
-
-IMAGE_PATH = "example.jpg"
-DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-
-model = AutoModel.from_pretrained("projectsidewalk/rampnet-model", trust_remote_code=True).to(DEVICE).eval()
-
-preprocess = transforms.Compose([
-    transforms.Resize((2048, 4096), interpolation=transforms.InterpolationMode.BILINEAR),
-    transforms.ToTensor(),
-    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
-])
-
-img = Image.open(IMAGE_PATH).convert("RGB")
-img_tensor = preprocess(img).unsqueeze(0).to(DEVICE)
-
-with torch.no_grad():
-    heatmap = model(img_tensor).squeeze().cpu().numpy()
-
-peaks = peak_local_max(np.clip(heatmap, 0, 1), min_distance=10, threshold_abs=0.5)
-scale_w = img.width / heatmap.shape[1]
-scale_h = img.height / heatmap.shape[0]
-coordinates = [(int(c * scale_w), int(r * scale_h)) for r, c in peaks]
-
-# Coordinates of detected curb ramps
-print(coordinates)
-```
-
-![Example of Curb Ramp Points](https://i.postimg.cc/66fQm4m5/labeled.png)
-
-
-If you get an error like this (see below), then you need to update your transformers library version.
-```
->>> model = AutoModel.from_pretrained("projectsidewalk/rampnet-model", trust_remote_code=True)
-Traceback (most recent call last):
-  File "<stdin>", line 1, in <module>
-  File "/gscratch/scrubbed/jsomeara/envs/sidewalk-validator-ai/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained
-    model_class = add_generation_mixin_to_remote_model(model_class)
-  File "/gscratch/scrubbed/jsomeara/envs/sidewalk-validator-ai/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 726, in add_generation_mixin_to_remote_model
-    has_custom_generate = "GenerationMixin" not in str(getattr(model_class, "generate"))
-AttributeError: type object 'KeypointModel' has no attribute 'generate'
->>>
-```
-
-Citation:
-
-```bibtex
-@inproceedings{omeara2025rampnet,
-  author    = {John S. O'Meara and Jared Hwang and Zeyu Wang and Michael Saugstad and Jon E. Froehlich},
-  title     = {{RampNet: A Two-Stage Pipeline for Bootstrapping Curb Ramp Detection in Streetscape Images from Open Government Metadata}},
-  booktitle = {{ICCV'25 Workshop on Vision Foundation Models and Generative AI for Accessibility: Challenges and Opportunities (ICCV 2025 Workshop)}},
-  year      = {2025},
-  doi       = {https://doi.org/10.48550/arXiv.2508.09415},
-  url       = {https://cv4a11y.github.io/ICCV2025/index.html},
-  note      = {DOI: forthcoming}
-}
-```
+---
+license: mit
+library_name: transformers
+pipeline_tag: keypoint-detection
+tags:
+  - curb-ramp-detection
+  - accessibility
+  - street-view
+  - keypoint-heatmap
+datasets:
+  - projectsidewalk/rampnet-dataset
+base_model:
+  - timm/convnextv2_base.fcmae_ft_in22k_in1k_384
+---
+
+# RampNet Curb Ramp Detection Model
+
+Stage-2 model from **RampNet: A Two-Stage Pipeline for Bootstrapping Curb Ramp Detection in
+Streetscape Images from Open Government Metadata** (O'Meara et al., ICCV'25 CV4A11y workshop,
+[arXiv:2508.09415](https://arxiv.org/abs/2508.09415)).
+
+Takes a 2048x4096 equirectangular street-view panorama (ImageNet-normalized) and predicts a
+512x1024 heatmap of curb ramp locations. Extract detections with `skimage.feature.peak_local_max`.
+
+## Provenance
+
+| Field | Value |
+| :--- | :--- |
+| Training code | https://github.com/ProjectSidewalk/RampNet @ `842ec33` |
+| Source checkpoint | `epoch_1_step_9378.pth` (sha256 prefix `b0c3ff7a10fc`) |
+| Training dataset | [projectsidewalk/rampnet-dataset](https://huggingface.co/datasets/projectsidewalk/rampnet-dataset) revision `main` |
+| Exported | 2026-07-24 by `scripts/export_hf_model.py` |
+
+The model **weights are unchanged** from the originally published artifact — this revision only
+fixes the packaging (a `transformers`-compatible wrapper; see [Requirements](#requirements)) and
+corrects the evaluation numbers (see [Erratum](#erratum-evaluation-metrics)). The prior artifact
+remains addressable at its tagged Hub revision.
+
+## Evaluation (1,000-panorama manually labeled gold set)
+
+| Metric | Value |
+| :--- | :--- |
+| Average Precision (interpolated, full sweep) | 0.9205 |
+| Precision @ threshold 0.55 | 0.9492 |
+| Recall @ threshold 0.55 | 0.8727 |
+| Ground-truth points | 3919 |
+| Matching radius (normalized) | 0.022 |
+| Flip TTA | on |
+
+**Important:** these numbers were measured **with horizontal-flip test-time augmentation**
+(evaluate the original and mirrored panorama, combine heatmaps with elementwise max). Single-pass
+inference will land somewhat below them; derive your own threshold curve without TTA before
+choosing an operating point. Average Precision is the interpolated area under the full
+precision–recall sweep; precision and recall are reported at the operating threshold.
+
+### Erratum: Evaluation Metrics
+
+After publication we found that the evaluation protocol described in §3.3 of the paper (and
+implemented at tag [`v1.0-iccv2025`](https://github.com/ProjectSidewalk/RampNet/tree/v1.0-iccv2025))
+differs from standard detection evaluation in two ways that bias precision and recall upward:
+predictions matching an already-claimed ground-truth point were counted as additional true
+positives rather than false positives, and one prediction could satisfy multiple ground-truth
+points. The numbers above use corrected one-to-one matching. For transparency, the originally
+published figures are kept here alongside the corrected ones:
+
+| Metric | Originally published | Corrected (this revision) |
+| :--- | :--- | :--- |
+| Precision @ 0.55 | 0.938 | **0.949** |
+| Recall @ 0.55 | 0.935 | **0.873** |
+| Average Precision | 0.9236 | **0.9205** |
+
+Swapping only the matching rule on identical model outputs moves precision by −1.0 points and
+recall by −4.4 points; the remainder is reproduction drift (environment, JPEG re-encoding). The
+comparison with prior work is unaffected (both systems were scored under the same protocol, and the
+gap is far larger than the correction). Full write-up:
+[`docs/eval_protocol_verification.html`](https://github.com/ProjectSidewalk/RampNet/blob/main/docs/eval_protocol_verification.html)
+and the repository README's erratum.
+
+## Choosing a detection threshold
+
+- `0.55` is the recommended default operating point (see evaluation above).
+- Sweeping thresholds: run `stage_two/evaluate.py` in the training repo, which emits full
+  precision/recall-vs-confidence curves as CSV.
+- Per-city deployments should calibrate on ~100 locally labeled panoramas; see the repo README's
+  "Choosing a Detection Threshold" section.
+
+## Requirements
+
+Loads via `trust_remote_code`, so a compatible `transformers` is required:
+
+- `transformers >= 5.13` — supported (the custom-model auto-class contract this
+  version introduced is satisfied; see issue #19).
+- `transformers 5.12.x` — supported.
+
+Both are covered by a round-trip load smoke test (`tests/test_hf_load.py` in the
+training repo). Earlier revisions of this model fail to load on
+`transformers >= 5.13` with `AttributeError: ... 'KeypointModel' has no attribute
+'register_for_auto_class'` (or `... has no attribute 'generate'`) — pull the current revision.
+
+## Usage
+
+*For a step-by-step walkthrough, see the [Google Colab notebook](https://colab.research.google.com/drive/1TOtScud5ac2McXJmg1n_YkOoZBchdn3w?usp=sharing),
+which adds a visualization to the code below.*
+
+```python
+import torch
+from transformers import AutoModel
+from PIL import Image
+import numpy as np
+from torchvision import transforms
+from skimage.feature import peak_local_max
+
+DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model = AutoModel.from_pretrained("projectsidewalk/rampnet-model", trust_remote_code=True).to(DEVICE).eval()
+
+preprocess = transforms.Compose([
+    transforms.Resize((2048, 4096), interpolation=transforms.InterpolationMode.BILINEAR),
+    transforms.ToTensor(),
+    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+])
+
+img = Image.open("panorama.jpg").convert("RGB")
+with torch.no_grad():
+    heatmap = model(preprocess(img).unsqueeze(0).to(DEVICE)).squeeze().cpu().numpy()
+
+peaks = peak_local_max(np.clip(heatmap, 0, 1), min_distance=10, threshold_abs=0.55)
+scale_w, scale_h = img.width / heatmap.shape[1], img.height / heatmap.shape[0]
+print([(int(c * scale_w), int(r * scale_h)) for r, c in peaks])
+```
+
+## Citation
+
+```bibtex
+@inproceedings{omeara2025rampnet,
+  author    = {John S. O'Meara and Jared Hwang and Zeyu Wang and Michael Saugstad and Jon E. Froehlich},
+  title     = {{RampNet: A Two-Stage Pipeline for Bootstrapping Curb Ramp Detection in Streetscape Images from Open Government Metadata}},
+  booktitle = {{ICCV'25 Workshop on Vision Foundation Models and Generative AI for Accessibility: Challenges and Opportunities (ICCV 2025 Workshop)}},
+  year      = {2025},
+  doi       = {https://doi.org/10.48550/arXiv.2508.09415},
+}
+```

config.json

@@ -1,11 +1,33 @@
-{
-  "_name_or_path": "projectsidewalk/rampnet-model",
-  "architectures": [
-    "KeypointModel"
-  ],
-  "auto_map": {
-    "AutoModel": "modeling.KeypointModel"
-  },
-  "heatmap_size": [ 512, 1024 ],
-  "model_type": "convnextv2"
-}
+{
+  "architectures": [
+    "RampNetModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_rampnet.RampNetConfig",
+    "AutoModel": "modeling_rampnet.RampNetModel"
+  },
+  "dtype": "float32",
+  "heatmap_size": [
+    512,
+    1024
+  ],
+  "image_mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "image_std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "input_size": [
+    2048,
+    4096
+  ],
+  "model_type": "rampnet",
+  "recommended_min_distance": 10,
+  "recommended_threshold": 0.55,
+  "transformers_version": "5.14.1",
+  "tta_recommended": true
+}

configuration_rampnet.py

@@ -0,0 +1,25 @@
+from transformers import PretrainedConfig
+
+
+class RampNetConfig(PretrainedConfig):
+    model_type = "rampnet"
+
+    def __init__(
+        self,
+        input_size=(2048, 4096),
+        heatmap_size=(512, 1024),
+        image_mean=(0.485, 0.456, 0.406),
+        image_std=(0.229, 0.224, 0.225),
+        recommended_threshold=0.55,
+        recommended_min_distance=10,
+        tta_recommended=True,
+        **kwargs,
+    ):
+        self.input_size = list(input_size)
+        self.heatmap_size = list(heatmap_size)
+        self.image_mean = list(image_mean)
+        self.image_std = list(image_std)
+        self.recommended_threshold = recommended_threshold
+        self.recommended_min_distance = recommended_min_distance
+        self.tta_recommended = tta_recommended
+        super().__init__(**kwargs)

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4122c254fde7de4fce26e61604f3cdce3296b264ab9c2894113a09660d58d29
-size 360245468
+oid sha256:f2119e3becb0b551fa1470f7b7ba85b82122a3f73a6ed2a85609dd57617866b5
+size 360247780

modeling.py

@@ -1,27 +0,0 @@
-
-import torch
-import torch.nn as nn
-import timm
-from huggingface_hub import PyTorchModelHubMixin
-
-class KeypointModel(nn.Module, PyTorchModelHubMixin):
-    def __init__(self, config, **kwargs):
-        super().__init__()
-
-        upsample_size = config.heatmap_size
-
-        backbone = timm.create_model('convnextv2_base.fcmae_ft_in22k_in1k_384', pretrained=False)
-
-        self.feature_extractor = nn.Sequential(*list(backbone.children())[:-2])
-        in_channels = backbone.num_features
-        self.head = nn.Sequential(
-            nn.Conv2d(in_channels, 256, kernel_size=3, padding=1),
-            nn.ReLU(inplace=True),
-            nn.Upsample(size=upsample_size, mode='bilinear', align_corners=False),
-            nn.Conv2d(256, 1, kernel_size=1)
-        )
-
-    def forward(self, image):
-        features = self.feature_extractor(image)
-        heatmap = self.head(features)
-        return heatmap

modeling_rampnet.py

@@ -0,0 +1,26 @@
+from transformers import PreTrainedModel
+
+from .configuration_rampnet import RampNetConfig
+# rampnet_model.py is copied verbatim from rampnet/model.py by
+# scripts/export_hf_model.py at export time — it is generated, not a fork.
+from .rampnet_model import KeypointModel
+
+
+class RampNetModel(PreTrainedModel):
+    config_class = RampNetConfig
+    main_input_name = "pixel_values"
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = KeypointModel(
+            heatmap_size=tuple(config.heatmap_size),
+            pretrained_backbone=False,
+        )
+        # Required under transformers >= 5.x: initializes loading-related state
+        # (e.g. all_tied_weights_keys) that from_pretrained expects on every
+        # PreTrainedModel. Harmless no-op extras under 4.x.
+        self.post_init()
+
+    def forward(self, pixel_values):
+        """Returns the predicted curb ramp keypoint heatmap (B, 1, H, W)."""
+        return self.model(pixel_values)

rampnet_model.py

@@ -0,0 +1,44 @@
+import timm
+import torch.nn as nn
+
+BACKBONE_NAME = 'convnextv2_base.fcmae_ft_in22k_in1k_384'
+
+# Panorama model (stage_two): equirectangular 2048x4096 input -> 512x1024 heatmap.
+PANO_INPUT_SIZE = (2048, 4096)
+PANO_HEATMAP_SIZE = (512, 1024)
+
+# Crop model (stage_one): perspective 1024x352 input -> 256x88 heatmap.
+CROP_INPUT_SIZE = (1024, 352)
+CROP_HEATMAP_SIZE = (256, 88)
+
+IMAGENET_MEAN = [0.485, 0.456, 0.406]
+IMAGENET_STD = [0.229, 0.224, 0.225]
+
+
+class KeypointModel(nn.Module):
+    """The one canonical RampNet keypoint-heatmap model.
+
+    The state_dict key layout (feature_extractor as Sequential over
+    backbone.children()[:-2], i.e. dropping norm_pre and head) is the layout
+    every released checkpoint was trained and saved with — including the
+    HuggingFace weights. Do not restructure the modules (e.g. features_only=True
+    or named submodules): that changes the keys and breaks strict loading of
+    all existing checkpoints.
+    """
+
+    def __init__(self, heatmap_size=PANO_HEATMAP_SIZE, pretrained_backbone=False):
+        super().__init__()
+        backbone = timm.create_model(BACKBONE_NAME, pretrained=pretrained_backbone)
+        self.feature_extractor = nn.Sequential(*list(backbone.children())[:-2])
+        in_channels = backbone.num_features
+        self.head = nn.Sequential(
+            nn.Conv2d(in_channels, 256, kernel_size=3, padding=1),
+            nn.ReLU(inplace=True),
+            nn.Upsample(size=heatmap_size, mode='bilinear', align_corners=False),
+            nn.Conv2d(256, 1, kernel_size=1)
+        )
+
+    def forward(self, image):
+        features = self.feature_extractor(image)
+        heatmap = self.head(features)
+        return heatmap