skblv/lemonfm-linear-probes-surgical-video

LemonFM linear probes — surgical video understanding benchmarks

Linear-probe heads trained on frozen LEMON (ConvNeXt-Large) features for the surgical video understanding leaderboard. One subfolder per benchmark, each containing probe_weights.pt, metrics.json, and the probe training curve.

Trained as baselines for the SDSC × Chicago Booth surgical video understanding leaderboard.

Prompt examples

These closed-set examples mirror the leaderboard format, not a text-input API for this checkpoint.

CholecT50 instruments

[surgical frame]

Which instruments are visible in this laparoscopic cholecystectomy frame?
Select every matching label.

- grasper
- bipolar
- hook
- scissors
- clipper
- irrigator

PitVis-2023 instruments

[surgical frame]

Which instruments are visible in this endoscopic pituitary-surgery frame?
Select every matching label.

- bipolar forceps
- cottle
- cup forceps
- dural scissors
- freer elevator
- haemostatic foam
- irrigation syringe
- kerrisons
- micro doppler
- nasal cutting forceps
- pituitary rongeurs
- retractable knife
- ring curette
- spatula dissector
- stealth pointer
- suction
- surgical drill
- tissue glue

SurgVU instruments

[surgical frame]

Which instruments are visible in this robotic surgical frame?
Select every matching label.

- bipolar dissector
- bipolar forceps
- cadiere forceps
- clip applier
- force bipolar
- grasping retractor
- monopolar curved scissors
- needle driver
- permanent cautery hook/spatula
- potts scissors
- prograsp forceps
- stapler
- suction irrigator
- synchroseal
- tenaculum forceps
- tip-up fenestrated grasper
- vessel sealer

DSAD anatomy

[surgical frame]

Which anatomical structures are visible in this laparoscopic frame?
Select every matching label.

- abdominal wall
- colon
- inferior mesenteric artery
- intestinal veins
- liver
- pancreas
- small intestine
- spleen
- stomach
- ureter
- uterus
- vesicular glands

PitVQA phase and step

[surgical frame]

What is the current surgical phase and surgical step in this endoscopic pituitary frame?
Choose one phase and one step.

Phase (choose one)
- closure
- nasal sphenoid
- sellar

Step (choose one)
- anterior sphenoidotomy
- debris clearance
- dural sealant
- durotomy
- fat graft placement
- gasket seal construct
- haemostasis
- nasal corridor creation
- nasal packing
- sellotomy
- septum displacement
- sphenoid sinus clearance
- synthetic graft placement
- tumour excision

CholecT50 verbs

[surgical frame]

Which surgical actions are being performed in this cholecystectomy frame?
Select every matching label.

- grasp
- retract
- dissect
- coagulate
- clip
- cut
- aspirate
- irrigate
- pack
- idle

SAP-Bench next action

[surgical frame]

Based on this cholecystectomy frame, what should the surgeon do next?
Choose one label.

- Dissection
- Vessel Clipping
- Tissue Retraction
- Coagulation
- Aspiration

SAR-RARP50 suturing action

[surgical frame]

What suturing action is being performed in this frame?
Choose one label.

- Other
- Picking Up The Needle
- Positioning The Needle Tip
- Pushing The Needle Through The Tissue
- Pulling The Needle Out Of The Tissue
- Tying A Knot
- Cutting The Suture
- Returning Or Dropping The Needle

Protocol

Features are extracted from the frozen LemonFM ConvNeXt-Large encoder; a single linear layer is trained with BCE loss on the training split (seed 42). Decision rules: per-class threshold for multi-label instrument and anatomy tasks, grouped argmax for PitVQA (one phase + one step), plain argmax for the single-label action tasks. Training code: s56_lemonfm_probe.py (instruments) and s62_lemonfm_probe_new.py (newer domains).

The LemonFM encoder itself is not included; obtain it from its authors (Visurg AI) under their terms. This repository contains only the probe heads trained by us.

Results (full validation splits, 95% bootstrap CI)

Benchmark	Task	Exact match	Micro-F1
cholect50/	6-instrument multi-label	67.7% (64.7–70.7)	85.62% (84.16–87.13)
pitvis/	18-instrument multi-label	63.6% (60.3–66.7)	68.75% (65.65–71.63)
surgvu/	17-instrument multi-label	40.0% (37.0–43.1)	68.8% (66.79–70.77)
dsad/	12-structure multi-label anatomy	17.3% (15.7–19.1)	57.6% (56.2–59.1)
pitvqa/	phase + step recognition	63.3% (62.7–63.8)	76.1% (75.7–76.5)
sapbench/	5-way next action	46.2% (41.4–51.6)	= exact match
sarrarp50/	8-way suturing gesture	42.9% (39.3–47.0)	= exact match

For single-label tasks micro-F1 coincides with accuracy. Full comparisons on the leaderboard.

References

• Skobelev, K., Fithian, E., Baranovski, Y., et al. A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling. arXiv:2603.27341, 2026.
• Dataset: Nwoye, C. I., Yu, T., Gonzalez, C., et al. Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos. Medical Image Analysis, 78, 102433 (2022).
• Dataset: Das, A., Khan, D. Z., Psychogyios, D., et al. PitVis-2023 Challenge: Workflow Recognition in Videos of Endoscopic Pituitary Surgery. Medical Image Analysis, 106, 103716 (2025).
• Dataset: Zia, A., Berniker, M., Nespolo, R., et al. Surgical Visual Understanding (SurgVU) Dataset. arXiv:2501.09209, 2025.
• Dataset: Carstens, M., Rinner, F. M., Bodenstedt, S., et al. The Dresden Surgical Anatomy Dataset for Abdominal Organ Segmentation in Surgical Data Science. Scientific Data, 10, 3 (2023).
• Dataset: He, R., Xu, M., Das, A., et al. PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery. MICCAI 2024.
• Dataset: Xu, M., Huang, Z., Imans, D., et al. SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planning. arXiv:2506.07196, 2025.
• Dataset: Psychogyios, D., Colleoni, E., Van Amsterdam, B., et al. SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge. arXiv:2401.00496, 2024.

Limitations

Research baselines only. Not a medical device. Probe heads are only meaningful on top of the exact LemonFM encoder and preprocessing used in the training script.