litert-community/RF-DETR-Nano-LiteRT

RF-DETR Nano — LiteRT (CompiledModel GPU)

RF-DETR (Roboflow 2025, an LW-DETR derivative) object detection, converted to LiteRT and running 100% on the CompiledModel GPU (ML Drift) on a phone — the first transformer/DETR detector to ride the LiteRT GPU API with no CPU/ONNX fallback.

RF-DETR is a transformer detector (windowed DINOv2-S backbone + deformable-attention DETR decoder). Off-the-shelf it is GPU-incompatible (deformable grid_sample → GATHER_ND, windowed attention → 5D/6D tensors, two-stage query selection → TOPK/GATHER). Here it is converted with litert-torch and split into two GPU graphs with a tiny host step between them, so the whole detector runs on the GPU.

Files

File	What it is	Size (fp16)
rfdetr_graphA_fp16.tflite	backbone + encoder + proposal heads → enc_class[1,576,91], enc_coord[1,576,4], memory[1,576,256]	48.6 MB
rfdetr_graphB_fp16.tflite	two-stage combine + decoder + heads → boxes[1,300,4] (cxcywh), logits[1,300,91]	7.6 MB

How it runs (two-graph split)

image[1,3,384,384]
  →[GPU Graph A]→ enc_class, enc_coord, memory
  →[host: top-300 by max class score → gather coords]→ refpoint_ts[1,300,4]
  →[GPU Graph B  (memory, refpoint_ts)]→ boxes[1,300,4], logits[1,300,91]
  →[host: sigmoid + threshold + cxcywh→xyxy + per-class NMS]→ detections

The two-stage query selection (TOPK/GATHER) has no GPU op, but the proposal grid is image-independent, so the model splits at exactly that point — the standard two-stage-DETR edge split. Both graphs are 100% GPU-resident.

Minimal usage

Android (Kotlin, CompiledModel GPU)

val ga = CompiledModel.create(context.assets, "rfdetr_graphA_fp16.tflite",
    CompiledModel.Options(Accelerator.GPU), null)
val gb = CompiledModel.create(context.assets, "rfdetr_graphB_fp16.tflite",
    CompiledModel.Options(Accelerator.GPU), null)
val aIn = ga.createInputBuffers(); val aOut = ga.createOutputBuffers()
val bIn = gb.createInputBuffers(); val bOut = gb.createOutputBuffers()
aIn[0].writeFloat(chw)               // [1,3,384,384] RGB, ImageNet mean/std, NCHW
ga.run(aIn, aOut)                    // -> enc_class[1,576,91], enc_coord[1,576,4], memory[1,576,256]
// host step: top-300 by max class logit -> gather enc_coord -> refpoint_ts[1,300,4]
// (resolve buffer slots by float size; see the Python below / litert-samples object_detection sample)
bIn[0].writeFloat(memory); bIn[1].writeFloat(refpointTs)
gb.run(bIn, bOut)
val boxes = bOut[0].readFloat()      // [1,300,4] cxcywh in [0,1]
val logits = bOut[1].readFloat()     // [1,300,91] -> sigmoid + threshold + per-class NMS

Python (desktop verification)

import numpy as np
from PIL import Image
from ai_edge_litert.interpreter import Interpreter

NP_, NQ, NC, H = 576, 300, 91, 256
img = Image.open("photo.jpg").convert("RGB").resize((384, 384))
x = np.asarray(img, np.float32) / 255.0
x = ((x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).astype(np.float32).transpose(2, 0, 1)[None]

def run(path, feeds):  # feed/fetch tensors by shape (converter slot order is arbitrary)
    it = Interpreter(model_path=path); it.allocate_tensors()
    for d in it.get_input_details(): it.set_tensor(d["index"], feeds[tuple(d["shape"][1:])])
    it.invoke(); return {tuple(d["shape"][1:]): it.get_tensor(d["index"]) for d in it.get_output_details()}

a = run("rfdetr_graphA_fp16.tflite", {(3, 384, 384): x})
enc_cls, enc_coord, mem = a[(NP_, NC)][0], a[(NP_, 4)][0], a[(NP_, H)][0]

top = np.argsort(-enc_cls.max(-1))[:NQ]                     # top-300 by max class logit
ts = enc_coord[top]                                         # gather -> refpoint_ts [300,4]

b = run("rfdetr_graphB_fp16.tflite", {(NP_, H): mem[None], (NQ, 4): ts[None]})
boxes, logits = b[(NQ, 4)][0], b[(NQ, NC)][0]               # cxcywh in [0,1] / 91-way (index = COCO category id)
score = 1 / (1 + np.exp(-logits.max(-1))); cls = logits.argmax(-1)
for q in np.where((score > 0.45) & (cls > 0))[0]:           # id 0 unused; + per-class NMS (IoU 0.6) in a real app
    cx, cy, w, h = boxes[q]
    print(f"COCO id {cls[q]:2d}  {score[q]:.2f}  xyxy=({cx-w/2:.3f},{cy-h/2:.3f},{cx+w/2:.3f},{cy+h/2:.3f})")

On-device (Pixel 8a, Tensor G3 — verified)

graph	nodes on GPU	time
Graph A	1381/1381 LITERT_CL	~22 ms
Graph B	404/404 LITERT_CL	~5 ms

Full pipeline ≈ 27 ms (model) / ~100 ms end-to-end incl. host pre/post-processing. On a real image the device chain reproduces the PyTorch detections at IoU 0.98–0.99 with matching class and score.

Preprocessing / outputs

• Input: square resize to 384×384, RGB, ImageNet mean/std ([0.485,0.456,0.406]/[0.229,0.224,0.225]), NCHW.
• Output: Graph B boxes are cxcywh normalized to [0,1]; logits are 91-way (index = COCO category id). Host applies sigmoid + score threshold + cxcywh→xyxy + per-class NMS.

Conversion notes

Converted with litert-torch (NCHW preserved — onnx2tf destroys ViT attention). Re-authoring (per-graph tflite-vs-torch correlation 1.0): windowed DINOv2 backbone (6D window-partition → ≤4D, SDPA → manual attention), deformable grid_sample → a GATHER/CAST-free tent-matmul, MSDeformAttn ≤4D, baked sine pos-embed, and a down-scaled fp16-safe LayerNorm in the projector and decoder (the Mali delegate computes in fp16, and those LayerNorm channel-sums otherwise overflow). The two-stage topk/gather runs on the host between the two graphs.

A runnable Android sample (CompiledModel GPU) and the conversion scripts are in the official ai-edge-litert/litert-samples object_detection example.

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
TFLite benchmark_model (TfLiteGpuDelegateV2) — rfdetr_graphB_fp16.tflite	GPU (OpenCL)	68 / 404	did not run
TFLite benchmark_model (TfLiteGpuDelegateV2) — rfdetr_graphA_fp16.tflite	GPU (OpenCL)	1381 / 1381	did not run
TFLite benchmark_model — rfdetr_graphB_fp16.tflite	CPU (XNNPACK, 4 threads)	—	79.8 ms
TFLite benchmark_model — rfdetr_graphA_fp16.tflite	CPU (XNNPACK, 4 threads)	—	375.0 ms

Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.

Note that the GPU does not take the whole graph here (68 / 404 in rfdetr_graphB_fp16.tflite); the remainder runs on the CPU and the split costs a per-partition round trip.

Snapdragon NPU (Hexagon)

• rfdetr_graphA_fp16.tflite — the NPU is 1.22x faster than the GPU (15.11 ms against 18.45 ms) and loads 12.94x faster (176 ms against 2272 ms).
• rfdetr_graphB_fp16.tflite — the GPU is faster: 7.41 ms against 12.82 ms on the NPU, a factor of 1.73. The NPU still loads 10.43x faster (125 ms against 1299 ms).

file	backend	compiled	inference (median / min)	load
rfdetr_graphA_fp16.tflite	NPU (Hexagon v81)	on-device JIT	15.11 ms / 14.75 ms	176 ms
rfdetr_graphA_fp16.tflite	GPU (Adreno)	—	18.45 ms / 18.12 ms	2272 ms
rfdetr_graphB_fp16.tflite	NPU (Hexagon v81)	on-device JIT	12.82 ms / 12.17 ms	125 ms
rfdetr_graphB_fp16.tflite	GPU (Adreno)	—	7.41 ms / 7.12 ms	1299 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.76–0.78, where 1.0 is the throttling threshold.

The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. Those first compiles took 14 s to 36 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.

GPU wiring: GPU guide.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT benchmark_model tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).

File	Inference (median)	Spread (min–max)	Runs	Peak memory
rfdetr_graphA_fp16.tflite	458.4 ms	454.6–462.7 ms	150	180 MB
rfdetr_graphB_fp16.tflite	64.0 ms	62.7–70.7 ms	150	138 MB

License

Apache-2.0, inherited from roboflow/rf-detr.