DINOv3 Video Tracking
π
76
In-browser video tracking, powered by Transformers.js
Gordon
GordonM
AI & ML interests
Data Science for good
Recent Activity
liked
a Space
about 1 month ago
webml-community/DINOv3-video-tracking
new activity
about 1 month ago
webml-community/DINOv3-video-tracking:Negative points
Organizations
liked
a
Space
about 1 month ago
Negative points
#2 opened about 1 month ago
by
GordonM
reacted to
Xenova's
post with π₯
about 1 month ago
Post
13445
Okay this is insane... WebGPU-accelerated semantic video tracking, powered by DINOv3 and Transformers.js! π€―
Demo (+ source code): webml-community/DINOv3-video-tracking
This will revolutionize AI-powered video editors... which can now run 100% locally in your browser, no server inference required (costs $0)! π
How does it work? π€
1οΈβ£ Generate and cache image features for each frame
2οΈβ£ Create a list of embeddings for selected patch(es)
3οΈβ£ Compute cosine similarity between each patch and the selected patch(es)
4οΈβ£ Highlight those whose score is above some threshold
... et voilΓ ! π₯³
You can also make selections across frames to improve temporal consistency! This is super useful if the object changes its appearance slightly throughout the video.
Excited to see what the community builds with it!
Demo (+ source code): webml-community/DINOv3-video-tracking
This will revolutionize AI-powered video editors... which can now run 100% locally in your browser, no server inference required (costs $0)! π
How does it work? π€
1οΈβ£ Generate and cache image features for each frame
2οΈβ£ Create a list of embeddings for selected patch(es)
3οΈβ£ Compute cosine similarity between each patch and the selected patch(es)
4οΈβ£ Highlight those whose score is above some threshold
... et voilΓ ! π₯³
You can also make selections across frames to improve temporal consistency! This is super useful if the object changes its appearance slightly throughout the video.
Excited to see what the community builds with it!
- 2 replies
reacted to
MoritzLaurer's
post with π
8 months ago
Post
2646
Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification!
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai , @LightOnIO and collaborators like @tomaarsen !
Paper and models here πhttps://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai , @LightOnIO and collaborators like @tomaarsen !
Paper and models here πhttps://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
- 3 replies
reacted to
MoritzLaurer's
post with π₯
8 months ago
Post
2293
π Releasing a new zeroshot-classifier based on ModernBERT! Some key takeaways:
- β‘ Speed & efficiency: It's multiple times faster and uses significantly less memory than DeBERTav3. You can use larger batch sizes and enabling bf16 (instead of fp16) gave me a ~2x speed boost as well
- π Performance tradeoff: It performs slightly worse than DeBERTav3 on average across my zeroshot classification task collection
- π§ Use cases: I recommend using it for scenarios requiring speed and a larger context window (8k).
- π‘ Whatβs next? Iβm preparing a newer version trained on better + longer synthetic data to fully leverage the 8k context window and improve upon the training mix of my older zeroshot-v2.0 models. I also hope that there will be a multilingual variant in the future.
Great work by
answerdotai
!
If youβre looking for a high-speed zeroshot classifier, give it a try!
π Resources below: π
Base model: MoritzLaurer/ModernBERT-base-zeroshot-v2.0
Large model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0
Updated zeroshot collection: MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f
ModernBERT collection with paper: answerdotai/modernbert-67627ad707a4acbf33c41deb
- β‘ Speed & efficiency: It's multiple times faster and uses significantly less memory than DeBERTav3. You can use larger batch sizes and enabling bf16 (instead of fp16) gave me a ~2x speed boost as well
- π Performance tradeoff: It performs slightly worse than DeBERTav3 on average across my zeroshot classification task collection
- π§ Use cases: I recommend using it for scenarios requiring speed and a larger context window (8k).
- π‘ Whatβs next? Iβm preparing a newer version trained on better + longer synthetic data to fully leverage the 8k context window and improve upon the training mix of my older zeroshot-v2.0 models. I also hope that there will be a multilingual variant in the future.
Great work by
If youβre looking for a high-speed zeroshot classifier, give it a try!
π Resources below: π
Base model: MoritzLaurer/ModernBERT-base-zeroshot-v2.0
Large model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0
Updated zeroshot collection: MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f
ModernBERT collection with paper: answerdotai/modernbert-67627ad707a4acbf33c41deb
liked
a
model
9 months ago
liked
a
Space
9 months ago
upvoted
a
collection
about 1 year ago
updated
a
Space
about 1 year ago
liked
2
Spaces
over 1 year ago
reacted to
Xenova's
post with β€οΈππ
over 1 year ago
Post
15213
I'm excited to announce that Transformers.js V3 is finally available on NPM! π₯ State-of-the-art Machine Learning for the web, now with WebGPU support! π€―β‘οΈ
Install it from NPM with:
πππ π @πππππππππππ/ππππππππππππ
or via CDN, for example: https://v2.scrimba.com/s0lmm0qh1q
Segment Anything demo: webml-community/segment-anything-webgpu
Install it from NPM with:
πππ π @πππππππππππ/ππππππππππππ
or via CDN, for example: https://v2.scrimba.com/s0lmm0qh1q
Segment Anything demo: webml-community/segment-anything-webgpu
- 5 replies
liked
a
model
over 1 year ago
reacted to
Xenova's
post with π
over 1 year ago
Post
14107
I can't believe this... Phi-3.5-mini (3.8B) running in-browser at ~90 tokens/second on WebGPU w/ Transformers.js and ONNX Runtime Web! π€― Since everything runs 100% locally, no messages are sent to a server β a huge win for privacy!
- π€ Demo: webml-community/phi-3.5-webgpu
- π§βπ» Source code: https://github.com/huggingface/transformers.js-examples/tree/main/phi-3.5-webgpu
- π€ Demo: webml-community/phi-3.5-webgpu
- π§βπ» Source code: https://github.com/huggingface/transformers.js-examples/tree/main/phi-3.5-webgpu
- 11 replies
upvoted
a
collection
over 1 year ago
reacted to
Xenova's
post with π₯
over 1 year ago
Post
15213
I'm excited to announce that Transformers.js V3 is finally available on NPM! π₯ State-of-the-art Machine Learning for the web, now with WebGPU support! π€―β‘οΈ
Install it from NPM with:
πππ π @πππππππππππ/ππππππππππππ
or via CDN, for example: https://v2.scrimba.com/s0lmm0qh1q
Segment Anything demo: webml-community/segment-anything-webgpu
Install it from NPM with:
πππ π @πππππππππππ/ππππππππππππ
or via CDN, for example: https://v2.scrimba.com/s0lmm0qh1q
Segment Anything demo: webml-community/segment-anything-webgpu
-
-
-
- 5 replies
liked
a
model
over 1 year ago
liked
a
Space
over 1 year ago