Update README.md
Browse files
README.md
CHANGED
|
@@ -187,10 +187,34 @@ We used a two-stage curriculum:
|
|
| 187 |
| Stage 1 | ~3.5 hours |
|
| 188 |
| Stage 2 | ~30 minutes |
|
| 189 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 190 |
## Disclaimer
|
| 191 |
Our work was performed and arXiv'ed in parallel with [LLaVA-NeXT-Med: Medical Multimodal Large Language Model](https://ieeexplore.ieee.org/document/10936400) by Yunfei Guo, Wu Huang. It is **NOT** the same model but trained in a very similar fashion. We chose to add clarifier *-OLAB* to ours to avoid confusion.
|
| 192 |
|
| 193 |
-
##
|
| 194 |
|
| 195 |
```bibtex
|
| 196 |
@article{alyakin2026cnsobsidian,
|
|
@@ -206,7 +230,7 @@ Our work was performed and arXiv'ed in parallel with [LLaVA-NeXT-Med: Medical Mu
|
|
| 206 |
```bibtex
|
| 207 |
@misc{alyakin2025cnsobsidian-preprint,
|
| 208 |
title={Repurposing the scientific literature with vision-language models},
|
| 209 |
-
author={Anton Alyakin and Jaden Stryker and Daniel Alexander Alber and Karl L. Sangwon and
|
| 210 |
year={2025},
|
| 211 |
eprint={2502.19546},
|
| 212 |
archivePrefix={arXiv},
|
|
|
|
| 187 |
| Stage 1 | ~3.5 hours |
|
| 188 |
| Stage 2 | ~30 minutes |
|
| 189 |
|
| 190 |
+
## Technical Specifications
|
| 191 |
+
|
| 192 |
+
### Model Architecture and Objective
|
| 193 |
+
|
| 194 |
+
See above.
|
| 195 |
+
|
| 196 |
+
### Compute Infrastructure
|
| 197 |
+
|
| 198 |
+
NYU Langone’s UltraViolet HPC cluster.
|
| 199 |
+
|
| 200 |
+
#### Hardware
|
| 201 |
+
|
| 202 |
+
Training:
|
| 203 |
+
- 13 nodes × 8× NVIDIA H100 80GB = 104 GPUs total (via PyTorch FSDP)
|
| 204 |
+
|
| 205 |
+
Inference:
|
| 206 |
+
- 2× 80GB GPUs (A100/H100) (or analogous) for bfloat16 precision
|
| 207 |
+
|
| 208 |
+
#### Software
|
| 209 |
+
|
| 210 |
+
- Pytorch 2.4
|
| 211 |
+
- CUDA 12.1
|
| 212 |
+
- HuggingFace Transformers 4.44.0
|
| 213 |
+
|
| 214 |
## Disclaimer
|
| 215 |
Our work was performed and arXiv'ed in parallel with [LLaVA-NeXT-Med: Medical Multimodal Large Language Model](https://ieeexplore.ieee.org/document/10936400) by Yunfei Guo, Wu Huang. It is **NOT** the same model but trained in a very similar fashion. We chose to add clarifier *-OLAB* to ours to avoid confusion.
|
| 216 |
|
| 217 |
+
## Citation
|
| 218 |
|
| 219 |
```bibtex
|
| 220 |
@article{alyakin2026cnsobsidian,
|
|
|
|
| 230 |
```bibtex
|
| 231 |
@misc{alyakin2025cnsobsidian-preprint,
|
| 232 |
title={Repurposing the scientific literature with vision-language models},
|
| 233 |
+
author={Anton Alyakin and Jaden Stryker and Daniel Alexander Alber and Karl L. Sangwon and Brandon Duderstadt and Akshay Save and David Kurland and Spencer Frome and Shrutika Singh and Jeff Zhang and Eunice Yang and Ki Yun Park and Cordelia Orillac and Aly A. Valliani and Sean Neifert and Albert Liu and Aneek Patel and Christopher Livia and Darryl Lau and Ilya Laufer and Peter A. Rozman and Eveline Teresa Hidalgo and Howard Riina and Rui Feng and Todd Hollon and Yindalon Aphinyanaphongs and John G. Golfinos and Laura Snyder and Eric Leuthardt and Douglas Kondziolka and Eric Karl Oermann},
|
| 234 |
year={2025},
|
| 235 |
eprint={2502.19546},
|
| 236 |
archivePrefix={arXiv},
|