Upload Chimera 510M at step 230500

README.md

@@ -31,8 +31,12 @@ thumbnail: auron_banner.png
 - **GDN:Attn ratio:** 3:1 (every 4th layer is attention)
 - **Virtual equivalent:** ~1,020,434,560 params
 
+## Training Curves
+
+![Training Curves](training_curves.png)
+
 ## Training
-- **Step:** 229,000
+- **Step:** 230,500
 - **Data:** Mixed (75% FineWeb-Edu, 18% StarCoder, 5% FineMath, 2% UltraChat)
 - **Optimizer:** Muon + AdamW (decoupled embedding LR)
 - **Schedule:** WSD (Warmup-Stable-Decay)

config.json

@@ -26,7 +26,7 @@
   "architecture": "Chimera",
   "config_class": "ChimeraConfig",
   "topology": "4 bottom + 4x3 top = 16 virtual",
-  "step": 229000,
+  "step": 230500,
   "total_params": 510217280,
   "size_label": "510M",
   "model_type": "zara-ml"

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ed1d709dc0fbbe9152d1eb828d8c6010e380d0fdfd62dbf28bbf1332e97348b0
+oid sha256:46f8fade36fbf56f8b463bace4e63aa94d771f5bcd96b9b74325ea11484c7483
 size 1487197656

training.log

@@ -96,3 +96,28 @@ The rocket’s mass one was due to the fact that it’s a large rocket. So, what
 There has been a lot of buzz around the anti-inflammatory properties of turmeric. In fact, a recent study in the Journal of the American Medical Association found that there was a significant improvement on the blood sugar, and that turmeric was effective at relieving pain
 step 229200/250000 | loss 2.7851 | lr 7.46e-04 emb 3.73e-04 | 700ms/step | 46,825 tok/s | epoch 1
 step 229400/250000 | loss 2.7797 | lr 7.40e-04 emb 3.70e-04 | 693ms/step | 47,318 tok/s | epoch 1
+  >>> val_loss: 3.0922 | bpt: 4.4611 | true_bpb: 1.4351 *BEST*
+  >>> [The] The arguably the best-selling book for classroom use, this book should be a staple of any pre-K classroom. It will help your students master basic math skills in a fun, interactive way. A fun and engaging way to teach kids about basic math skills.
+- 31 pages
+- 14 pages
+- 313 colors
+- 3 items
+- 34 versions
+
+  >>> [Scientists have discovered] Scientists have discovered that living organisms do not have a wombs, and the ones that do are able to eat, have the muscles that enable them to walk and flutter and breathe. But some animals, like tiny mice and spiders, have a true uterus, and lay eggs in a hollow. Researchers have found that, in the lab tests, scientists can find out whether the womb is functioning properly. It is possible that
+step 229600/250000 | loss 2.7647 | lr 7.35e-04 emb 3.68e-04 | 695ms/step | 47,123 tok/s | epoch 1
+step 229800/250000 | loss 2.7489 | lr 7.29e-04 emb 3.65e-04 | 690ms/step | 47,504 tok/s | epoch 1
+step 230000/250000 | loss 2.7523 | lr 7.24e-04 emb 3.62e-04 | 685ms/step | 47,827 tok/s | epoch 1
+  >>> val_loss: 3.0918 | bpt: 4.4606 | true_bpb: 1.4349 *BEST*
+  >>> [The] The Bardwell Debate:
+The upper limit on the number of words on a word from which the word can be derived is 44,000, and the lower limit, 6,900, on the number of words in which a word can be derived, at the end of the age, is 2,200. The word Mandarin is 100,
+  >>> [Scientists have discovered] Scientists have discovered a mechanism for concern on the molecule that may be causing a major health hazard, as well as be just a few hints of the world's way of life. A new study finds that there may be a new and increasing ingredient that may be contributing to environmental problems, and in some cases may be to blame for a number of different health problems.
+In a lecture, Dr. Le-King Chung from
+step 230200/250000 | loss 2.7702 | lr 7.18e-04 emb 3.59e-04 | 689ms/step | 47,559 tok/s | epoch 1
+step 230400/250000 | loss 2.7721 | lr 7.11e-04 emb 3.56e-04 | 685ms/step | 47,835 tok/s | epoch 1
+  >>> val_loss: 3.0886 | bpt: 4.4559 | true_bpb: 1.4334 *BEST*
+  >>> [The] The Sunderboal and the Sunderboal Bank are the main investments of the Indian National Bank. However, first the Sunderboal Bank, Nehru Bank and Jajana Bank were launched in 1957. The bank was also known for developing and developing in the middle of the Mysore region. From 1977 to 1982, the financial stream
+  >>> [Scientists have discovered] Scientists have discovered a new way to protect babies from the dreaded Zombie Coronavirus. The virus’s respiratory viruses are spread directly from infected people to healthy people, so the Centers for Disease Control and Prevention says it’s important to stop the spread of respiratory viruses and keep them out of the environment. The CDC says the virus’s influence can be found on any surface, including the nose, eyes, mouth, etc.
+If you
+step 230600/250000 | loss 2.7692 | lr 7.05e-04 emb 3.53e-04 | 687ms/step | 47,664 tok/s | epoch 1
+step 230800/250000 | loss 2.7698 | lr 6.98e-04 emb 3.49e-04 | 684ms/step | 47,904 tok/s | epoch 1