🛡️ Meet Spartacus-1B: Shattering the Memory Wall with True O(1) Inference! 🚀
text
🛡️ Meet Spartacus-1B: Shattering the Memory Wall with True O(1) Inference! 🚀
new_line
https://huggingface.co/NoesisLab/Spartacus-1B-Instruct
NoesisLab/Spartacus-1B-Instruct
model
resource
https://huggingface.co/NoesisLab/Spartacus-1B-Instruct
new_line
https://huggingface.co/spaces/NoesisLab/ChatSpartacus
NoesisLab/ChatSpartacus
space
resource
https://huggingface.co/spaces/NoesisLab/ChatSpartacus
new_line
At NoesisLab, we've entirely ripped out Softmax Attention and replaced it with Causal Monoid State Compression.
text
At NoesisLab, we've entirely ripped out Softmax Attention and replaced it with Causal Monoid State Compression.
new_line
Say hello to Spartacus-1B-Instruct (1.3B) 🗡️.
text
Say hello to Spartacus-1B-Instruct (1.3B) 🗡️.
new_line
Instead of maintaining a massive, ever-growing list of past tokens, Spartacus compresses its entire causal history into a fixed-size state matrix per head. The result?
text
Instead of maintaining a massive, ever-growing list of past tokens, Spartacus compresses its entire causal history into a fixed-size state matrix per head. The result?
new_line
⚡ True O(1) Inference: Memory footprint and generation time per token remain absolutely constant, whether you are on token 10 or token 100,000.
text
⚡ True O(1) Inference: Memory footprint and generation time per token remain absolutely constant, whether you are on token 10 or token 100,000.
new_line
🧠 Explicit Causality: We threw away RoPE and attention masks. The model learns when to forget using dynamic, content-aware vector decay.
text
🧠 Explicit Causality: We threw away RoPE and attention masks. The model learns when to forget using dynamic, content-aware vector decay.
new_line
🔥 Blazing Fast Training: Full hardware utilization via our custom Triton-accelerated JIT parallel prefix scan.
text
🔥 Blazing Fast Training: Full hardware utilization via our custom Triton-accelerated JIT parallel prefix scan.
new_line
📊 Zero-Shot Benchmarks that Hit Hard:
text
📊 Zero-Shot Benchmarks that Hit Hard:
new_line
O(1) architectures usually sacrifice zero-shot accuracy. Not Spartacus. It is punching way above its weight class, beating established sub-quadratic models (like Mamba-1.4B and RWKV-6-1.6B):
text
O(1) architectures usually sacrifice zero-shot accuracy. Not Spartacus. It is punching way above its weight class, beating established sub-quadratic models (like Mamba-1.4B and RWKV-6-1.6B):
new_line
🏆 ARC-Challenge: 0.3063 (vs Mamba 0.284)
text
🏆 ARC-Challenge: 0.3063 (vs Mamba 0.284)
new_line
🏆 ARC-Easy: 0.5518
text
🏆 ARC-Easy: 0.5518
new_line
🏆 PIQA: 0.6915
text
🏆 PIQA: 0.6915
🛡️ Meet Spartacus-1B: Shattering the Memory Wall with True O(1) Inference! 🚀
https://huggingface.co/NoesisLab/Spartacus-1B-Instruct
https://huggingface.co/spaces/NoesisLab/ChatSpartacus
At NoesisLab, we've entirely ripped out Softmax Attention and replaced it with Causal Monoid State Compression.
Say hello to Spartacus-1B-Instruct (1.3B) 🗡️.
Instead of maintaining a massive, ever-growing list of past tokens, Spartacus compresses its entire causal history into a fixed-size state matrix per head. The result?
⚡ True O(1) Inference: Memory footprint and generation time per token remain absolutely constant, whether you are on token 10 or token 100,000.
🧠 Explicit Causality: We threw away RoPE and attention masks. The model learns when to forget using dynamic, content-aware vector decay.
🔥 Blazing Fast Training: Full hardware utilization via our custom Triton-accelerated JIT parallel prefix scan.
📊 Zero-Shot Benchmarks that Hit Hard:
O(1) architectures usually sacrifice zero-shot accuracy. Not Spartacus. It is punching way above its weight class, beating established sub-quadratic models (like Mamba-1.4B and RWKV-6-1.6B):
🏆 ARC-Challenge: 0.3063 (vs Mamba 0.284)
🏆 ARC-Easy: 0.5518
🏆 PIQA: 0.6915
🤗
OzTianlu lambdago John6666 ehsankk