Echo-DSRN-114M-Base v0.1.1

#4
by mrs83 - opened
ethicalabs.ai org

We are currently pushing a major architectural fix for the Echo-DSRN-114M-Base model .

What was happening? The core of the DSRN architecture relies on a continuous memory bottleneck controlled by the surprise_lambda gate.

Under heavy TBPTT scaling, this raw parameter slipped into negative values, tricking the recurrent engine into mathematically "inverting" its prior state.

We have refactored the paramete and clamped it entirely through a F.softplus activation boundary. This guarantees that the memory gate remains strictly positive.

We've re-aligned the entire codebase and are currently recompiling the 114M-Base on 6B FineWeb Web + SmolTalk2 tokens

image

ethicalabs.ai org

v0.1.2 base model is ready

mrs83 changed discussion status to closed

Sign up or log in to comment