Qwen3-8B-nla — Natural Language Autoencoder pair

This repository contains both halves of a Natural Language Autoencoder (NLA) pair, fine-tuned from Qwen/Qwen3-8B:

• Activation verbalizer (AV / actor) — at the repository root. Given a residual-stream activation vector from Qwen3-8B, it produces a short natural-language description of the vector's semantic content.
• Activation reconstructor (AR / critic) — in the ar/ subfolder. Given such a description, it predicts the original activation vector.

NLAs are interpretability tools: the verbalizer maps a hidden-state vector to a description, and the reconstructor maps the description back to a vector. The pair is trained end-to-end as an autoencoder — the verbalizer's RL reward is how accurately the reconstructor can recover the original activation from the description alone. These checkpoints are not useful as general-purpose language models — fine-tuning repurposes them entirely for activation decoding.

This is an independent training run using the NLA recipe, not one of the officially released NLA checkpoints.

• 📄 Paper: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
• 📄 Blog post: anthropic.com/research/natural-language-autoencoders
• Inference code + worked examples: kitft/nla-inference

Activation verbalizer (repository root)

• Base model: Qwen/Qwen3-8B (full fine-tune of the actor)
• Reads activations from: the residual stream at layer 24 of Qwen3-8B (d_model 4096, unnormalized)

Activation reconstructor (ar/)

• Architecture: the first 28 of Qwen3-8B's 36 layers, plus a linear reconstruction head (ar/value_head.safetensors, 4096×4096) applied to the output of its final block
• Input: the verbalizer's explanation, wrapped in the critic prompt template (see ar/nla_meta.yaml)
• Initialized from a supervised-trained critic, then trained jointly with the verbalizer through the same RL run
• Loading with vanilla transformers (subfolder="ar") gives the transformer only — the reconstruction head requires the nla-inference code

Training

• Data: ~500k activations captured at 5 positions each across ~100k FineWeb documents
• Run: RL (policy gradient, group size 8) against the reconstruction reward, global batch 256, lr 5e-6 (June 2026); both checkpoints are from iteration 400 of the same run

Usage

The verbalizer expects its training prompt, with the activation vector injected at the position of a reserved placeholder token:

• Injection token: ㈎ (U+320E), token id 149705, injection scale 150.0
• Machine-readable settings (prompt templates, token ids, scales) are in nla_meta.yaml (AV) and ar/nla_meta.yaml (AR)

See the nla-inference README for the embedding-injection mechanics (SGLang launch, NLAClient/NLACritic).