arxiv:2607.03065

Spectral Rewiring for Exploration, Purification, and Model Merging

Published on Jul 3

· Submitted by

Authors:

Abstract

Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.

View arXiv page View PDF Add to collection

Community

c7w

Paper submitter 1 day ago

zzl-ai

1 day ago

RL post-training works incredibly well, but its parameter update is still a black box.

🧐Our question: can we understand and locate the effective part of an RL update, remove the noisy directions, and even improve the model after training?

We find the answer is yes.

📌The reasoning-effective part of an RL update can be represented as a compact rewiring matrix in the base model’s spectral space.

🤖 This leads to SAR: a training-free post-hoc method that projects the raw RL update onto this compact reasoning core, enabling us to understand, purify, and merge RL-trained models.

Key results:
1️⃣ The reasoning core of RL is highly compact:
With less than 1% spectral parameters, SAR can recover or improve full-RL gains.

2️⃣ Dropping noisy directions helps:
For math, SAR solves the exploration degradation often observed after RL. It also improves agentic coding on large-scale in-house models.

3️⃣ SAR purifies mixed-domain RL:
On a 32B model jointly trained for math, code, instruction following, and chat, SAR improves coding and math exploration while keeping instruction following stable.

4️⃣ SAR makes model merging stronger:
After SAR purification, merged models can surpass the best single-domain experts. We observe the same trend on production-scale in-house models.

Overall, SAR is training-free, broadly useful, and gives us a new geometric lens on what RL is really changing inside reasoning models.