Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining
While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities.
69b3731a2ad6c5d4cf4869dc
Zhiyuan Zeng
69b3731a2ad6c5d4cf4869dd
Yichi Zhang
claimed_verified
2026-03-24T09:23:13.889Z
66bb3e4b4de574a3298f8f64
/avatars/6eff3ec8016b8f2a59182b5f046248c5.svg
Catill Zhang
Catill520
user
Catill520
69b3731a2ad6c5d4cf4869de
Yong Shan
69b3731a2ad6c5d4cf4869df
Kai Hua
claimed_verified
2026-03-13T16:50:17.009Z
64e851825ddcace745ba15bd
/avatars/7b6612c411222974d9ea181784eef915.svg
Kai Hua
kkish
user
kkish
69b3731a2ad6c5d4cf4869e0
Siyuan Fang
69b3731a2ad6c5d4cf4869e1
Zhaiyu Liu
69b3731a2ad6c5d4cf4869e2
Jiaheng Liu
69b3731a2ad6c5d4cf4869e3
Haozhe Wang
claimed_verified
2026-04-16T20:22:41.593Z
65bf52f0259bc6caeb74f8bf
/avatars/b38392e954466df784a5760ded5df804.svg
Haozhe Wang
JasperHaozhe
user
JasperHaozhe
69b3731a2ad6c5d4cf4869e4
Yining Zheng
69b3731a2ad6c5d4cf4869e5
Ming Ding
69b3731a2ad6c5d4cf4869e6
Ke Shen
69b3731a2ad6c5d4cf4869e7
Ge Zhang
claimed_verified
2026-03-16T10:43:52.466Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
zhangysk
user
zhangysk
69b3731a2ad6c5d4cf4869e8
Wenhao Huang
69b3731a2ad6c5d4cf4869e9
Xipeng Qiu
Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining
While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities.
Large language models trained on reconstructed agent trajectories from multi-agent simulations show improved performance in long-context understanding, coding proficiency, and agentic capabilities.
Large Language Models code generation long-horizon reasoning pre-training data static software repositories latent agentic trajectories reverse-engineering multi-agent simulation structural realities dependency graphs file hierarchies search-based optimization Chain-of-Thought CoT reasoning empirical results Llama-3-8B benchmarks long-context understanding coding proficiency agentic capabilities