arxiv:2610.34241

Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?

Published on Oct 25

· Submitted by

Vaibhav Jain on Oct 27

Upvote

Authors:

Vaibhav Jain

Abstract

Empirical studies of open-weight coding agents reveal that verified code success depends on serving setup, protocol design, and hidden behavioral checks rather than text coherence alone.

Generated by thinkingmachines/Inkling-Small

Making an open-weight language model generate text is not the same as making it finish coding work. This retrospective empirical systems experience report follows my engineering work from July to October 2026 across coding clients, model-specific serving recipes, vLLM, SGLang, llama.cpp, and MLX. Models behind a nominally compatible endpoint still differed in tool-call formatting, turn control, context behavior, cache reuse, and reliability. A served model, coherent text, executed tool calls, and code that passes independent behavioral checks proved to be distinct levels of progress. The evidence includes 4,165 coding-agent gateway turns from my own sessions, 623 debug jobs, and three separate delegation studies with hidden behavioral checks and a fixed Claude Opus 5.5 parent. Saved cold-prefix requests had a median time to first token of about 69.4 s, versus 2.8 s for high-reuse requests (observational). In a 45-trial study, direct coding passed 15/15 and delegation to a quantized Qwen3-Coder 30B-A3B worker passed 12/15 and 14/15. On a second 15-task suite under a fixed artifact contract, the completed replacement cohort had 45/45 for direct coding and 12/45 for a Nemotron Cascade 2 worker (11 worker substitutions). Sixteen sessions stopped by a subscription limit before any model call were replaced; counting them as failures gives 37/45 and 8/45. A Qwen3-Coder worker passed 0/45. In both studies with a direct arm, direct coding had lower API list-price-equivalent frontier cost per verified success. A post-hoc diagnostic found that 19 of 36 artifacts rejected only for renamed parameters passed the hidden checks unchanged; they are not counted as successes. The contribution is an evidence-qualified account of serving, agent protocol, session continuity, artifact validation, and verification in practice. The studies are not pooled and do not rank models or runtimes.

View arXiv page View PDF Add to collection

Community

jainofficials

Paper author Paper submitter 1 day ago

Most coding benchmarks measure whether a model can solve a problem. But what happens when we ask smaller open-weight models to complete real software engineering tasks?

In Beyond Benchmarks, I investigate this gap through practical coding workflows, looking beyond benchmark scores to understand task completion, reliability, human intervention, and the economics of using open-weight models.

The goal is not to declare a winning model, but to understand where smaller models are genuinely useful, where they struggle, and what it takes to make them effective in real development environments.

I hope these findings help researchers and engineers building more practical, affordable, and accessible AI coding systems.

Feedback, criticism, and independent replication are very welcome!

Paper: https://arxiv.org/abs/2610.34241

guicybercode

about 23 hours ago

The gap between cold-prefix and high-reuse requests, about 69.4 s against 2.8 s median time to first token, is what I would expect when a coding client rewrites the start of the prompt between turns. Did you find which clients broke prefix reuse most often, and was it the system prompt, tool definitions, or reordered history? The finding that 19 of 36 artifacts rejected only for renamed parameters passed the hidden checks also seems worth its own section: how much of the Qwen3-Coder 0/45 was protocol and contract mismatch rather than wrong code?

jainofficials

Paper author about 14 hours ago

Thanks, good question. Most of these sessions used Claude Code, pointed at local models through my own gateway, so this is one client plus my translation layer rather than a comparison across clients. I didn't measure cold prefixes per client or per cause, so I'd rather not give a ranking the paper can't support. Qualitatively, the main culprits were changes near the start of the prompt, often from my own gateway rather than the client: per-turn state or metadata ending up in the system prefix, which is why §7.3 makes keeping it frozen a rule. Auxiliary requests such as title generation also took the single cache slot away from the main conversation (§7.4). I didn't see reordered history; it mostly grew by appending or was compacted. One caution on 69.4 s versus 2.8 s: those are selected, differently composed groups, and many cold requests are simply a session's first turn, so the gap isn't a direct measure of prompt rewriting.

On Qwen's 0/45: the 45 sessions break down as follows.

26 were rejected by the parameter-signature check. 23 of those were rejected only because a leading underscore was dropped; with that one rule bypassed, 12 passed the hidden checks and 11 still failed. The other 3 changed the signature in other ways and weren't tested.
13 returned invalid JSON, so their code was never tested.
5 were rejected by the parent after inspection.
1 was cancelled.

So at most 12 of the 45 were functionally correct code blocked purely by the contract. Even those never went through parent review, so they aren't counterfactual successes. That's why the next experiment I'd run is a bounded repair turn that sends the exact validation error back to the worker. The diagnostic has its own subsection, §9.4.

MLSystemsRI

about 18 hours ago

Splitting progress into served, coherent text, executed tool calls, and code that passes hidden checks is a useful ladder. Most demos stop on the second rung. Keeping the behavioral checks hidden from the model is the part more teams should adopt.

jainofficials

Paper author about 14 hours ago

Thanks, that's the framing I found most useful too. In the delegation studies there was one more rung: the worker's code also had to pass the interface contract and be accepted and applied by the parent before its hidden checks counted. Section 9.4 shows why that matters, because a strict contract rejected some code that would have passed. On hidden checks, the main benefit was independence from the agent's own claims. A harness or agent can report PASS while the requested behavior is missing. For transparency, the tasks and checks were written with AI assistance and hash-recorded before the scored sessions, and each check was confirmed to fail on the original stub and pass on a reference solution.