Once Upon an Input: Reasoning via Per-Instance Program Synthesis
Large language models (LLMs) excel at zero-shot inference but continue to
struggle with complex, multi-step reasoning. Recent methods that augment LLMs
with intermediate reasoning steps such as Chain of Thought (CoT) and Program of
Thought (PoT) improve performance but often produce undesirable solutions,
especially in algorithmic domains. We introduce Per-Instance Program Synthesis
(PIPS), a method that generates and refines programs at the instance-level
using structural feedback without relying on task-specific guidance or explicit
test cases. To further improve performance, PIPS incorporates a confidence
metric that dynamically chooses between direct inference and program synthesis
on a per-instance basis. Experiments across three frontier LLMs and 30
benchmarks including all tasks of Big Bench Extra Hard (BBEH), visual question
answering tasks, relational reasoning tasks, and mathematical reasoning tasks
show that PIPS improves the absolute harmonic mean accuracy by up to 8.6% and
9.4% compared to PoT and CoT respectively, and reduces undesirable program
generations by 65.1% on the algorithmic tasks compared to PoT with
Gemini-2.0-Flash.
69004a3922d452aac6dd43ec
Adam Stein
claimed_verified
2025-10-29T12:49:43.085Z
64b6d0052662b49f164de3b3
/avatars/12e70c77c3bb0698cb64a878ff72de21.svg
Adam Stein
steinad
user
steinad
69004a3922d452aac6dd43ed
Neelay Velingker
69004a3922d452aac6dd43ee
Mayur Naik
69004a3922d452aac6dd43ef
Eric Wong
Once Upon an Input: Reasoning via Per-Instance Program Synthesis
Large language models (LLMs) excel at zero-shot inference but continue to
struggle with complex, multi-step reasoning. Recent methods that augment LLMs
with intermediate reasoning steps such as Chain of Thought (CoT) and Program of
Thought (PoT) improve performance but often produce undesirable solutions,
especially in algorithmic domains. We introduce Per-Instance Program Synthesis
(PIPS), a method that generates and refines programs at the instance-level
using structural feedback without relying on task-specific guidance or explicit
test cases. To further improve performance, PIPS incorporates a confidence
metric that dynamically chooses between direct inference and program synthesis
on a per-instance basis. Experiments across three frontier LLMs and 30
benchmarks including all tasks of Big Bench Extra Hard (BBEH), visual question
answering tasks, relational reasoning tasks, and mathematical reasoning tasks
show that PIPS improves the absolute harmonic mean accuracy by up to 8.6% and
9.4% compared to PoT and CoT respectively, and reduces undesirable program
generations by 65.1% on the algorithmic tasks compared to PoT with
Gemini-2.0-Flash.
Per-Instance Program Synthesis (PIPS) enhances LLM performance by generating and refining instance-level programs with structural feedback, improving accuracy and reducing undesirable solutions.
Large language models zero-shot inference Chain of Thought Program of Thought Per-Instance Program Synthesis structural feedback confidence metric Big Bench Extra Hard visual question answering relational reasoning mathematical reasoning harmonic mean accuracy