SemCoT: Accelerating Chain-of-Thought Reasoning through
  Semantically-Aligned Implicit Tokens
The verbosity of Chain-of-Thought (CoT) reasoning hinders its mass deployment
in efficiency-critical applications. Recently, implicit CoT approaches have
emerged, which encode reasoning steps within LLM's hidden embeddings (termed
``implicit reasoning'') rather than explicit tokens. This approach accelerates
CoT by reducing the reasoning length and bypassing some LLM components.
However, existing implicit CoT methods face two significant challenges: (1)
they fail to preserve the semantic alignment between the implicit reasoning
(when transformed to natural language) and the ground-truth reasoning,
resulting in a significant CoT performance degradation, and (2) they focus on
reducing the length of the implicit reasoning; however, they neglect the
considerable time cost for an LLM to generate one individual implicit reasoning
token. To tackle these challenges, we propose a novel semantically-aligned
implicit CoT framework termed SemCoT. In particular, for the first challenge,
we design a contrastively trained sentence transformer that evaluates semantic
alignment between implicit and explicit reasoning, which is used to enforce
semantic preservation during implicit reasoning optimization. To address the
second challenge, we introduce an efficient implicit reasoning generator by
finetuning a lightweight language model using knowledge distillation. This
generator is guided by our sentence transformer to distill ground-truth
reasoning into semantically aligned implicit reasoning, while also optimizing
for accuracy. SemCoT is the first approach that enhances CoT efficiency by
jointly optimizing token-level generation speed and preserving semantic
alignment with ground-truth reasoning. Extensive experiments demonstrate the
superior performance of SemCoT compared to state-of-the-art methods in both
efficiency and effectiveness. Our code can be found at
https://github.com/YinhanHe123/SemCoT/.
6908c7b7812eca10f9cc6082
Yinhan He
claimed_verified
2026-02-27T16:47:11.912Z
654f058c35d81e5153c4dce0
/avatars/87480482d8191cd63cac39e7a6a00285.svg
Jonathan Yinhan He
jonathanhe123
user
jonathanhe123
6908c7b7812eca10f9cc6083
Wendy Zheng
claimed_verified
2025-11-04T20:26:59.330Z
68d6bd7beae29772aee704dc
/avatars/1bf8c05ac6948d12312a315c484e916c.svg
Wendy Zheng
wendyz123
user
wendyz123
6908c7b7812eca10f9cc6084
Yaochen Zhu
claimed_verified
2025-11-03T20:51:08.065Z
647a248bed75e95d3e98e3d6
/avatars/e10e2f8516d1451fd85e17b5a0ba978d.svg
Yaochen Zhu
yaochenzhu
user
yaochenzhu
6908c7b7812eca10f9cc6085
Zaiyi Zheng
6908c7b7812eca10f9cc6086
Lin Su
6908c7b7812eca10f9cc6087
Sriram Vasudevan
claimed_verified
2025-11-05T16:25:11.243Z
690a46fdad60413f0efbc84e
/avatars/042d1171374bfd889c68a567872743b0.svg
Sriram Vasudevan
sriramvasudevan
user
sriramvasudevan
6908c7b7812eca10f9cc6088
Qi Guo
6908c7b7812eca10f9cc6089
Liangjie Hong
6908c7b7812eca10f9cc608a
Jundong Li
SemCoT: Accelerating Chain-of-Thought Reasoning through
  Semantically-Aligned Implicit Tokens
The verbosity of Chain-of-Thought (CoT) reasoning hinders its mass deployment
in efficiency-critical applications. Recently, implicit CoT approaches have
emerged, which encode reasoning steps within LLM's hidden embeddings (termed
``implicit reasoning'') rather than explicit tokens. This approach accelerates
CoT by reducing the reasoning length and bypassing some LLM components.
However, existing implicit CoT methods face two significant challenges: (1)
they fail to preserve the semantic alignment between the implicit reasoning
(when transformed to natural language) and the ground-truth reasoning,
resulting in a significant CoT performance degradation, and (2) they focus on
reducing the length of the implicit reasoning; however, they neglect the
considerable time cost for an LLM to generate one individual implicit reasoning
token. To tackle these challenges, we propose a novel semantically-aligned
implicit CoT framework termed SemCoT. In particular, for the first challenge,
we design a contrastively trained sentence transformer that evaluates semantic
alignment between implicit and explicit reasoning, which is used to enforce
semantic preservation during implicit reasoning optimization. To address the
second challenge, we introduce an efficient implicit reasoning generator by
finetuning a lightweight language model using knowledge distillation. This
generator is guided by our sentence transformer to distill ground-truth
reasoning into semantically aligned implicit reasoning, while also optimizing
for accuracy. SemCoT is the first approach that enhances CoT efficiency by
jointly optimizing token-level generation speed and preserving semantic
alignment with ground-truth reasoning. Extensive experiments demonstrate the
superior performance of SemCoT compared to state-of-the-art methods in both
efficiency and effectiveness. Our code can be found at
https://github.com/YinhanHe123/SemCoT/.
SemCoT enhances CoT efficiency by optimizing token-level generation speed and preserving semantic alignment with ground-truth reasoning using contrastive training and knowledge distillation.
Chain-of-Thought CoT reasoning implicit CoT LLM hidden embeddings implicit reasoning explicit tokens semantic alignment sentence transformer knowledge distillation SemCoT