Virtual Width Networks
We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.
691a99126bfd5965c0fd34b6
Seed
691a99126bfd5965c0fd34b7
Baisheng Li
691a99126bfd5965c0fd34b8
Banggu Wu
691a99126bfd5965c0fd34b9
Bole Ma
691a99126bfd5965c0fd34ba
Bowen Xiao
691a99126bfd5965c0fd34bb
Chaoyi Zhang
691a99126bfd5965c0fd34bc
Cheng Li
691a99126bfd5965c0fd34bd
Chengyi Wang
691a99126bfd5965c0fd34be
Chenyin Xu
691a99126bfd5965c0fd34bf
Chi Zhang
691a99126bfd5965c0fd34c0
Chong Hu
691a99126bfd5965c0fd34c1
Daoguang Zan
691a99126bfd5965c0fd34c2
Defa Zhu
claimed_verified
2025-11-18T14:09:12.184Z
667505f4361b960c79e35486
/avatars/d352639c520075220f6abaae23c39376.svg
Defa Zhu
mathfinder
user
mathfinder
691a99126bfd5965c0fd34c3
Dongyu Xu
691a99126bfd5965c0fd34c4
Du Li
691a99126bfd5965c0fd34c5
Faming Wu
691a99126bfd5965c0fd34c6
Fan Xia
691a99126bfd5965c0fd34c7
Ge Zhang
claimed_verified
2025-11-18T14:09:07.984Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
zhangysk
user
zhangysk
691a99126bfd5965c0fd34c8
Guang Shi
691a99126bfd5965c0fd34c9
Haobin Chen
691a99126bfd5965c0fd34ca
Hongyu Zhu
691a99126bfd5965c0fd34cb
Hongzhi Huang
691a99126bfd5965c0fd34cc
Huan Zhou
691a99126bfd5965c0fd34cd
Huanzhang Dou
claimed_verified
2025-11-19T08:40:59.073Z
64c8ad3a527d763655600f6f
/avatars/6a4ab87fe040d666313f59d93712cf73.svg
Huanzhang Dou
Huanzhang
user
Huanzhang
691a99126bfd5965c0fd34ce
Jianhui Duan
691a99126bfd5965c0fd34cf
Jianqiao Lu
691a99126bfd5965c0fd34d0
Jianyu Jiang
691a99126bfd5965c0fd34d1
Jiayi Xu
691a99126bfd5965c0fd34d2
Jiecao Chen
691a99126bfd5965c0fd34d3
Jin Chen
691a99126bfd5965c0fd34d4
Jin Ma
691a99126bfd5965c0fd34d5
Jing Su
691a99126bfd5965c0fd34d6
Jingji Chen
691a99126bfd5965c0fd34d7
Jun Wang
691a99126bfd5965c0fd34d8
Jun Yuan
691a99126bfd5965c0fd34d9
Juncai Liu
691a99126bfd5965c0fd34da
Jundong Zhou
691a99126bfd5965c0fd34db
Kai Hua
claimed_verified
2025-11-17T10:26:23.121Z
64e851825ddcace745ba15bd
/avatars/7b6612c411222974d9ea181784eef915.svg
Kai Hua
kkish
user
kkish
691a99126bfd5965c0fd34dc
Kai Shen
691a99126bfd5965c0fd34dd
Kai Xiang
691a99126bfd5965c0fd34de
Kaiyuan Chen
claimed_verified
2025-11-21T09:02:26.572Z
636b4d796e6981ebad73f398
/avatars/bcd405b98c12afaf1e32d85ad8ce7f23.svg
Kaiyuan Chen
Lucky2022
user
Lucky2022
691a99126bfd5965c0fd34df
Kang Liu
691a99126bfd5965c0fd34e0
Ke Shen
691a99126bfd5965c0fd34e1
Liang Xiang
691a99126bfd5965c0fd34e2
Lin Yan
691a99126bfd5965c0fd34e3
Lishu Luo
691a99126bfd5965c0fd34e4
Mengyao Zhang
691a99126bfd5965c0fd34e5
Ming Ding
691a99126bfd5965c0fd34e6
Mofan Zhang
691a99126bfd5965c0fd34e7
Nianning Liang
691a99126bfd5965c0fd34e8
Peng Li
691a99126bfd5965c0fd34e9
Penghao Huang
691a99126bfd5965c0fd34ea
Pengpeng Mu
691a99126bfd5965c0fd34eb
Qi Huang
691a99126bfd5965c0fd34ec
Qianli Ma
691a99126bfd5965c0fd34ed
Qiyang Min
claimed_verified
2025-11-21T09:02:23.635Z
645507b89d37c3fb33279fe3
/avatars/ec1122137c49204ab968182d1f726c35.svg
min
qiyang-attn
user
qiyang-attn
691a99126bfd5965c0fd34ee
Qiying Yu
691a99126bfd5965c0fd34ef
Renming Pang
691a99126bfd5965c0fd34f0
Ru Zhang
691a99126bfd5965c0fd34f1
Shen Yan
691a99126bfd5965c0fd34f2
Shen Yan
691a99126bfd5965c0fd34f3
Shixiong Zhao
691a99126bfd5965c0fd34f4
Shuaishuai Cao
691a99126bfd5965c0fd34f5
Shuang Wu
691a99126bfd5965c0fd34f6
Siyan Chen
691a99126bfd5965c0fd34f7
Siyu Li
691a99126bfd5965c0fd34f8
Siyuan Qiao
691a99126bfd5965c0fd34f9
Tao Sun
691a99126bfd5965c0fd34fa
Tian Xin
691a99126bfd5965c0fd34fb
Tiantian Fan
691a99126bfd5965c0fd34fc
Ting Huang
691a99126bfd5965c0fd34fd
Ting-Han Fan
691a99126bfd5965c0fd34fe
Wei Jia
691a99126bfd5965c0fd34ff
Wenqiang Zhang
691a99126bfd5965c0fd3500
Wenxuan Liu
691a99126bfd5965c0fd3501
Xiangzhong Wu
691a99126bfd5965c0fd3502
Xiaochen Zuo
691a99126bfd5965c0fd3503
Xiaoying Jia
691a99126bfd5965c0fd3504
Ximing Yang
691a99126bfd5965c0fd3505
Xin Liu
691a99126bfd5965c0fd3506
Xin Yu
691a99126bfd5965c0fd3507
Xingyan Bin
691a99126bfd5965c0fd3508
Xintong Hao
691a99126bfd5965c0fd3509
Xiongcai Luo
691a99126bfd5965c0fd350a
Xujing Li
691a99126bfd5965c0fd350b
Xun Zhou
691a99126bfd5965c0fd350c
Yanghua Peng
691a99126bfd5965c0fd350d
Yangrui Chen
691a99126bfd5965c0fd350e
Yi Lin
691a99126bfd5965c0fd350f
Yichong Leng
691a99126bfd5965c0fd3510
Yinghao Li
691a99126bfd5965c0fd3511
Yingshuan Song
691a99126bfd5965c0fd3512
Yiyuan Ma
691a99126bfd5965c0fd3513
Yong Shan
claimed_verified
2025-12-16T14:10:49.544Z
64b8cc7ebe76d2ff0703bfb3
/avatars/f1c7ff17fd923f1460d362333d9fbfe3.svg
yong
yo37
user
yo37
691a99126bfd5965c0fd3514
Yongan Xiang
691a99126bfd5965c0fd3515
Yonghui Wu
691a99126bfd5965c0fd3516
Yongtao Zhang
691a99126bfd5965c0fd3517
Yongzhen Yao
691a99126bfd5965c0fd3518
Yu Bao
691a99126bfd5965c0fd3519
Yuehang Yang
691a99126bfd5965c0fd351a
Yufeng Yuan
691a99126bfd5965c0fd351b
Yunshui Li
691a99126bfd5965c0fd351c
Yuqiao Xian
691a99126bfd5965c0fd351d
Yutao Zeng
claimed_verified
2025-11-18T14:09:10.196Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
Taoer
user
Taoer
691a99126bfd5965c0fd351e
Yuxuan Wang
691a99126bfd5965c0fd351f
Zehua Hong
691a99126bfd5965c0fd3520
Zehua Wang
691a99126bfd5965c0fd3521
Zengzhi Wang
691a99126bfd5965c0fd3522
Zeyu Yang
691a99126bfd5965c0fd3523
Zhengqiang Yin
claimed_verified
2025-12-08T08:36:08.245Z
638ae78dfb4c3732d06bb8bb
/avatars/be23af81394a08ed9d09052a2fea3b3d.svg
Zhengqiang Yin
Codle
user
Codle
691a99126bfd5965c0fd3524
Zhenyi Lu
claimed_verified
2026-02-12T13:59:11.669Z
63607ace9ddc44e710e13f0f
/avatars/b5f331549562aea4a5c8b681fd9da1ff.svg
zy
lu-vae
user
lu-vae
691a99126bfd5965c0fd3525
Zhexi Zhang
691a99126bfd5965c0fd3526
Zhi Chen
691a99126bfd5965c0fd3527
Zhi Zhang
691a99126bfd5965c0fd3528
Zhiqi Lin
691a99126bfd5965c0fd3529
Zihao Huang
claimed_verified
2025-11-17T10:26:25.138Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
FetchFortune
user
FetchFortune
691a99126bfd5965c0fd352a
Zilin Xu
691a99126bfd5965c0fd352b
Ziyun Wei
691a99126bfd5965c0fd352c
Zuo Wang
Virtual Width Networks
We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.
Virtual Width Networks (VWN) enhance model efficiency by expanding representational width without increasing computational cost, accelerating optimization and improving loss reduction.
Virtual Width Networks VWN representational width backbone width embedding space next-token prediction next-2-token prediction token-efficient loss reduction