Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths
Designing a unified neural network to efficiently and inherently process sequential data with arbitrary lengths is a central and challenging problem in sequence modeling. The design choices in Transformer, including quadratic complexity and weak length extrapolation, have limited their ability to scale to long sequences. In this work, we propose Gecko, a neural architecture that inherits the design of Mega and Megalodon (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability to capture long range dependencies, including timestep decay normalization, sliding chunk attention mechanism, and adaptive working memory. In a controlled pretraining comparison with Llama2 and Megalodon in the scale of 7 billion parameters and 2 trillion training tokens, Gecko achieves better efficiency and long-context scalability. Gecko reaches a training loss of 1.68, significantly outperforming Llama2-7B (1.75) and Megalodon-7B (1.70), and landing close to Llama2-13B (1.67). Notably, without relying on any context-extension techniques, Gecko exhibits inherent long-context processing and retrieval capabilities, stably handling sequences of up to 4 million tokens and retrieving information from contexts up to 4times longer than its attention window. Code: https://github.com/XuezheMax/gecko-llm
69662fa087c71000b5a9106f
Xuezhe Ma
69662fa087c71000b5a91070
Shicheng Wen
69662fa087c71000b5a91071
Linghao Jin
69662fa087c71000b5a91072
Bilge Acun
69662fa087c71000b5a91073
Ruihang Lai
69662fa087c71000b5a91074
Bohan Hou
69662fa087c71000b5a91075
Will Lin
69662fa087c71000b5a91076
Hao Zhang
69662fa087c71000b5a91077
Songlin Yang
69662fa087c71000b5a91078
Ryan Lee
69662fa087c71000b5a91079
Mengxi Wu
69662fa087c71000b5a9107a
Jonathan May
69662fa087c71000b5a9107b
Luke Zettlemoyer
69662fa087c71000b5a9107c
Carole-Jean Wu
Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths
Designing a unified neural network to efficiently and inherently process sequential data with arbitrary lengths is a central and challenging problem in sequence modeling. The design choices in Transformer, including quadratic complexity and weak length extrapolation, have limited their ability to scale to long sequences. In this work, we propose Gecko, a neural architecture that inherits the design of Mega and Megalodon (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability to capture long range dependencies, including timestep decay normalization, sliding chunk attention mechanism, and adaptive working memory. In a controlled pretraining comparison with Llama2 and Megalodon in the scale of 7 billion parameters and 2 trillion training tokens, Gecko achieves better efficiency and long-context scalability. Gecko reaches a training loss of 1.68, significantly outperforming Llama2-7B (1.75) and Megalodon-7B (1.70), and landing close to Llama2-13B (1.67). Notably, without relying on any context-extension techniques, Gecko exhibits inherent long-context processing and retrieval capabilities, stably handling sequences of up to 4 million tokens and retrieving information from contexts up to 4times longer than its attention window. Code: https://github.com/XuezheMax/gecko-llm
Gecko is a neural architecture that improves long-range dependency capture through exponential moving average with gated attention and additional components like timestep decay normalization and sliding chunk attention, achieving efficient processing of arbitrary-length sequential data with superior long-context scalability compared to existing models.
Transformer exponential moving average gated attention timestep decay normalization sliding chunk attention mechanism adaptive working memory long-range dependencies sequence modeling attention window context-extension techniques