LitePT: Lighter Yet Stronger Point Transformer
Modern neural architectures for 3D point cloud processing contain both convolutional layers and attention blocks, but the best way to assemble them remains unclear. We analyse the role of different computational blocks in 3D point cloud networks and find an intuitive behaviour: convolution is adequate to extract low-level geometry at high-resolution in early layers, where attention is expensive without bringing any benefits; attention captures high-level semantics and context in low-resolution, deep layers more efficiently. Guided by this design principle, we propose a new, improved 3D point cloud backbone that employs convolutions in early stages and switches to attention for deeper layers. To avoid the loss of spatial layout information when discarding redundant convolution layers, we introduce a novel, training-free 3D positional encoding, PointROPE. The resulting LitePT model has 3.6times fewer parameters, runs 2times faster, and uses 2times less memory than the state-of-the-art Point Transformer V3, but nonetheless matches or even outperforms it on a range of tasks and datasets. Code and models are available at: https://github.com/prs-eth/LitePT.
6940de9365f1e24a11780640
Yuanwen Yue
6940de9365f1e24a11780641
Damien Robert
6940de9365f1e24a11780642
Jianyuan Wang
6940de9365f1e24a11780643
Sunghwan Hong
6940de9365f1e24a11780644
Jan Dirk Wegner
6940de9365f1e24a11780645
Christian Rupprecht
6940de9365f1e24a11780646
Konrad Schindler
LitePT: Lighter Yet Stronger Point Transformer
Modern neural architectures for 3D point cloud processing contain both convolutional layers and attention blocks, but the best way to assemble them remains unclear. We analyse the role of different computational blocks in 3D point cloud networks and find an intuitive behaviour: convolution is adequate to extract low-level geometry at high-resolution in early layers, where attention is expensive without bringing any benefits; attention captures high-level semantics and context in low-resolution, deep layers more efficiently. Guided by this design principle, we propose a new, improved 3D point cloud backbone that employs convolutions in early stages and switches to attention for deeper layers. To avoid the loss of spatial layout information when discarding redundant convolution layers, we introduce a novel, training-free 3D positional encoding, PointROPE. The resulting LitePT model has 3.6times fewer parameters, runs 2times faster, and uses 2times less memory than the state-of-the-art Point Transformer V3, but nonetheless matches or even outperforms it on a range of tasks and datasets. Code and models are available at: https://github.com/prs-eth/LitePT.
A new 3D point cloud backbone model, LitePT, uses convolutions for early stages and attention for deeper layers, incorporating PointROPE for positional encoding, achieving efficient performance with fewer resources.
convolutional layers attention blocks 3D point cloud networks low-level geometry high-resolution high-level semantics context spatial layout information LitePT PointROPE Point Transformer V3