A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers. 
text
A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers. 
new_line
new_line
Key points:
text
Key points:
new_line
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
text
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
new_line
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
text
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
new_line
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.
text
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.
new_line
new_line
Congrats to the authors for their work!
text
Congrats to the authors for their work!
new_line
new_line
Paper: 
text
Paper: 
ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect (2403.03853)
https://huggingface.co/papers/2403.03853
2403.03853
paper
resource
https://huggingface.co/papers/2403.03853
new_line
new_line
A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers. 

Key points:
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2403.03853


👍
Dlbk emran86 dev7halo MexIvanov Pretam
❤️
emran86