GPT-Gen

The paper "meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting" from ICML 2017 by researchers at Peking University. This paper presents a technique to speed up machine learning model training by gradient sparsification. The key idea is sparsifying backpropagation gradients by retaining only the top-k

Pruning is a popular technique for reducing the size of deep neural networks without sacrificing accuracy. However, traditional pruning methods can be computationally expensive and lack to hardware support. In this paper, Towards Fully Sparse Training: Information Restoration with Spatial Similarity, the authors propose a new approach to structured pruning