Skip to content

0. Paper List

Gongfan Fang edited this page Jan 16, 2024 · 16 revisions

Welcome to the Torch-Pruning Paper List!

This list is designed for beginners who are interested in learning more about the technical details behind Torch-Pruning & DepGraph. DepGraph is actually not a pruning algorithm itself. Rather, it serves as a grouping algorithm used to analyze the dependency in networks. To enable pruning, DepGraph should be combined with existing pruning methods like Magnitude Pruning or Taylor Pruning. Most of the ideas in these papers have been implemented in Torch-Pruning.

For a more comprehensive paper list, please refer to MingSun-Tse/Efficient-Deep-Learning.

Paper List

Title & Authors Figure Implementation
in Torch-Pruning
Publish
DepGraph: Towards Any Structural Pruning
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang
National University of Singapore

"We propose a general and fully automatic method, Dependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning."
image DependencyGraph
GroupNormImportance
GroupTaylorImportance
GroupHessianImportance
Publish
Optimal Brain Damage
Yann LeCun, John Denker, Sara Solla
AT&T Bell Laboratories

"The basic idea is to use second-derivative information to make a tradeoff between network complexity and training set error."
image HessianImportance
Publish
Pruning Convolutional Neural Networks for Resource Efficient Inference
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz
NVIDIA

"We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with finetuning by backpropagation—a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters."
TaylorImportance
Iterative Pruning
Publish
Pruning Filters for Efficient ConvNets
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf
University of Maryland

"We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications."
MagnitudeImportance
Publish
Learning Efficient Convolutional Networks Through Network Slimming
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang
Tsinghua University

"In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way."
image BNScaleImportance
Publish
Neural Pruning via Growing Regularization
Huan Wang, Can Qin, Yulun Zhang, Yun Fu
Northeastern University

"Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring."
image GrowingRegPruner
Clone this wiki locally