0. Paper List
This list is designed for beginners who are interested in learning more about the technical details behind Torch-Pruning & DepGraph. DepGraph is actually not a pruning algorithm itself. Rather, it serves as a grouping algorithm used to analyze the dependency in networks. To enable pruning, DepGraph should be combined with existing pruning methods like Magnitude Pruning or Taylor Pruning.
| Title & Authors | Figure | Implementation in Torch-Pruning |
|---|---|---|
 Optimal Brain Damage Yann LeCun, John Denker, Sara Solla AT&T Bell Laboratories "The basic idea is to use second-derivative information to make a tradeoff between network complexity and training set error." |
 |
HessianImportance |
 Pruning Convolutional Neural Networks for Resource Efficient Inference Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz NVIDIA "We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with finetuning by backpropagation—a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters." |
 |
TaylorImportance Iterative Pruning |
|
Pruning Filters for Efficient ConvNets Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf University of Maryland, NEC Labs America "We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications." |
 |
MagnitudeImportance |
 DepGraph: Towards Any Structural Pruning Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang National University of Singapore, Zhejiang University, Huawei Technologies Ltd. "we propose a general and fully automatic method, Dependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning." |
 |
DependencyGraph GroupNormImportance GroupTaylorImportance GroupHessianImportance |