这篇论文是神经网络剪枝的远古工作,主要是提出了一种衡量权重参数重要性的指标,并且针对涉及的二阶导数计算提出了一个近似方法,以减少计算复杂性。 本文将权重的显著性定义为删除该参数引起的目标函数的变化,因此求解目标函数对权重参数的导数进行分析。 OBD算法 ...
GPTQ运用海塞矩阵找下一个量化权重,论文中没有相关证明。因此翻阅OBD和OBS论文,OBD是这篇Yann Le Cun 1989年的论文(彼时还在贝尔实验室)。总体感觉上下文说明比较多,比较容易读懂。 Optimal Brain Damage,可顾名思义,最优大脑损坏,大脑神经元如果要选择损坏一部分权重,应该去掉哪些?就是经典的神经元剪枝问题。剪枝...
OptimalBrainDamage YannLeCun,JohnS.DenkerandSaraA.Solla presentedby ChaitanyaPolumetla Overview • Introduction • NeedforOBD • TheIdea • AuthorsProposal • WhyOBDcouldwork? • Experiments • Results • Conclusion • Drawbacks
We have used Optimal Brain Damage interactively to reduce the number of parameters in a practical neural network by a factor of four. We obtained an additional factor of more than two by using OBD to delete parameters automatically. The network's speed improved signi cantly, and its ...
In order to reduce the number of model parameters (i.e. the number of weights), the Optimal Brain Damage (OBD) pruning algorithm is adopted for the recurrent neural models. Efficiency of the OBD algorithm is demonstrated for pruning neural models of a neutralisation reactor benchmark process. ...
Optimal Brain Damage (OBD) and Optimal Brain Surgeon (OBS) represent two popular pruning procedures; however, pruning large networks trained on voluminous data sets using these methods easily becomes intractable. We present a number of approximations and discuss practical issues in real-world pruning,...
A quantitative study of pruning by optimal brain damage The optimal brain damage (OBD) scheme of Le Cun, Denker and Solla for pruning of feedforward networks has been implemented and applied to the contiguity cl... J Gorodkin,LK Hansen,A Krogh,... - 《International Journal of Neural System...
OBD and SNIP remove weights closer to start. This is actually good as later weights are considered to be more important. Effect of extreme pruning percentages comparison a) Without retraining: SNIP was pruned at initialisation, rest are pruned after training. b) With retraining: Top-3 pruning ...
Various methods have been proposed to assess the value of a weight (e.g. saliency [Le Cun et al . 90] in the Optimal Brain-Damage -OBD- procedure): along similar ideas, we derive a method, called Optimal Cell Damage -OCD-, which evaluates the usefulness of input variables in a Multi...
Architecture optimization is\nperformed by approximative combinatorial search among the relevant\nsubsets of an initial neural network architecture by employing a\nvalidation set based optimal brain damage/surgeon (OBD/OBS) or a mean\nfield combinatorial optimization approach. Numerical results with linear...