概述

      深度神经网络在人工智能应用中,包括计算机视觉、语音识别、自然语言处理方面,取得了巨大成功,但这些深度神经网络需要巨大的计算开销和内存开销,阻碍了在资源有限环境下的使用,如移动或嵌入式设备,为解决此问题,我们做了大量关于深度神经网络压缩与加速的研究工作。

       主流的压缩与加速神经网络的方法可以分成5种:(1)参数剪枝(parameter pruning);(2)参数共享(parameter sharing);(3)低秩分解(low rank decomposition);(4)紧性卷积核的设计(designing compact convolutional filters);(5)知识蒸馏(knowledge distillation)。参数剪枝主要通过设计判断参数重要与否的准则,移除冗余的参数。参数共享主要探索模型参数的冗余性,利用Hash或量化等技术对权值紧性压缩。低秩分解利用矩阵或张量分解技术估计并分解深度模型中的原始卷积核。紧性卷积核的设计主要通过设计特殊的结构化卷积核或紧性卷积计算单元,减少模型的存储与计算复杂度。知识蒸馏主要利用大型网络的知识,并将其知识迁移到紧性蒸馏的模型中。

近期研究

      针对深度神经网络参数存在大量冗余性问题,从压缩与加速深度神经网络两种不同任务出发,对深度神经网络的低秩分解(low-rank decomposition)和参数剪枝(parameterpruning)通用性方法展开深入研究,特别对于卷积神经网络 (convolutional neural networks, CNNs)的压缩与加速。具体研究内容和创新点包括:

      (1)文章[1]提出了一种新的结构化剪枝方案结构稀疏正则化(SSR),通过向原始目标函数中加入了两种不同的结构稀疏正则化,从而充分协调全局输出结果和局部剪枝操作,实现了滤波器的自适应剪枝。此外,我们还提出了一种基于Lagrange乘子方案的替代更新方法,有效地解决了其优化问题。

      (2)文章[2]提出了一种基于生成对抗学习的最优结构化网络剪枝方法。针对全局动态 剪枝方法缺乏松弛性和强标签依赖问题,提出了一种无需标签的端对端训练的异构剪枝方法。

      (3)文章[3]从神经网络的解释性角度出发,分析卷积神经网络特征图的冗余性问题,发现特征图的重要性取决于它的稀疏性和信息丰富度。但直接计算特征图的稀疏性与信息丰富度,我们建立了特征图和其对应二维卷积核之间的联系,通过卷积核的稀疏性和密度熵来表征对应特征图的重要程度,并得到判定特征图重要性的得分函数。在此基础上,我们采用较为细粒度压缩的卷积核聚类代替传统的剪枝方式压缩模型。

      (4)文章[4]提出了一种基于低秩分解和知识迁移的全局卷积神经网络压缩方法,利用一种带有闭合解的低秩分解技术分别加速卷积计算和压缩内存开销。为了有 效提高压缩后模型的准确率及克服网络训练中的梯度消失问题,提出了新的知识迁移,用于对齐压缩网络与原始网络之间的隐层的输出及最终网络输出结果。

代表论文

  • [1] Shaohui Lin, Rongrong Ji*, Yuchao Li, Cheng Deng, Xuelong Li.
    Towards Compact ConvNets via Structure-sparsity Regularized Filter Pruning. [pdf] [code] [bibtex]
    IEEE Transactions on Neural Networks and Learning Systems (TNN),2019.
    
    		 
    		 @inproceedings{TNN: 19,
    		 
    			Author={Shaohui Lin, Rongrong Ji*, Yuchao Li, Cheng Deng, Xuelong Li},
    		 
    		 Title={Towards Compact ConvNets via Structure-sparsity Regularized Filter Pruning},
    		 
    		 Booktitle={IEEE Transactions on Neural Networks and Learning Systems},
    		 
    		 Year={2019}, Accept}
    		 
    												 
    		 
    												 
  • [2] Shaohui Lin, Rongrong Ji*, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, David Doermann.
    Towards Optimal Structured CNN Pruning via Generative Adversarial Learning. [pdf] [code] [bibtex]
    IEEE International Conference on Computer Vision and Pattern Recognition (CVPR),2019.
    
    				 
    				 @inproceedings{CVPR: 19,
    				 
    					Author={Shaohui Lin, Rongrong Ji*, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, David Doermann},
    				 
    				 Title={Towards Optimal Structured CNN Pruning via Generative Adversarial Learning},
    				 
    				 Booktitle={IEEE International Conference on Computer Vision and Pattern Recognition},
    				 
    				 Year={2019}, Accept}
    				 
    														 
    				 
    														 
  • [3] Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, Rongrong Ji*.
    Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression. [pdf] [code] [bibtex]
    IEEE International Conference on Computer Vision and Pattern Recognition (CVPR),2019.
    
    						 
    						 @inproceedings{CVPR: 19,
    						 
    							Author={Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, Rongrong Ji*},
    						 
    						 Title={Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression},
    						 
    						 Booktitle={IEEE International Conference on Computer Vision and Pattern Recognition},
    						 
    						 Year={2019}, Accept}
    						 
    																 
    						 
    																 
  • [4] Shaohui Lin, Rongrong Ji*, Chao Chen, Dacheng Tao, Jiebo Luo.
    Holistic CNN Compression via Low-rank Decomposition with Knowledge Transfer. [pdf] [code] [bibtex]
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),2018.
    
    							 
    							 @inproceedings{PAMI: 18,
    							 
    								Author={Shaohui Lin, Rongrong Ji*, Chao Chen, Dacheng Tao, Jiebo Luo},
    							 
    							 Title={Holistic CNN Compression via Low-rank Decomposition with Knowledge Transfer},
    							 
    							 Booktitle={ IEEE Transactions on Pattern Analysis and Machine Intelligence},
    							 
    							 Year={2018}, Accept}
    							 
    																	 
    							 
    																	 
  • [5] 纪荣嵘, 林绍辉, 晁飞, 吴永坚, 黄飞跃.
    深度神经网络压缩与加速综述. [pdf] [bibtex]
    计算机研究与发展 (JCRD),2018.
    
    								 
    								 @inproceedings{JCRD: 18,
    								 
    									Author={纪荣嵘, 林绍辉, 晁飞, 吴永坚, 黄飞跃},
    								 
    								 Title={深度神经网络压缩与加速综述},
    								 
    								 Booktitle={计算机研究与发展},
    								 
    								 Year={2018}, ,Vol={55},No={9},pp={1871-1888}}
    								 
    																		 
    								 
    																		 
  • [6] Shaohui Lin, Rongrong Ji*, Yuchao Li, Yongjian Wu, Feiyue Huang, Baochang Zhang.
    Accelerating Convolutional Networks via Global & Dynamic Filter Pruning. [pdf] [code] [bibtex]
    International Joint Conference on Artificial Intelligence (IJCAI),2018.
    
    										 
    										 @inproceedings{CVPR: 18,
    										 
    											Author={Shaohui Lin, Rongrong Ji*, Yuchao Li, Yongjian Wu, Feiyue Huang, Baochang Zhang},
    										 
    										 Title=Accelerating Convolutional Networks via Global & Dynamic Filter Pruning},
    										 
    										 Booktitle={International Joint Conference on Artificial Intelligence},
    										 
    										 Year={2018}, Accept}
    										 
    																				 
    										 
    																				 
  • [7] Shaohui Lin, Rongrong Ji*, Chao Chen and Feiyue Huang
    ESPACE: Accelerating Convolutional Neural Networks via Eliminating Spatial & Channel Redundancy. [pdf] [bibtex]
    Thirty-First AAAI Conference on Artificial Intelligence (AAAI),2017.
    
    											
    											@inproceedings{AAAI: 17,
    											
    											 Author={Shaohui Lin, Rongrong Ji*, Chao Chen and Feiyue Huang},
    											
    											Title={ESPACE: Accelerating Convolutional Neural Networks via Eliminating Spatial & Channel Redundancy},
    											
    											Booktitle={Thirty-First AAAI Conference on Artificial Intelligence},
    											
    											Year={2017}, Accept}
    											
    																					
    											
    																					
  • [8] Shaohui Lin, Rongrong Ji, Yongjian Wu, and Xuelong Li.
    Towards Convolutional Neural Networks Compressing via Global Error Reconstruction. [pdf] [bibtex]
    Twitten-fifth International Joint Conference on Artificial Intelligence (IJCAI),2016.
    
    															
    															@inproceedings{IJCAI: b,
    															
    															 Author={Shaohui Lin, Rongrong Ji, Yongjian Wu, and Xuelong Li},
    															
    															Title={Towards Convolutional Neural Networks Compressing via Global Error Reconstruction},
    															
    															Booktitle={Twitten-fifth International Joint Conference on Artificial Intelligence},
    															
    															Year={2016}, Accept}