ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

人工智能69
  • Institute:MAC Lab, Department of Artificial Intelligence, Xiamen University
  • Author:Bohong Chen, Mingbao Lin, Kekai Sheng, Mengdan Zhang, Peixian Chen, Ke Li, Liujuan Cao*, Rongrong Ji
  • GitHub:https://github.com/chenbong/ARM-Net

Introduction

SISR平台存在有以下三种特点:

1.内存和计算能力有限

2.不同硬件设备上的资源配置不同

3.同一设备上硬件资源可用性随时间而改变

而新开发的SISR模型无法部署在资源匮乏的平台,作者研究了SISR网络中的计算冗余(Fig.1)

观察结果:

(1)A higher PSNR for an "easy" patch, and vice versa。"Hard" patches often contain more edge information.(Low Edge Score对应的PSNR更高,困难图像块的边缘信息更多(High Edge Score))

(2)For "easy" patch, the bilinear interpolation leads to the best results (while FSRCNN not only requires more computation, but degrades the performance).In contrast, a wider FSRCNN benefits the "hard" patch.(轻量模型足以处理简单图块,更宽的模型适合处理困难图块/超出轻量模型处理能力。)

(3)A tradeoff between computation overhead and performance of the reconstructed image.(需要平衡性能和计算开销)

据此,作者提出了一种Any-time super-Resolution Method,根据输入图像块的复杂度和当前可用的硬件资源,动态选择不同宽度的SR模型子网。

Related Work

Interpolation-based SISR。基于插值的方案适用于缺少计算资源的平台,但在重建复杂纹理的图像存在细节缺失。

Region-irrelevant CNNs-based SISR。区域无关的的方案无法根据输入图像(块)的复杂程度调整模型体量,来平衡模型性能和计算消耗。

Region-aware CNNs-based SISR。masks和branches的引入带来更多的参数。

Resource-aware Supernet。对不同尺寸子网进行统一采样的训练(大尺寸子网需要更多的训练)。

Method

1.Train several weight-shared subnets

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

每次迭代(每个batch)只对特定子网进行优化。

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

大规模子网能够得到更多的训练。

2.Edge-to-PSNR lookup table

采用laplacian edge detection operator生成边缘得分。

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

第k个子区间::

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

3.Choose the subnet (in inference)

目标是达到Computation-Performance Tradeoff.

computation-performance tradeoff function:

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

Experiments

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

与主干相比,PSNR有提升,Params没变,FLOPS降了。

与ClassSR相比,PSNR相近,Params降了。

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

相较于主干,相同计算量更好的性能。

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

定性比较没有明显差异。

Ablation Study

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

Edge-to-PSNR Lookup Tables:带有Edge-to-PSNR查找表的子网选择优于手动设置阈值(图7(a)中▲)和三个单独子网(图7(a)中♦)

Edge Detection Operators:图7(a)中不同的边缘检测算子展示了相似的性能,证明了方法的泛化性。

Subnet Number:子网数量越多,PSNR越高。

ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method

评价:Slimmable的思想应用在超分任务上,并且基于图像块自身属性进行子网的切换,而不引入更多的参数或计算。方法讲的很清楚,思路比较直观容易理解。

Original: https://www.cnblogs.com/huang-hz/p/16598895.html
Author: hhzcarl
Title: ECCV2022_Slimmable:(ARM-Net)ARM Any-Time Super-Resolution Method