DeepHyperX代码理解-HamidaEtAl

人工智能63

代码复现自论文《3-D Deep Learning Approach for Remote Sensing Image Classification》

先对部分基础知识做一些整理:

一、局部连接与参数共享(都减少了参数计算量)

局部连接:基于图像局部相关的原理,保留了图像局部结构,同时减少了网络的权值个数,加快了学习速率,同时也在一定程度上减少了过拟合的可能。

参数共享:下面用图例解释参数共享与不共享的区别。

如图是一个3*3大小的卷积核在进行特征提取,channel=1, 在每个位置进行特征提取的时候都是共享一个卷积核,假设有k个channel,则参数

总量为33k,注意不同channel的参数是不能共享的。

DeepHyperX代码理解-HamidaEtAl

假设现在不使用参数共享,则卷积核作用于矩阵上的每一个位置时其参数都是不一样的,则卷积核的参数数量就与像素矩阵的大小保持一致了,假
设有k个channel,则参数数量为weightheightk,这对于尺寸较大的图片来说明显是不可取的。

DeepHyperX代码理解-HamidaEtAl

二、多通道输入输出:

首先理解fliter与kernel区别:kernel是fliter的组成成分,一个fliter就会对应生成一个特征图。

下图为多通道输入,单通道输出。有一个fliter,一个fliter包括三个kernel,最后生成一个特征图(一个fliter就对应一个特征图)

DeepHyperX代码理解-HamidaEtAl

下图为直观展示,输入为883的RGB三通道图片,设置了4个fliter对应四个输出通道,也就是输出4个特征图。每个fliter有3个kernel。

DeepHyperX代码理解-HamidaEtAl

三、2D卷积代码示例:

1 import torch as t
 2 import torch.nn as nn
 3
 4 class A(nn.Module):
 5     def __init__(self):
 6         super(A, self).__init__()
 7         #三个2D卷积层结构,每个卷积层的输入和输出通道数均为2,卷积核和大小是(3, 3)
 8         #torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1,
 9         #padding=0, dilation=1, groups=1, bias=True,
10         #padding_mode='zeros', device=None, dtype=None)
11         self.conv1 = nn.Conv2d(2, 2, kernel_size=(3, 3))
12         self.conv2 = nn.Conv2d(2, 2, kernel_size=(3, 3))
13         self.conv3 = nn.Conv2d(2, 2, kernel_size=(3, 3))
14
15 a = A()
16 print(list(a.parameters()))

运行结果:

1 [Parameter containing:
 2 tensor([[[[ 0.0213, -0.2054,  0.0985],
 3           [ 0.1254,  0.1202, -0.1115],
 4           [-0.0280,  0.0256,  0.1975]],
 5
 6          [[-0.2118, -0.1683,  0.0056],
 7           [-0.0554,  0.2255, -0.1548],
 8           [ 0.1747,  0.0449,  0.1606]]],
 9
10
11         [[[-0.1328, -0.1700, -0.1048],
12           [ 0.0911, -0.2230,  0.0685],
13           [ 0.0886,  0.1765, -0.1879]],
14
15          [[-0.0581,  0.1266, -0.1030],
16           [-0.0170,  0.0387, -0.0641],
17           [-0.0127, -0.2099, -0.2213]]]], requires_grad=True), Parameter containing:
18 tensor([ 0.0877, -0.1969], requires_grad=True), Parameter containing:
19 tensor([[[[-0.0266,  0.1511, -0.0034],
20           [-0.1070, -0.1734, -0.1017],
21           [ 0.0053,  0.1358, -0.0542]],
22
23          [[-0.0436,  0.1587, -0.0375],
24           [ 0.0125, -0.0431, -0.0877],
25           [-0.0766,  0.0405, -0.1306]]],
26
27
28         [[[-0.1449, -0.0315, -0.0236],
29           [ 0.0118,  0.2230, -0.2137],
30           [-0.1108, -0.1178,  0.0027]],
31
32          [[ 0.2184,  0.1964, -0.0959],
33           [-0.0385, -0.0523,  0.2135],
34           [-0.0387,  0.1951, -0.1546]]]], requires_grad=True), Parameter containing:
35 tensor([-0.0942, -0.2029], requires_grad=True), Parameter containing:
36 tensor([[[[ 8.9598e-02, -3.3190e-02,  1.0606e-01],
37           [-2.2397e-02,  2.0944e-01, -6.8180e-02],
38           [-1.7312e-01,  2.2318e-01,  1.9368e-01]],
39
40          [[ 7.0529e-02, -2.0741e-01, -1.2648e-01],
41           [-1.7503e-01,  1.7972e-01, -1.0417e-01],
42           [-1.9124e-01, -4.2022e-02,  1.4635e-01]]],
43
44
45         [[[-8.0857e-02,  8.5098e-03,  7.0629e-02],
46           [ 1.6926e-01, -1.7654e-02, -9.3033e-02],
47           [-2.9836e-02, -2.2935e-01,  1.1450e-01]],
48
49          [[ 1.3848e-01, -5.7713e-02,  1.5293e-04],
50           [-6.3998e-03, -1.0745e-01,  2.8835e-03],
51           [-1.7894e-01,  2.2133e-01,  5.2435e-02]]]], requires_grad=True), Parameter containing:
52 tensor([-0.1026,  0.0207], requires_grad=True)]

上面代码是定义的一个2D卷积网络架构,有3个卷积层,每个卷积层的输入通道和输出通道(对应fliter为2)都为2,每个fliter中有两个33的kernel(对应两个输入通道)。代码中的两个12的向量为bias。

四、3D卷积的示意图和示例代码:

DeepHyperX代码理解-HamidaEtAl

1 class B(nn.Module):
 2     def __init__(self):
 3         super(B, self).__init__()
 4         self.conv1 = nn.Conv3d(
 5             1,  # 输入图像的channel数,C_in
 6             3,  # 卷积产生的channel数,C_out
 7             kernel_size=2,  # 卷积核的尺寸,这里实际是(2,2,2),第一维表示卷积核处理的帧数
 8             stride=(1,1,1),  # 卷积步长,(D,H,W)
 9             padding=(0,0,0),  # 输入的每一条边补充0的层数,(D,H,W)
10             bias=False)
11         self.conv2 = nn.Conv3d(1, 3, kernel_size=(2, 2, 2), stride=(1, 1, 1), padding=(0, 0, 0), bias=False)
12
13 b = B()
14 print(list(b.parameters()))

运行结果:

1 [Parameter containing:
 2 tensor([[[[[-0.0895, -0.1651],
 3            [-0.1319,  0.2510]],
 4
 5           [[-0.1616, -0.2614],
 6            [ 0.0383,  0.2656]]]],
 7
 8
 9
10         [[[[-0.0809, -0.1821],
11            [-0.0624, -0.1401]],
12
13           [[-0.3170,  0.1499],
14            [-0.3449,  0.2639]]]],
15
16
17
18         [[[[ 0.2779, -0.0731],
19            [ 0.0439, -0.0353]],
20
21           [[ 0.2853, -0.3177],
22            [ 0.0559, -0.3290]]]]], requires_grad=True), Parameter containing:
23 tensor([[[[[-0.2529, -0.0537],
24            [ 0.0955, -0.1513]],
25
26           [[ 0.0310,  0.2558],
27            [-0.1903, -0.1561]]]],
28
29
30
31         [[[[-0.2919, -0.3142],
32            [ 0.0190, -0.1089]],
33
34           [[-0.0586,  0.1541],
35            [-0.2402, -0.3339]]]],
36
37
38
39         [[[[-0.3260, -0.0500],
40            [ 0.1392,  0.1486]],
41
42           [[ 0.1664,  0.1514],
43            [ 0.0345, -0.1979]]]]], requires_grad=True)]

三维数据输入通道为1,输出通道为3,那么fliter数为3,每个fliter中有1个kernel(2,2,2),因为两个3D-Conv参数一样,所以每个3D-Conv有3个(2,2,2)的tensor。这里如果对应到高光谱数据集,输入通道也为1。因为整个高光谱数据集就是一个3维的volume,整个数据集图像由长光谱维组成。

论文复现:

原论文网络结构:

DeepHyperX代码理解-HamidaEtAl

复现网络:

输入维度:Image has dimensions 610x340 and 103 channels

网络结构及参数:

DeepHyperX代码理解-HamidaEtAl

复现代码:

1     elif name == "hamida":
2         patch_size = kwargs.setdefault("patch_size", 5)
3         center_pixel = True
4         model = HamidaEtAl(n_bands, n_classes, patch_size=patch_size)
5         lr = kwargs.setdefault("learning_rate", 0.01)
6         optimizer = optim.SGD(model.parameters(), lr=lr, weight_decay=0.0005)
7         kwargs.setdefault("batch_size", 100)
8         criterion = nn.CrossEntropyLoss(weight=kwargs["weights"])
1 class HamidaEtAl(nn.Module):
 2     """
 3     3-D Deep Learning Approach for Remote Sensing Image Classification
 4     Amina Ben Hamida, Alexandre Benoit, Patrick Lambert, Chokri Ben Amar
 5     IEEE TGRS, 2018
 6     https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8344565
 7     """
 8
 9     @staticmethod
10     def weight_init(m):#权重初始化
11         if isinstance(m, nn.Linear) or isinstance(m, nn.Conv3d):
12             init.kaiming_normal_(m.weight) #kaiming_normal_均匀初始化方法
13             init.zeros_(m.bias)
14
15     def __init__(self, input_channels, n_classes, patch_size=5, dilation=1):
16         super(HamidaEtAl, self).__init__()
17         # The first layer is a (3,3,3) kernel sized Conv characterized
18         # by a stride equal to 1 and number of neurons equal to 20
19         self.patch_size = patch_size#每次截取图像patch大小(5,5)
20         self.input_channels = input_channels#输入通道数(波段数)
21         dilation = (dilation, 1, 1)
22
23         if patch_size == 3:
24             self.conv1 = nn.Conv3d(
25                 1, 20, (3, 3, 3), stride=(1, 1, 1), dilation=dilation, padding=1
26             )
27         else:
28             self.conv1 = nn.Conv3d(
29                 1, 20, (3, 3, 3), stride=(1, 1, 1), dilation=dilation, padding=0
30             )
31         # Next pooling is applied using a layer identical to the previous one
32         # with the difference of a 1D kernel size (1,1,3) and a larger stride
33         # equal to 2 in order to reduce the spectral dimension
34         #为什么要用卷积层要替代池化层:
35         # 使用2×2的最大池化,与使用卷积(stride为2)来做down sample性能并没有明显差别,
36         # 而且使用卷积(stride为2)相比卷积(步进为1)+池化,还可以减少卷积运算量和一个池化层。何乐而不为呢。
37         self.pool1 = nn.Conv3d(
38             20, 20, (3, 1, 1), dilation=dilation, stride=(2, 1, 1), padding=(1, 0, 0)
39         )
40         # Then, a duplicate of the first and second layers is created with
41         # 35 hidden neurons per layer.

42         self.conv2 = nn.Conv3d(
43             20, 35, (3, 3, 3), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
44         )
45         self.pool2 = nn.Conv3d(
46             35, 35, (3, 1, 1), dilation=dilation, stride=(2, 1, 1), padding=(1, 0, 0)
47         )
48         # Finally, the 1D spatial dimension is progressively reduced
49         # thanks to the use of two Conv layers, 35 neurons each,
50         # with respective kernel sizes of (1,1,3) and (1,1,2) and strides
51         # respectively equal to (1,1,1) and (1,1,2)
52         self.conv3 = nn.Conv3d(
53             35, 35, (3, 1, 1), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
54         )
55         self.conv4 = nn.Conv3d(
56             35, 35, (2, 1, 1), dilation=dilation, stride=(2, 1, 1), padding=(1, 0, 0)
57         )
58
59         # self.dropout = nn.Dropout(p=0.5)
60
61         self.features_size = self._get_final_flattened_size()
62         # The architecture ends with a fully connected layer where the number
63         # of neurons is equal to the number of input classes.

64         self.fc = nn.Linear(self.features_size, n_classes)
65
66         self.apply(self.weight_init)
67
68     def _get_final_flattened_size(self):#计算fc前的输出tensor大小,用随机的X去测试得到,这样就不用自己去推到了
69         with torch.no_grad():
70             x = torch.zeros(
71                 (1, 1, self.input_channels, self.patch_size, self.patch_size)
72             )
73             x = self.pool1(self.conv1(x))
74             x = self.pool2(self.conv2(x))
75             x = self.conv3(x)
76             x = self.conv4(x)
77             _, t, c, w, h = x.size()
78         return t * c * w * h
79
80     def forward(self, x):
81         x = F.relu(self.conv1(x))
82         x = self.pool1(x)
83         x = F.relu(self.conv2(x))
84         x = self.pool2(x)
85         x = F.relu(self.conv3(x))
86         x = F.relu(self.conv4(x))
87         x = x.view(-1, self.features_size)#对tensor resize相当于做展平(flatten)操作
88         # x = self.dropout(x)
89         x = self.fc(x)
90         return x

关于参数dilation的解释:https://blog.csdn.net/qimo601/article/details/112624091

运行结果:

DeepHyperX代码理解-HamidaEtAl

通过阅读代码发现和原论文中的结构并不完全相同,试着去修改了一层网络参数去接近原网络结构:

self.conv3 = nn.Conv3d(
            35, 35, (3, 1, 1), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
        )

->>

self.conv3 = nn.Conv3d(
            35, 35, (3, 3, 3), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
        )

但发现最终效果不如上面结果,可能是对原论文提出的结构稍作改进了吧。

DeepHyperX代码理解-HamidaEtAl

DeepHyperX代码理解-HamidaEtAl

参考资料:

https://github.com/nshaud/DeepHyperX

https://blog.csdn.net/weixin_38481963/article/details/109906338

https://blog.csdn.net/abbcdc/article/details/123332063

https://wenku.baidu.com/view/fa899796f221dd36a32d7375a417866fb84ac087.html

Original: https://www.cnblogs.com/AllFever/p/16700359.html
Author: AllFever
Title: DeepHyperX代码理解-HamidaEtAl



相关阅读

Title: DEBERTA: DECODING-ENHANCED BERT WITH DIS- ENTANGLED ATTENTION glue榜首论文解读

一、概览

DeepHyperX代码理解-HamidaEtAl

; 二、详细内容

  1. abstract
    a. 两个机制来improve bert和 roberta
    ⅰ. disentangled attention mechanism
    ⅱ. enhanced mask decoder
    b. fine-tuning阶段
    ⅰ. virtual adversarial training -> 提升泛化
    c. 效果
    ⅰ. 对nlu和nlg下游任务,提升都比较大
    ⅱ. 用一半的训练数据,效果就比roberta-large好了
    ⅲ. 48层的deberta,在21年6月登顶superGLUE
  2. introduction
    a. Disentangled attention(分散注意力)
    ⅰ. content embedding+相对位置embedding
    ⅱ. 所以强调的是相对位置?
    b. Enhanced mask decoder
    ⅰ. 出发点:预测mask的时候,有时候绝对位置也非常重要,这里引入绝对位置来辅助预测mask的token
    ⅱ. DeBERTa在softmax层之前引入了绝对单词位置嵌入,其中模型基于单词内容和位置的聚合上下文嵌入来解码被屏蔽的单词
    c. 对抗训练来提升fine-tuning下游任务的泛化能力
  3. background
    a. transformer
    ⅰ. 标准的self-attention缺乏有效的机制去编码位置信息
    ⅱ. 有论文显示相对位置编码比绝对位置编码更有效
    ⅲ. mlm:mask 15% token来预测
  4. 10%不变,10%随机词,80% mask
    b. deberta
    ⅰ. input
  5. 每个位置有个{Hi, Pi|j}:代表content和相对位置信息
  6. cross attention score:
  7. 感觉就是把他们分开了,并且结合了content-to-position的信息
    ⅱ. enhanced mask decoder accounts for absolute word positions
  8. 又来考虑绝对位置信息?

  9. 出发点:只用相对位置,不用绝对位置信息也是不够的

  10. 如何编码绝对位置?
    a. bert是在最开始的时候,利用了绝对的位置编码信息
    b. deberta是在encoder了后,softmax之前,才用了绝对的位置信息
    c. 总结一下,transformer层的时候,用相对位置信息,要decode mask的时候了,才添加绝对位置信息作为补偿信息,所以叫enhanced mask decoder
  11. scalue-invariant fine-tuning
    a. 正则化方法去提高泛化
    b. perturbation:扰动?

c. 方法:在normalized的word embedding中添加扰动来实现
d. SiFT first normalizes the word embedding vectors into stochastic vectors, and then applies the perturbation to the normalized embedding vectors
e. SiFT首先将单词嵌入向量归一化为随机向量,然后将扰动应用于归一化的嵌入向量
5. 实验
a. NLI比roberta_base高1个点,squad比roberta_base高2-3个点

5.1 base模型对比:

DeepHyperX代码理解-HamidaEtAl

5.2 消歧实验:
● 位置信息的作用
DeepHyperX代码理解-HamidaEtAl

● 扰动的作用
DeepHyperX代码理解-HamidaEtAl

● 附录:

  1. roberta比较
    DeepHyperX代码理解-HamidaEtAl
  2. attention矩阵观察
    DeepHyperX代码理解-HamidaEtAl
  3. fine-tuning阶段variance也小一些
    DeepHyperX代码理解-HamidaEtAl

三、其他参考

【NLP笔记】GLUE榜首DeBerta解读
https://zhuanlan.zhihu.com/p/348522530

Original: https://blog.csdn.net/qq_16949707/article/details/123507390
Author: HxShine
Title: DEBERTA: DECODING-ENHANCED BERT WITH DIS- ENTANGLED ATTENTION glue榜首论文解读