padding='same' conversion to PyTorch padding=#

The formula is: k = (n – 1) / 2, where n is kernel size. Here’s a visualization:

self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Conv2d(64, 64, kernel_size=3, padding = 1
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2, padding = 32),
        )

A complete formula to calculate padding can be found in the documentation of PyTorch:

Source: https://pytorch.org/docs/master/generated/torch.nn.Conv2d.html?highlight=conv2d#torch.nn.Conv2d

This formula includes kernel size, stride, and dilation.
Based on this equation, you can iterate using different padding sizes (as guesses), until you can find an appropiate value to solve the equation.

Depending on the size of your images, you can create a solver using binary search to find the optimal padding value, or you can just try different values incrementing padding += 1 if the images are not too large.

In PyTorch you can directly use integer in padding.

In convolution padding = 1 for 3×3 kernel and stride=1 is ~ "same" in keras.

And In MaxPool you should set padding=0 (default), for 2×2 kernel, stride=2 is ~ "same" in keras.

You can use Formula:

Out = (W+2P-K)/S + 1

Let see some mathematical calculation:

For Convolution:

Case 1:

input is 30×30, kernel_size(K) is 3×3, stride=1, padding=1:

Out = (30+2*1-3)/1 + 1 = floor(29/1) + 1 = 30 i.e 30×30 (~ padding="same")

Case 2:

input is 30×30, kernel_size(K) is 3×3, stride=1, padding=0:

Out = (30+2*0-3)/1 + 1 = floor(27/1) + 1 = 28 i.e 28×28 (~ padding="valid")

For MaxPooling:

Case 1:

input is 30×30, kernel_size(K) is 2×2, stride=2, padding=0:

Out = (30+2*0-2)/2 + 1 = floor(28/2) + 1 = 15 i.e 15×15 (~ padding="same")

Case 2:

input is 30×30, kernel_size(K) is 2×2, stride=2, padding=1:

Out = (30+2*1-2)/2 + 1 = floor(30/2) + 1 = 16 i.e 16×16 (~ padding="valid")

Here is a program implemented in PyTorch same as above keras code:

model = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=(3, 3), padding=1, bias=False),
    nn.BatchNorm2d(64),
    nn.ReLU(inplace=True),
    nn.Dropout(0.3),

    nn.Conv2d(64, 64, kernel_size=(3, 3), padding=1, bias=False),
    nn.BatchNorm2d(64),
    nn.ReLU(inplace=True),
    nn.MaxPool2d(kernel_size=(2, 2), stride=2, padding=0)
)

X = torch.rand((1, 3, 30, 30))

print(model)
for layer in model:
    X = layer(X)
    print(X.shape)

Output:

Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU()
  (3): Dropout(p=0.3, inplace=False)
  (4): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (5): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (6): ReLU()
  (7): MaxPool2d(kernel_size=(2, 2), stride=2, padding=0, dilation=1, ceil_mode=False)
)
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 30, 30])
torch.Size([1, 64, 15, 15])

Please use the below conv2d when you want to use padding = 'same' as in keras with stride = 2 or more

import torch
import torch.nn.functional as F

class Conv2dSame(torch.nn.Conv2d):

    def calc_same_pad(self, i: int, k: int, s: int, d: int) -> int:
        return max((math.ceil(i / s) - 1) * s + (k - 1) * d + 1 - i, 0)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        ih, iw = x.size()[-2:]

        pad_h = self.calc_same_pad(i=ih, k=self.kernel_size[0], s=self.stride[0], d=self.dilation[0])
        pad_w = self.calc_same_pad(i=iw, k=self.kernel_size[1], s=self.stride[1], d=self.dilation[1])

        if pad_h > 0 or pad_w > 0:
            x = F.pad(
                x, [pad_w // 2, pad_w - pad_w // 2, pad_h // 2, pad_h - pad_h // 2]
            )
        return F.conv2d(
            x,
            self.weight,
            self.bias,
            self.stride,
            self.padding,
            self.dilation,
            self.groups,
        )

conv_layer_s2_same = Conv2dSame(in_channels=3, out_channels=64, kernel_size=(7, 7), stride=(2, 2), groups=1, bias=True)
out = conv_layer_s2_same(torch.zeros(1, 3, 224, 224)) 

credit : Captum