How does DropPath's implementation relate to the paper?

[amaarora]

Hey @rwightman !

I saw that drop_path/stochastic depth has been added widely to networks but in the timm training examples always found that you've used drop_connect instead. Have you tried using drop_path too?

Also, regarding the implementation of drop_path, in timm this looks like:

def drop_path(x, drop_prob: float = 0., training: bool = False):
    if drop_prob == 0. or not training:
        return x
    keep_prob = 1 - drop_prob
    shape = (x.shape[0],) + (1,) * (x.ndim - 1)  # work with diff dim tensors, not just 2D ConvNets
    random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
    random_tensor.floor_()  # binarize
    output = x.div(keep_prob) * random_tensor
    return output


class DropPath(nn.Module):
    def __init__(self, drop_prob=None):
        super(DropPath, self).__init__()
        self.drop_prob = drop_prob

    def forward(self, x):
        return drop_path(x, self.drop_prob, self.training)

How does this relate to the paper please? In the paper, the Residual Block is either active or inactive which is not the case for the implementation above. It updates the tensor and doesn't quite return either zeros or the tensor itself?

As an example,

import torch.nn as nn
import torch 

def drop_path(x, drop_prob: float = 0., training: bool = False):
    if drop_prob == 0. or not training:
        return x
    keep_prob = 1 - drop_prob
    shape = (x.shape[0],) + (1,) * (x.ndim - 1)  # work with diff dim tensors, not just 2D ConvNets
    random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
    random_tensor.floor_()  # binarize
    output = x.div(keep_prob) * random_tensor
    return output


class DropPath(nn.Module):
    """Drop paths (Stochastic Depth) per sample  (when applied in main path of residual blocks).
    """
    def __init__(self, drop_prob=None):
        super(DropPath, self).__init__()
        self.drop_prob = drop_prob

    def forward(self, x):
        return drop_path(x, self.drop_prob, self.training)

path = DropPath(0.25)
x    = torch.ones(1, 3)
path(x)

>> tensor([[1.3333, 1.3333, 1.3333]])

My tensor just got updated to 1.333 which is a version of Stochastic Depth that I do not understand. Is this expected?

[rwightman]

@amaarora they are the same (as used here), drop_connect was the name used for stochastic depth in the original TF efficientnet code, and I adopted that name, then realized it conflicts with another, different use of 'drop connect'. Since 'stochastic depth' is a mouth/keyboard full and more of a concept name than a good layer name, I used drop_path as the layer to implement stochastic depth (by dropping paths in the residual). The arg for drop_connect -> drop_path

[rwightman]

I find it's a useful regularization technique and all of my current hparams typically have it in the range of 0.08-.3 (.1 most common).

To my knowledge, I was the first to try it in the ViT model transformer layer stack residual paths where it is now pretty common in many training hparams for DeiT anot other ViT siblings.
https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py#L185

[amaarora]

I saw for ResNet/other models, that drop_path get's added to the residual branch like so:

class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, inplanes, planes, stride=1, downsample=None, cardinality=1, base_width=64,
                 reduce_first=1, dilation=1, first_dilation=None, act_layer=nn.ReLU, norm_layer=nn.BatchNorm2d,
                 attn_layer=None, aa_layer=None, drop_block=None, drop_path=None):
        super(Bottleneck, self).__init__()

        width = int(math.floor(planes * (base_width / 64)) * cardinality)
        first_planes = width // reduce_first
        outplanes = planes * self.expansion
        first_dilation = first_dilation or dilation
        use_aa = aa_layer is not None and (stride == 2 or first_dilation != dilation)

        self.conv1 = nn.Conv2d(inplanes, first_planes, kernel_size=1, bias=False)
        self.bn1 = norm_layer(first_planes)
        self.act1 = act_layer(inplace=True)

        self.conv2 = nn.Conv2d(
            first_planes, width, kernel_size=3, stride=1 if use_aa else stride,
            padding=first_dilation, dilation=first_dilation, groups=cardinality, bias=False)
        self.bn2 = norm_layer(width)
        self.act2 = act_layer(inplace=True)
        self.aa = aa_layer(channels=width, stride=stride) if use_aa else None

        self.conv3 = nn.Conv2d(width, outplanes, kernel_size=1, bias=False)
        self.bn3 = norm_layer(outplanes)

        self.se = create_attn(attn_layer, outplanes)

        self.act3 = act_layer(inplace=True)
        self.downsample = downsample
        self.stride = stride
        self.dilation = dilation
        self.drop_block = drop_block
        self.drop_path = drop_path

    def zero_init_last_bn(self):
        nn.init.zeros_(self.bn3.weight)

    def forward(self, x):
        residual = x

        x = self.conv1(x)
        x = self.bn1(x)
        if self.drop_block is not None:
            x = self.drop_block(x)
        x = self.act1(x)

        x = self.conv2(x)
        x = self.bn2(x)
        if self.drop_block is not None:
            x = self.drop_block(x)
        x = self.act2(x)
        if self.aa is not None:
            x = self.aa(x)

        x = self.conv3(x)
        x = self.bn3(x)
        if self.drop_block is not None:
            x = self.drop_block(x)

        if self.se is not None:
            x = self.se(x)

        if self.drop_path is not None:
            x = self.drop_path(x)

        if self.downsample is not None:
            residual = self.downsample(residual)
        x += residual
        x = self.act3(x)

        return x

Especially this line of code

        if self.drop_path is not None:
            x = self.drop_path(x)

This based on the implementation would update the tensor and not just keep it or drop it?

[rwightman]

Oh, like dropout it rescales the signal by the keep prob. That's done regardless of the random selection for that batch. Some impl make that rescale optional. Dropout has a similar scale

[rwightman]

Re resnet, I think that's just confusion re the naming, the original PyTorch code called the shortcut residual and mine was based on that old code. In that net x sticks with the residual path, and then it adds the shortcut (called residual) to x (residual). Confusing. So drop path drops samples from x right before the merge.

They've fixed it (https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L122) but I haven't.

pytorch/vision#729

[amaarora]

Oh, like dropout it rescales the signal by the keep prob. That's done regardless of the random selection for that batch. Some impl make that rescale optional. Dropout has a similar scale

Thanks Ross - I never knew that inputs get scaled. Really appreciate the help. :)