Standard neural network Convolution layers involve a lot of multiplications that make them unsuitable for deployment.
image
In this above scenario, we have an input image of 12x12x3 pixels and we apply a 5x5 convolution(no padding, stride = 1). We stack 256 such kernels so that we get an output of dimensions 8x8x256.
Here, there are 256 5x5x3 kernels that move 8x8 times which leads to 256x3x5x5x8x8 = 1,28,800 multiplications.
Depthwise separable convolution separates this process into two parts: a depthwise convolution and a pointwise convolution.
In depthwise convolution, we apply a kernel parallelly to each channel of the image.
image
We end up getting 3 different outputs (representing 3 channels of the image) to get an 8x8x1 image. These are stacked together to form a 8x8x3 image.
Pointwise Convolution now converts this 8x8x3 image input from the depthwise convolution back to an 8x8x1 output.
image
Stacking 256 1x1x3 kernels give us the final output as the standard convolution.
image
Total Number of multiplications:
For Depthwise convolution, we have 3 5x5x1 kernels moving 8x8 times, totalling 3x5x5x8x8=4800 multiplications.
In Pointwise convolution, we have 256 1x1x3 kernels moving 8x8 times, which is a total of 256x1x1x3x8x8=49152 multiplications.
Total number of multiplications = 4800 + 49152 = 53952 multiplications which is way lower than the standard convolution case.
Reference: https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728
