Computer vision

Image classification

Cat? (0/1)

Object detection

https://commons.wikimedia.org/wiki/File:Detected-with-YOLO--Schreibtisch-mit-Objekten.jpg

Style transfer

How to handle larger images?

• 64x64x3 means 12,288 inputs
• 1000x1000x3 means 3,000,000 inputs
  • With 1000 neurons in first layer, $W^{[1]}$ has 3 billion entries

Edge detection

https://commons.wikimedia.org/wiki/File:%C3%84%C3%A4retuvastuse_n%C3%A4ide.png

Convolution operation

In TensorFlow: tf.nn.conf2d und tf.keras.layers.Conv2D

Vertical edge detection?

Flipped image

Horizontal edge detection

Alternative edge detection filters

Idea: instead of handpicking numbers for the filter, we learn them

Convolution operation $X * W$ can be applied for inputs of any size while having fixed number of parameters in $W$.

Our output is smaller, than the input

Padding the input

Padding $p=1$

Other padding strategy: replication

How much to pad

• "valid": no padding
  $n \times n \quad * \quad f \times f \quad \to \quad n-f+1 \times n-f+1$
• "same": pad so that output has the same size as the input
  $p = \frac{f-1}{2}$

https://commons.wikimedia.org/wiki/File:2D_Convolution_Animation.gif

Strided convolutions

stride $s = 1$

• Input: $n_H \times n_W$
• Filter: $f \times f$
• Padding: $p$
• Stride: $s$

Output: $(\lfloor \frac{n_H + 2p - f}{s} \rfloor + 1) \times (\lfloor \frac{n_W + 2p - f}{s} \rfloor + 1)$

Now we can use convolutions on grayscale images.

How about RGB images

E.g. go from $1024 \times 1024$ to $1024 \times 1024 \times 3$.

Find vertical edges in red channel

Find vertical edges in all channel

Applying multiple filters

Dimensions:

• Input: $n_H \times n_W \times n_c$
• Filters: $f \times f \times n_c$
• Number of filters: $n_c'$

Output: $(n_H - f + 1) \times (n_W -f + 1) \times n_c'$

We now almost have one convolutional neural network layer.

What else do we need?

• Add bias term $b$
• Add non-linearity

If one layer has $10$ filters of size $3 \times 3 \times 3$. How many parameters has the layer?

Notation for layer $l$:

• $f^{[l]}$: filter size
• $p^{[l]}$: padding
• $s^{[l]}$: stride
• $n_c^{[l]}$: number of filters

Dimensions for layer $l$:

• Input size: $n_H^{[l-1]} \times n_W^{[l-1]} \times n_c^{[l-1]}$
• Output size: $n_H^{[l]} \times n_W^{[l]} \times n_c^{[l]}$
  • $n_H^{[l]} = \lfloor \frac{n_H^{[l-1]} + 2 p^{[l]} - f^{[l]}}{s^{[l]}}\rfloor + 1$
  • $n_W^{[l]} = \lfloor \frac{n_W^{[l-1]} + 2 p^{[l]} - f^{[l]}}{s^{[l]}}\rfloor + 1$
• Each filter: $f^{[l]} \times f^{[l]} \times n_c^{[l-1]}$
• $n_c^{[l]}$ filters
• All weights: $f^{[l]} \times f^{[l]} \times n_c^{[l-1]} \times n_c^{[l]}$
• Bias: $n_c^{[l]}$ (or $1 \times 1 \times 1 \times n_c^{[l]})$

Dimensions for layer $l$:

• Activations $a^{[l]}$: $n_H^{[l]} \times n_W^{[l]} \times n_c^{[l]}$
• Batch of activations $A^{[l]}$: $m \times n_H^{[l]} \times n_W^{[l]} \times n_c^{[l]}$

Let's build a neural network

Finally (Layer 4):

• Reshape to 1960 element vector
• Fully connected layer with logistic or softmax output

Max pooling

• $f = 2$
• $s = 2$

If we have multiple channels, pooling is applied for each channel independently.

Pooling can reduce dimensions significantly

Average pooling

Adaptive pooling

Idea: Fix target dimensions and set stride and filter size accordingly

\[ s^{[l]} = \lfloor \frac{n^{[l-1]}}{n^{[l]}} \rfloor \\[3mm] f^{[l]} = n^{[l-1]} - (n^{[l]} - 1) s^{[l]} \]

With one adaptive pooling layer we ensure that images of arbitrary size can be used.

$1 \times 1$ filters

A $1 \times 1$ filter can be used to make an operation only on the channels.

Layer types for convolutional networks

• Convolution
• Pooling
• Fully connected

Example (close to LeNet-5)

Common patterns

• Decreasing height and width with depth
• Increasing channels with depth
• Number of activations sinking with depth
• Convolutions followed by poolings
• Fully connected layers at the end

Parameter sharing

Feature detector that is useful for one part of the image should be useful on another part.

Translation invariance

Convolutions produce similar output when the image is shifted.

Sparsity / Locality

Each output is only affected by few inputs

How to decide on neural network architecture?

Check other models for similar tasks

LeNet-5 (1998)

Classify single digit images

• Input: $32 \times 32 \times 3$
• 6 Convolutions ($f=5, s=1$, tanh) : $28 \times 28 \times 6$
• Avg. pooling ($f=2, s=2$) : $14 \times 14 \times 6$
• 16 Convolutions ($f=5, s=1$, tanh) : $10 \times 10 \times 16$
• Avg. pooling ($f=2, s=2$) : $5 \times 5 \times 16$
• Flatten : $400$
• 120 Fully conn. (tanh): $120$
• 84 Fully conn. (tanh): $84$
• 10 Fully conn. (rbf): $10$

About 60k paramters

AlexNet (2012)

Image recognition

• Input: $227 \times 227 \times 3$
• 96 Convolutions ($f=11, s=4$, ReLU) : $55 \times 55 \times 96$
• Max Pooling ($f=3, s=2$) : $27 \times 27 \times 96$
• 256 Convolutions ($f=5, p=2$, ReLU) : $27 \times 27 \times 256$
• Max Pooling ($f=3, s=2$) : $13 \times 13 \times 256$
• 384 Convolutions ($f=3, p=1$, ReLU) : $13 \times 13 \times 384$
• 384 Convolutions ($f=3, p=1$, ReLU) : $13 \times 13 \times 384$
• 256 Convolutions ($f=3, p=1$, ReLU) : $13 \times 13 \times 256$
• Max Pooling ($f=3, s=2$) : $6 \times 6 \times 256$
• Flatten : $9216$
• 4096 Fully conn. (ReLU) : $4096$
• 4096 Fully conn. (ReLU) : $4096$
• 1000 Fully conn. (SoftMax) : $1000$

About 60 mio. paramters

VGG - 16 (2015)

Image recognition

CONV: $f=3, p=1$; POOL: $f=2, s=2$

• Input: $224 \times 224 \times 3$
• 2x CONV 64 : $224 \times 224 \times 64$
• POOL : $112 \times 112 \times 64$
• 2x CONV 128 : $112 \times 112 \times 128$
• POOL : $56 \times 56 \times 128$
• 3x CONV 256 : $56 \times 56 \times 256$
• POOL : $28 \times 28 \times 256$
• 3x CONV 512 : $14 \times 14 \times 512$
• POOL : $14 \times 14 \times 512$
• 3x CONV 512 : $14 \times 14 \times 512$
• POOL : $7 \times 7 \times 512$
• 2 x FC $4096$
• Softmax $1000$

About 138 mio. paramters

With very deep architectures, vanishing / exploding gradients become an issue

Idea: Add shortcuts to network (residual networks)

Residual block: \[ a^{[l+2]} = g(z^{[l+2]} {\color{red} + a^{[l]}}) \]

For very deep networks, the training becomes fragile (vanishing / exploding gradients problem)

With ResNets we can keep increasing depth more than with "plain" networks

Intuition why this works:

\[ a^{[l+2]} = g(z^{[l+2]} + a^{[l]}) \\[3mm] = g(W^{[l+2]} a^{[l+1]} + b^{[l+2]} + a^{[l]}) \]

Setting $W^{[l+2]} = 0, b^{[l+2]} = 0$: \[ a^{[l+2]} = g(a^{[l]}) \] which for ReLU implies $a^{[l+2]} = a^{[l]}$

For any residual block it is easy to not make things worse.

When using skip connections the dimensions of $a^{[l]}$ and $a^{[l+2]}$ must match.

Need to use convolutions with "same" padding strategy to keep dimensions.

What is a good filter size?

Idea: Use multiple sizes in same layer.

Inception network

All units have "same" padding strategy and stride 1, so that output dimensions match

Number of multiplications for applying $32$ filters of size $5 \times 5$ with "same" padding to an input with dimensions $28 \times 28 \times 192$:

• A single output needs $5 \cdot 5 \cdot 192 = 4800$ multiplications
• We have $28 \cdot 28 \cdot 32 = 25088$ outputs
• That makes $4800 \cdot 25088 = 120422400$ multiplications.

Bottleneck layer

Multiplications:

• $1 \cdot 1 \cdot 192 \cdot 28 \cdot 28 \cdot 16 = 2408448$
• $5 \cdot 5 \cdot 16 \cdot 28 \cdot 28 \cdot 32 = 10035200$

Total: $12443648$

Inception module

Inception network

https://knowyourmeme.com/photos/531557-we-need-to-go-deeper

MobileNets (2017)

Goal: Low computation cost for inference

Depthwise Separable Convolution

Idea: Apply filter to each channel independently and use 1x1 filter afterwards.

\[ n_W \times n_H \times n_c \\[3mm] \xrightarrow{\text{depthw. conv.}} n_W' \times n_H' \times n_c \\[3mm] \xrightarrow{n_c' \text{ conv. } 1\times 1} n_W' \times n_H' \times n_c' \]

Computation cost

Reduced by factor \[ \frac{1}{n_c'} + \frac{1}{f^2} \]

Original MobileNet v1: 13 layers depthwise separable convolution + POOL + FC + SoftMax

MobileNet v2 (2019)

"Bottleneck Block"

EfficientNet (2020)

How to select model for a given computation budget?

Do not reinvent the wheel

Use open source implementations of public models

• TensorFlow Hub
• PyTorch pre-trained models
• GitHub

Many computer vision models need huge amounts of data and are expensive to train

Idea: Reuse existing model and weights for new task

Transfer Learning

Adapt model for own final classes

Use new softmax layer, keep rest constant

With transfer learning we turn the first $L-1$ layers into a function $f(x) = a^{[L-1]}$.

Then we train a linear classfier on $a^{[L-1]}$: \[ \hat y = g(W^{[L]} x + b^{[L]}) \]

If we have more data, we can retrain more layers

Mixed strategy:

• Replace soft-max layer, freeze rest and train
• Unfreeze some layers from the back and train with smaller learning rate

Using a pre-trained model as a start almost always beats training a new model from scratch.