Object-Detection

Object Detection for Dummies Part 2: CNN, DPM and Overfeat

Part 1 of the “Object Detection for Dummies” series covered three foundations: (1) the concept of an image gradient vector, and how the HOG algorithm aggregates information from all gradient vectors within a single image; (2) the operation of image segmentation algorithms for identifying regions that may contain objects; and (3) how the Selective Search algorithm improves segmentation results to produce stronger region proposals.

· 7 min read · Curated and presented by

Part 2 introduces several classic convolutional neural network architecture designs for image classification (AlexNet, VGG, ResNet), along with the DPM (Deformable Parts Model) and Overfeat models for object recognition.

Part 1 of the “Object Detection for Dummies” series covered: (1) the concept of the image gradient vector and how the HOG algorithm aggregates information across all gradient vectors within an image; (2) how image segmentation methods identify regions that may contain objects; (3) how the Selective Search algorithm improves segmentation outputs to produce better region proposals.

In Part 2, we will examine classic convolutional neural network architectures for image classification. These models form the foundation for subsequent advances in deep learning based object detection. If you would like to learn about R-CNN and related models, see Part 3.

Links to all posts in the series: [Part 1] [Part 2] [Part 3] [Part 4].

CNN for Image Classification

CNN, short for “Convolutional Neural Network”, is the standard approach for many computer vision problems in modern deep learning. It was, to a certain extent, inspired by the way the human visual cortex system operates.

Convolution Operation

This guide to convolution arithmetic is highly recommended, as it offers a clear, rigorous explanation supported by extensive visualizations and examples. Here, we focus on two-dimensional convolution because this post works with images.

Briefly, the convolution operation slides a predefined kernel (also called a “filter”) over the input feature map (a matrix of image pixels). At each position, it multiplies corresponding values and sums the results to produce an output value. Collectively, these values form an output matrix. In most cases, the kernel is much smaller than the input image.

An illustration of applying a kernel to an input feature map to produce the output. (Image source: River Trail documentation)

Figure 2 presents two concrete examples of convolving a 3x3 kernel over a 5x5 2D matrix of numeric values to produce a 3x3 matrix. By adjusting the padding size and stride length, we can control the dimensions of the output matrix.

Two examples of 2D convolution operation: (top) no padding and 1x1 strides; (bottom) 1x1 border zeros padding and 2x2 strides. (Image source: deeplearning.net)

AlexNet (Krizhevsky et al, 2012)

  • 5 convolution [+ optional max pooling] layers + 2 MLP layers + 1 LR layer
  • Use data augmentation techniques to expand the training dataset, such as image translations, horizontal reflections, and patch extractions.
The architecture of AlexNet. (Image source: link)

VGG (Simonyan and Zisserman, 2014)

  • At the time, the network was regarded as “very deep”; 19 layers
  • The architecture is highly simplified, using only 3x3 convolutional layers and 2x2 pooling layers. Stacking small filters approximates a larger filter while using fewer parameters.

ResNet (He et al., 2015)

  • The network is genuinely very deep, with 152 layers in a straightforward architecture.
  • Residual Block: An input from a given layer can be routed to a component two layers later. Residual blocks are crucial for keeping a deep network trainable and making it work in practice. Without residual blocks, the training loss of a plain network does not monotonically decrease as the number of layers grows, due to vanishing and exploding gradients.
An illustration of the residual block in ResNet. In a sense, the design of residual blocks can be viewed as being inspired by V4 receiving input directly from V1 in the human visual cortex system. (left image source: Wang et al., 2017)

Evaluation Metrics: mAP

A widely used evaluation metric in many object recognition and detection tasks is “mAP”, short for “mean average precision”. It ranges from 0 to 100, and higher values indicate better performance.

  • Aggregate detections across all test images to plot a precision-recall curve (PR curve) for each class; “average precision” (AP) is the area under the PR curve.
  • Because target objects belong to different classes, compute AP separately per class, then average across classes.
  • A detection counts as a true positive if its “intersection over union” (IoU) with a ground-truth box exceeds a threshold (typically 0.5; in that case, the metric is “[email protected]”)

Deformable Parts Model

The Deformable Parts Model (DPM) (Felzenszwalb et al., 2010) recognizes objects using a mixture graphical model (Markov random fields) composed of deformable parts. The model has three primary components:

  1. A coarse root filter defines a detection window that roughly covers the full object. A filter assigns weights to a region feature vector.
  2. Multiple part filters that cover smaller object parts. Part filters are learned at twice the resolution of the root filter.
  3. A spatial model that scores part-filter locations relative to the root.
The DPM model contains (a) a root filter, (b) multiple part filters at twice the resolution, and (c) a model for scoring the location and deformation of parts.

Detection quality is measured as the filter scores minus the deformation costs. The matching score $f$, in laymen’s terms, is:

$ f(\text{model}, x) = f(\beta_\text{root}, x) + \sum_{\beta_\text{part} \in \text{part filters}} \max_y [f(\beta_\text{part}, y) - \text{cost}(\beta_\text{part}, x, y)] $

in which,

  • $x$ is an image with a specified position and scale;
  • $y$ is a sub region of $x$.
  • $\beta_\text{root}$ is the root filter.
  • $\beta_\text{part}$ is one part filter.
  • cost() measures the penalty for a part deviating from its ideal location relative to the root.

The basic scoring model is the dot product between the filter $\beta$ and the region feature vector $\Phi(x)$: $f(\beta, x) = \beta \cdot \Phi(x)$. The feature set $\Phi(x)$ can be defined by HOG or other similar algorithms.

A root location with a high score identifies a region that is likely to contain an object, while high-scoring part locations validate a recognized object hypothesis. The paper used latent SVM to model the classifier.

The matching process by DPM. (Image source: Felzenszwalb et al., 2010)

The author later argued that DPM and CNN models are not fundamentally separate approaches to object recognition. Instead, a DPM can be expressed as a CNN by unrolling the DPM inference procedure and mapping each step to an equivalent CNN layer. (For details, see Girshick et al., 2015!)

Overfeat

Overfeat [paper][code] is an early model that integrates object detection, localization, and classification into a single convolutional neural network. The core idea is to (i) perform image classification at multiple locations and over regions at multiple image scales using a sliding-window approach, and (ii) predict bounding box locations using a regressor trained on top of the same convolution layers.

The Overfeat architecture closely resembles AlexNet. Training proceeds as follows:

The training stages of the Overfeat model. (Image source: link)
  1. Train a CNN (similar to AlexNet) for the image classification task.
  2. Next, replace the top classifier layers with a regression network, then train it to predict object bounding boxes at each spatial location and scale. The regressor is class-specific, with one regressor produced for each image class.
    • Input: Images with classification and bounding box.
    • Output: $(x_\text{left}, x_\text{right}, y_\text{top}, y_\text{bottom})$, 4 values in total, representing the coordinates of the bounding box edges.
    • Loss: The regressor is trained to minimize the $l2$ norm between the predicted bounding box and the ground truth for each training example.

At detection time,

  1. Run classification at each location using the pretrained CNN.
  2. Predict object bounding boxes for all classified regions produced by the classifier.
  3. Merge bounding boxes that have sufficient overlap from localization and sufficient confidence from the classifier that they correspond to the same object.

Cited as:

@article{weng2017detection2,
  title   = "Object Detection for Dummies Part 2: CNN, DPM and Overfeat",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2017",
  url     = "https://lilianweng.github.io/posts/2017-12-15-object-recognition-part-2/"
}

Reference

[1] Vincent Dumoulin and Francesco Visin. “A guide to convolution arithmetic for deep learning.” arXiv preprint arXiv:1603.07285 (2016).

[2] Haohan Wang, Bhiksha Raj, and Eric P. Xing. “On the Origin of Deep Learning.” arXiv preprint arXiv:1702.07800 (2017).

[3] Pedro F. Felzenszwalb, Ross B. Girshick, David McAllester, and Deva Ramanan. “Object detection with discriminatively trained part-based models.” IEEE transactions on pattern analysis and machine intelligence 32, no. 9 (2010): 1627-1645.

[4] Ross B. Girshick, Forrest Iandola, Trevor Darrell, and Jitendra Malik. “Deformable part models are convolutional neural networks.” In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 437-446. 2015.

[5] Sermanet, Pierre, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks” arXiv preprint arXiv:1312.6229 (2013).