Complementing Machine Learning Algorithms with Image Processing
A short intro to commonly used Image Processing Algorithms in Python
Pics, or it did not happen. Taking photos of everyday moments has become today's default. Last year, Keypoint Intelligence projected that humanity would generate 1,436,300,000,000 images. Mylio, an image organization solutions provider, even forecasted this number to hit 1.6 Trillion in 2022. Wow. That's a lot of photos!
As a data scientist, learning to process and extract information from these images is crucial, especially in computer vision tasks like object detection, image segmentation, and self-driving cars. Several image processing libraries in Python like scikit-image, OpenCV, and Pillow/ PIL allow us to do precisely that.
How does Image Processing fit in the overall machine learning pipeline?
We usually read and clean digital images using our preferred image processing library and extract useful features that can be used by machine learning algorithms.
In the sample pipeline above, we carved out each leaf from the source image. We applied image enhancements (i.e., white balancing, thresholding/ morphology, and histogram equalization). After that, we also measured each leaf's geometric features such as convex area, perimeter, major and minor axes lengths, and eccentricity, which form our features table. Finally, we used these features as inputs to a classification algorithm and produced an acceptable F-1 score. We can mirror the last two steps to the popular iris dataset, but we generated the measurements ourselves this time. Isn't that nice?
Digital Images as NumPy arrays in Python
We can represent digital images as a 3-D function F(x, y, z) where x, y, and z refer to spatial coordinates. In contrast, F refers to the intensity of the image (usually from 0 to 255) at that specific point. In Python (and in Linear Algebra), we can represent this function as a NumPy array with three dimensions (a tensor). The z-axis can be interpreted as image channels (e.g., RGB for Red, Green, and Blue).
There are several color models available for images; Mathworks provides a good discussion of the different color spaces here. Our image processing libraries allow us to convert from one color space to another with a single code line.
Common Image Processing Algorithms
Image Enhancements. In neural network applications, we flip, rotate, upsample, downsample, and apply shear to images to augment the existing dataset. Sometimes, we want to correct the contrast and white balance of an image to accentuate the desired object or region. In the example below, we enhance the dark image using a linear function, allowing us to see the things that were not visible in the original image.
Morphological Filtering. In morphological filtering, we use a structuring element or kernel that inhibits or enhances regions of interest in a pixel neighborhood. Some common morphological operations include erosion, dilation, opening, closing, skeletonizing, and removing small holes.
Blob Detection. In image processing, blobs are defined as bright on dark or dark on bright regions in an image. Detected blobs usually signal an object or parts of an object in an image that helps object recognition and/or objects tracking. The three most common algorithms for blob detection are Laplacian of Gaussian, Difference of Gaussian, and Determinant of Hessian. All of which are based on derivatives of the function with the position.
Feature Extraction. Feature extraction takes advantage of connected components in the image. In the leaf classification example above, we isolated each leaf. We computed for each leaf's area, perimeter, and eccentricity, among others. We use these measurements as inputs to our machine learning algorithm. Features can be quickly extracted using the 'regionprops' module in scikit-image.
Image Segmentation. In image segmentation, we want to isolate some of the images through several thresholding operations. An example of this is binarizing a grayscale image using a > or < operator to generate a boolean mask. In some cases, we can use other color spaces like HSV or RGB in generating a threshold that will isolate an object of interest. In the example below, we generated a mask to isolate all the grapes in the fruit basket.
Wrapping It Up
This article discussed how we could incorporate image processing in the machine learning pipeline and showcased the different image processing algorithms commonly used in the field. In the next set of articles, we will be diving deeper into each of the common algorithms I shared here and discussed the concepts behind them.
