Colorizing the Prokudin-Gorskii Photo Collection
(R - (40, 107), G - (23, 49)) emir.tif with automatic cropping, edge detection, SSD
Overview
This project is inspired by the work of the Russian chemist and photographer, Sergey Prokudin-Gorskii. The main goal of this project is to align three separate images representing the Blue, Green, and Red color channels to create a single, cohesive color photograph. To achieve the best alignment, I used the Sum of Squared Differences (SSD) which helps minimize the pixel by pixel differences between the channels. Additionally, techniques such as automatic cropping and edge detection are also implemented to further enhancing alignment accuracy. These combined methods ensure that the final image is both sharp and well-aligned.
Project Details
First, I had to decide which mathematical equations to use to compute the best alignment between the reference and target color channels. I considered three options: L2 Norm, Normalized Cross-Correlation (NCC), and Sum of Squared Differences (SSD). I decided to use SSD because it is computationally simple, which made the run time faster than the other options.
L2 Norm
\( \sqrt{ \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 } \)
Normalized Cross-Correlation (NCC)
\[
\sum \left( \frac{\text{ref} - \mu_{\text{ref}}}{\sigma_{\text{ref}}} \cdot \frac{\text{val} - \mu_{\text{val}}}{\sigma_{\text{val}}} \right)
\]
Sum of Squared differencess (SSD)
\( \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \)
Searching for Best Alignment Implementation
I first implemented a naive method where I searched within the range [-15, 15] in both x and y directions to find the best alignment displacement. This approach worked for small images, like those in JPG format, but it took too long for .TIF files, as they were about 10 times larger than the JPG images.
To optimize and effectively search for the best displacement, I used a pyramid technique where I blurred the image with a sigma of 1 and downsampled the image by a factor of 2. I repeated this blurring using gaussian filter and downsampling process until the width or height of the image was reduced to fewer than 32 pixels. I then used a small step size, like 5 for fine alignment and 3 for coarser adjustments, to find the best alignment.
Bells & Whistles
With SSD by itself, it does a decent job of matching color channels; however, there is definitely room for improvement.
Automatic Cropping
When matching one channel to another, borders may mislead the alignment. So, before running SSD on the pixels and applying pyramid algorithms, I applied gradients to each channel and analyzed the sums of the gradients to find significant areas of the image. I then cropped out areas with little to no significant content on each side. Also, I made sure that cropping does not crop more than 20% of each side to ensure mis detect the border and over crops the image.
With gradient
Cropped blue channel
(R - (-5, 108), G - (0, 56)) SSD With Automatic Cropping
Edge Detection
Another feature that I added is Edge Detection, similar to cropping, I used gradient magnitude to make edges or areas with significant change in pixel emhpasized and compare image using SSD for getting gradient magnitude, I used Sobel for x and y axis of image and took squre root of square of each x and y sobel
With gradient
With gradient
(R - (3, 186), G - (-6, 71)) SSD With Edge Detection
The main difference when using Edge Detection is that it significantly improves the alignment accuracy by focusing on areas and pixels that have high gradient magnitude or significant pixel changes. As observed, the gradient helps emphasize the edges in the pictures, which improves the SSD score.
Final Result with SSD, Edge Detection, and Automatic Cropping
For the final result, I applied the features in this order:
Automatic Cropping → Edge Detection → SSD
By doing this, I was able to decrease the likelihood of detecting borders as edges of images, making the SSD score much more accurate.
(R - (-5, 58), G - (-6, 25))
(R - (10, 124), G - (17, 60))
(R - (8, 111), G - (8, 52))
(R - (1, 3), G - (0, -3))
(R - (-26, 140), G - (-11, 33))
(R - (29, 85), G - (-3, 41))
(R - (3, 6), G - (2, 3))
(R - (37, 176), G - (30, 79))
(R - (12, 177), G - (7, 79))
Acknowledgment
I used the Unemployables Portfolio Template for this website.