The first step is to take pictures of the images I will be using. I aimed for around 40% to 70% overlap between images to ensure smooth alignment. While taking the pictures, I needed to maintain the same center of projection and capture the images in quick succession to minimize changes in lighting and movement.

I took pictures at the library and plot points using (Link). By using these points, I need to find homography matrix of these two points for warping the image.

To find homography matrix using these points, I first stack each x and y points into the same matrix and use below equation to find unknowns, which are a, b, c, d, e, f, g and h. \[ \begin{bmatrix} wx' \\ wy' \\ w \end{bmatrix} = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \] Using systems of least squares, we can come up with this to find unknowns; \[ \begin{bmatrix} x & y & 1 & 0 & 0 & 0 & -xx' & -yx' \\ 0 & 0 & 0 & x & y & 1 & -xy' & -yy' \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \\ e \\ f \\ g \\ h \end{bmatrix} = \begin{bmatrix} x' \\ y' \end{bmatrix} \]

For warping the images, I created a function called warpImage that takes im (image) and H (homography) as inputs. I first define the corners of the image and warp them using the homography. By doing this, I can find the warp bounding box. Next, I get the minimum and maximum x & y values to create a mask, which I use to determine the destination coordinates for inverse warping. By mapping the coordinates from the output image back to the input image, we can ensure that the output image gets appropriate values. Finally, I used the map_coordinates function to set the pixels accordingly.

Using homography and warping functions that I made, I can now perform 'rectification' of an image. I took photos to rectify and pick coordinates that I want it to be rectify and warp to proper coordinate to make it rectangle.

For this rectification, I used [0, 0], [0,100], [750, 0], [750, 100].

For this rectification, I used [0, 0], [0,200], [300, 0], [300, 200].

In the previous part, I manually chose correspondences of images and blended the warped image to make a panorama. For this part, I am going to find corners of the image using the Harris Interest Point Detector.

First, I call the provided function to get Harris Corners and the Harris matrix.

As observed in both images, there are too many Harris corners, so we want to suppress some of them with ANMS. This allows us to have more evenly spread points, which is effective when obtaining homography or morphing the image. In the paper 'Multi-Image Matching using Multi-Scale Oriented Patches,' this mathematical equation is used to get filtered points: \[ r_i = \min |x_i - x_j|, \text{ s.t. } f(x_i) < c_{\text{robust}} f(x_j), \, x_j \in I \] While this is a effective way to find useful points, but as it runs in O(N^2), I first filter Harris Corners by its intensity in Harris Matrix and keep 2000 points.

Now that we have effective interest points, we need to find feature descriptors from the images to better compare homography. For the size of each feature extraction image, I use 40x40, with a feature size of 8. I first iterate through the ANMS coordinates to check for a 40x40 image, then downsample and resize. After this, I normalize the feature to have a mean of 0 and a standard deviation of 1, and then flatten it.

Here are some features that I extracted:

With feature descriptor extractions, we can now perform feature matching. To do this, I passed in the features extracted from the two images and calculated the Euclidean distance using the dist2 function. After this, I found the two nearest descriptors to the one I am comparing and calculated the distance ratio to check if it exceeds the threshold. By adding only pairs with a ratio below the threshold, we eliminate ambiguous matches.

Here is the example of matching image:

As observed, with proper threshold, it correctly removes some ambiguous pairs.

While the feature matching useful, there are some pairs that are not correctly connected together. So we use what we called RANSAC to find 4 correct points that can perform best homography.

Since I need to randomly select 4 pairs, I set the iterations to 10,000. To estimate the best homography matrix using RANSAC, for each iteration, I randomly selected 4 points and computed the homography, then determined the inliers by checking which points are within a specific threshold distance. I kept track of the largest set of inliers found. After the iterations, I recomputed the homography matrix using these inliers to obtain the best possible homography.

Here is the image that warped using the homography I got from RANSAC :

Here are results using Automatic Matchings with RANSAC with Manual Matchings:

Although, results may look similar, but there are differences and Automatic correspondences matchings are better than the other. It was intersting to see how panorama works and thorougly enjoyed process of implememting automatic correspondences. Also, I implementing RANSAC and ANMS were very interesting. These techniques opened up new possibilities I hadn’t considered applying before!