OpenCV image Stiching

Posted February 14, 2019 by Rokas Balsys



Image stiching first part

You already know that Google photos app has stunning automatic features like video making, panorama stitching, collage making, sorting out images based by the persons in the photo and many others. Have you ever wondered, how all these function work ? So I though, how hard can it be to make panorama stitching on my own by using Python language.

So what is image stitching ? In simple terms, for an input there should be a group of images, the output is a composite image such that it is a culmination of image scenes. At the same time, the logical flow between the images must be preserved.

For example, think about sea horizont while you are taking few photos of it. From a group of these images, we are essentially creating a single stitched image, that explains the full scene in detail. It is quite an interesting algorithm.

Let's first understand the concept of image stitching. Basically if you want to capture a big scene and your camera can only provide an image of a specific resolution and that resolution is 640 by 480, it is certainly not enough to capture the big panoramic view. So, what we can do is to capture multiple images of the entire scene and then put all bits and pieces together into one big image. Such photos of ordered scenes of collections are called panoramas. The entire process of acquiring multiple image and converting them into such panoramas is called as image stitching. And finally, we have one beautiful big and large photograph of the scenic view.

Firstly, let us install opencv version 3.4.2.16. If you have never version first do "pip uninstall opencv" bofore installing older version. If you will work with never version, you will be required to build opencv library by your self to enable image stitching function, so it's much easier to install older version:

pip install opencv-contrib-python==3.4.2.16

Next we are importing libraries that we will use in our code:

import cv2
import numpy as np

For our tutorial we are taking this beautiful photo, which we will slice into two left and right photos, and we'll try to get same or very similar photo back.


original_image.jpg

So I sliced this image into two images that they would have some kind of overlap region:


original_image_left.jpg


original_image_right.jpg

So here is the list of steps what we should do to get our final stiched result:

1. Compute the sift-key points and descriptors for left and right images.
2. Compute distances between every descriptor in one image and every descriptor in the other image.
3. Select the top best matches for each descriptor of an image.
4. Run RANSAC to estimate homography.
5. Warp to align for stitching.
6. Finally stitch them together.

So starting from the first step, we are importing these two images and converting them to grayscale, if you are using large images I recommend you to use cv2.resize because if you have older computer it may be very slow and take quite long. If you want to resize image size i.e. by 50% just change from fx=1 to fx=0.5.

img_ = cv2.imread('original_image_left.jpg')
img_ = cv2.resize(img_, (0,0), fx=1, fy=1)
img1 = cv2.cvtColor(img_,cv2.COLOR_BGR2GRAY)

img = cv2.imread('original_image_right.jpg')
img = cv2.resize(img, (0,0), fx=1, fy=1)
img2 = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

We still have to find out the features matching in both images. We shall be using opencv_contrib's SIFT descriptor. SIFT (Scale Invariant Feature Transform) is a very powerful OpenCV algorithm. You can read more OpenCV’s docs on SIFT for Image to understand more about features. These best matched features act as the basis for stitching. We extract the key points and sift descriptors for both the images as follows:

sift = cv2.xfeatures2d.SIFT_create()
# find the key points and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)

kp1 and kp2 are keypoints, des1 and des2 are the descriptors of the respective images. If we'll plot this image with features, this is how it will look:

cv2.imshow('original_image_left_keypoints',cv2.drawKeypoints(img_,kp1,None))

Image on left shows actual image. Image on the right is annotated with features detected by SIFT:


original_image_left.jpg


original_image_left_keypoints.jpg

Once you have got the descriptors and key points of two images, we will find correspondences between them. Why do we do this ? Well, in order to join any two images into a bigger images, we must find overlapping points. These overlapping points will give us an idea of the orientation of the second image according to first one. And based on these common points, we get an idea whether the second image is bigger or smaller or has it been rotated and then overlapped, or maybe scaled down/up and then fitted. All such information is yielded by establishing correspondences. This process is called registration.

For matching images can be used either FLANN or BFMatcher methods that are provided by opencv. I will write both examples prove that we'll get same result. Both examples matches the features which are more similar in both photos. When we set parameter k=2, this way we are asking the knnMatcher to give out 2 best matches for each descriptor. "matches" is a list of list, where each sub-list consists of "k" objects, to read more about this go here. And here is the code:

FLANN matcher code:

FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
match = cv2.FlannBasedMatcher(index_params, search_params)
matches = match.knnMatch(des1,des2,k=2)

BFMatcher matcher code:

match = cv2.BFMatcher()
matches = match.knnMatch(des1,des2,k=2)

Often in images there may be many chances that features may be existing in many places of the image. So we filter out through all the matches to obtain the best ones. So we apply ratio test using the top 2 matches obtained above. We consider a match if the ratio defined below is greater than the specified ratio.

good = []
for m,n in matches:
    if m.distance < 0.03*n.distance:
        good.append(m)

Now we are defining the parameters of drawing lines on image and giving the output to see how it looks like when we found all matches on image:

draw_params = dict(matchColor = (0,255,0), # draw matches in green color
                   singlePointColor = None,
                   flags = 2)

img3 = cv2.drawMatches(img_,kp1,img,kp2,good,None,**draw_params)
cv2.imshow("original_image_drawMatches.jpg", img3)

And here is the output image with matches drawn:


original_image_drawMatches.jpg

Here is the full code of this tutorial part:

import cv2
import numpy as np

img_ = cv2.imread('original_image_left.jpg')
#img_ = cv2.resize(img_, (0,0), fx=1, fy=1)
img1 = cv2.cvtColor(img_,cv2.COLOR_BGR2GRAY)

img = cv2.imread('original_image_right.jpg')
#img = cv2.resize(img, (0,0), fx=1, fy=1)
img2 = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

sift = cv2.xfeatures2d.SIFT_create()
# find the key points and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
#cv2.imshow('original_image_left_keypoints',cv2.drawKeypoints(img_,kp1,None))


#FLANN_INDEX_KDTREE = 0
#index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
#search_params = dict(checks = 50)
#match = cv2.FlannBasedMatcher(index_params, search_params)
match = cv2.BFMatcher()
matches = match.knnMatch(des1,des2,k=2)


good = []
for m,n in matches:
    if m.distance < 0.03*n.distance:
        good.append(m)


draw_params = dict(matchColor = (0,255,0), # draw matches in green color
                   singlePointColor = None,
                   flags = 2)

img3 = cv2.drawMatches(img_,kp1,img,kp2,good,None,**draw_params)
cv2.imshow("original_image_drawMatches.jpg", img3)


So now in this short tutorial we finished 1-3 steps we wrote above so 3 more steps left to do. So in the next tutorial we'll find homography for image transformation. Then we'll be able to proceed image stitching.