Tennis Ball Tracking with OpenCV

If you want to automate a task with even a moderate level of complexity, chances are your “robot” will need to be able to see. There are many ways to track an object but one of the simplest is tracking by color.

OpenCV

OpenCV is arguably the most powerful (and the most popular) computer vision library on the market. It can be used on Windows, Linux, Mac, and iOS/Android. Its got an interface for C++, Python, and Java.

The best part is: it’s free for both personal and commercial use. You can use it to identify objects, stitch together images, apply transformations and blurs, and a ton of other stuff that is way above my head.

Tennis Ball Tracking

What we’re going for is a very (and I do mean very) simple implementation of computer vision but it does work for a single object onscreen with no similar colors. If you want to track multiple objects at once or have similar colors in the background, this approach would have to be changed.

Setting up OpenCV

I’m going to use OpenCV 3.4.3 Visual Studio 2019 Community for this project. Let’s run through how to setup OpenCV.

Go to the OpenCV releases page and choose the version you need. If you’re on Windows you’ll be downloading an executable file. It’s actually not an installer, just a self-extracting archive.

I highly recommend extracting it to a folder such as C:\opencv and eliminating any spaces in your path since spaces can cause problems later.

Extracting OpenCV files.

We need to add OpenCV to our computer’s PATH environment variables. There are two equally valid ways of doing this: Using the windows system properties or the setx command. To do it with setx, open a command prompt and run:

setx /m OPENCV_DIR <where_openCV_extracted>\build\x64\vc15

Great, now we need to show Visual Studio where these files are located so it can compile and link them when we build our program. Open Visual Studio and right-click on your project (I’ll assume you know how to create a solution and project in Visual Studio. Click properties and then C/C++. You should see the screen below.

Adding OpenCV include directory.

Right-click on “Additional Include Directories” and add:

<where_openCV_extracted>\build\include>

Click on “Linker” and right-click on “Additional Library Directories”.

Adding OpenCV lib directory.
<where_openCV_extracted>\build\x64\vc15\lib>

At this point, Visual Studio should have all the files it needs for compiling and linking when we build our projects.

Finally, some computer vision

First, we have some boilerplate for importing libraries:

#include "openCV.h"
#include "track.h"

#define MINTRACKAREA 50

using namespace cv;

Now, let’s define a function that takes a stream from the webcam.

Mat Tracker::trackBall(VideoCapture cap){

	Mat frame;
		
	//Resize large images to reduce processing load
	cap >> frame;

Okay, now we are going to take the image frame and convert it from RGB to HSV. HSV is a little easier to handle when we begin thresh-holding the colors of the tennis ball later.

        //Convert RGB to HSV colormap
	//and apply Gaussain blur
	Mat hsvFrame;
	cvtColor(frame, hsvFrame, CV_RGB2HSV);

Applying a small 1 x 1 Gaussian blur will help reduce the noise in the image and improve the accuracy of our track.

        blur(hsvFrame, hsvFrame, cv::Size(1, 1));

Okay, we are finally ready to threshold the image. The inRange function assigned any pixel in its range to a 1 and any pixel outside it’s range to a 0. This should result in a black and white picture of a tennis ball.

        //Threshold 
	Scalar lowerBound = cv::Scalar(55, 100, 50);
	Scalar upperBound = cv::Scalar(90, 255, 255);
	Mat threshFrame;
	inRange(hsvFrame, lowerBound, upperBound, threshFrame);

We are well on our way. Now that we have a black and white image, we need to find the center of the ball. OpenCV includes a function known as moments that can automatically calculate the centroid of the binary image.

        //Calculate X,Y centroid
	Moments m = moments(threshFrame, false);
	Point com(m.m10 / m.m00, m.m01 / m.m00);

Finally, let’s just draw a marker over the centroid and show the image.

        //Draw crosshair
	Scalar color = cv::Scalar(0, 0, 255);
	drawMarker(frame, com, color, cv::MARKER_CROSS, 50, 5);

	imshow("Tennis Ball", frame);
	imshow("Thresholded Tennis Ball", threshFrame);

	return threshFrame;
	};

Awesome! This code will repeat every time the webcam sends a new frame. The end result is a ball with a crosshair drawn right over the center of it.

There are plenty of improvements to be made to this code in the future. It could use a mechanism to detect if the ball is too far or off-screen. Support for multiple colors would also be a great improvement for the future.

Leave a Reply