Realtime Tracking With a Pan-Tilt Camera

Created by Blake Foster, Rui Wang, and Erik Learned-Miller

Introduction

The human eye is amazingly adept at tracking moving objects. The process is so natural to humans that it happens without any conscious effort. While this remarkable ability depends in part on the human brain's immense processing power, the fast response of the extraocular muscles and the eyeball's light weight are also vital. Even a small point and shoot camera mounted on a servo is typically too heavy and slow to move with the agility of the human eye. How, then, can we give a computer the ability to track movement quickly and responsively?

Thanks to recent progress in camera miniaturization, small, easily manipulable cameras are now readily available. In this project, we use a first person view (FPV) camera intended for use on model airplanes. The camera is mounted on servo motors which can aim the camera with two degrees of freedom. The entire assembly weighs only 32 grams, only slightly more than a typical human eyeball. Coupled with a GPU-based tracking algorithm, the FPV camera allows the computer to robustly track a wide array of patterns and objects with excellent speed and stability.

The above video clip shows a short demonstration. We built a simple camera tracking system using the FPV camera. The video demonstrates how the tracking camera snaps to a person moving in front of it. We show both the view captured by the tracking camera (the smaller video), and the view from a different camera that shows the movement of the tracking camera (the larger video).

How to build it

Parts List (links to parts included)

• Pan-Tilt Camera
• Video Cable
• Portable 12V Power Supply
• Video Transmitter (optional)
• Video Receiver (optional)
• Antenna (optional)
• RF Link Transmitter (optional)
• RF Link Receiver (optional)
• USB Frame Grabber
  
• Arduino Diecimila
• Voltage Regulator
• Heat Sink (optional)
• Capacitors (0.1 μF)

Note: the software (downloadable below) requires a PC with CUDA-capable graphics hardware (GPU).

The Camera
We used a 420-line pan-tilt camera manufactured by Fat Shark. The camera is mounted on two servo motors, which allow for about 170° of rotation on the yaw axis and 90° of rotation on the pitch axis. The camera produces composite video in PAL format. An NTSC version of the camera is available as well, but it was out of stock when we ordered our parts.

Power
Because the video transmitter requires the 12 volts, we power the camera with a 12V rechargeable Lithium battery. We use a voltage regulator to provide 5 volts for the camera and the servos. We added capacitors before and after the regulator to eliminate any voltage fluctuations. Both the video and servo cables from the camera connect to headers on the voltage regulator circuit, which provide the regulated 5V power supply. The servo control signals and the video output are passed through to a second set of headers, which connect to the frame grabber and the Arduino, respectively. The video output header additionally provides a 12V power supply for the video transmitter.

A picture and the schematic of the voltage regulator circuit.

Digitizing the Video

We used a USB frame-grabber manufactured by StarTech to read the video into the host PC. The frame grabber supports both NTSC and PAL composite video, so the NTSC camera could be used without any hardware changes. We used a video cable sold by Digital Products Company to connect the frame grabber to the video output header on the voltage regulator circuit. The cable also has a power jack, which provides 12 volts to the video transmitter.

The camera is connected to the frame grabber (L), and the Arduino is connected to the servos (R).

The frame-grabber provides 640 x 480 interlaced video at 25 FPS. For efficiency, we downsample the video to half resolution for tracking. Our downsampling filter discards the even lines to eliminate errors due to combing artifacts. We display the video at full resolution, after eliminating combing artifacts with a standard deinterlacing filter.


Controlling the Servos

We used an Arduino Diecimila to generate the control signal for the servos. The Arduino receives the desired pulse widths for the servos over its serial port. Each pulse width is encoded as a 16-bit integer, with 1 bit reserved to select one of the two servos. We use the servo library included with the Arduino software to generate the PWM signals.

Wireless Operation

We can eliminate the wired connections to the host PC with a wireless transmitter and receiver for the video, and a wireless RF link for the servos.

The RF Link

The wireless RF link transmits the servo angles digitally, with a range of up to 500 feet. Both the transmitter and receiver connect to Arduinos running the Virtual Wire library. The transmitter Arduino (connected to the host PC) broadcasts each 2-byte angle, followed by a byte of all zeros to keep the transmitter and receiver in sync. The receiver Arduino updates the servo angles when an angle is transmitted correctly (that is, all 3 bytes are received). Because Virtual Wire is incompatible with the Arduino's servo library, we use the Software Servo library to control the servos.

Wireless Video

The wireless video transmitter broadcasts NTSC or PAL video with a range of up to 500 meters. The transmitter plugs directly into the RCA jack from the camera. The 12 volt line on the video output header powers the transmitter. The RCA jack on the receiver plugs directly into the frame grabber. The receiver can be powered by a generic 12 volt power adapter.

Software

The software for the project, including source code and build files, is available via the links below.   The tracking software is based upon an algorithm developed jointly by the UMass Computer Vision and UMass Computer Graphics Labs. It is written for an NVidia GPU using the CUDA specification. Running on the GPU is necessary to get the real time rates we show in the video. Our implementation has only been tested in Windows 7, but we do not foresee major difficulties in porting it to other versions of Windows. The software can also be built for Linux (Ubuntu), but we did not have a frame grabber that worked under Linux, so we have not built a complete system under Linux. Still, we have tested each of the components of our system under Linux and they all work properly. More details about the software and how to build it are given in the attached README files.

Download Software