Friday, July 2, 2010

Realtime Tracking With a Pan-Tilt Camera





Introduction

The human eye is amazingly adept at tracking moving objects. The process is so natural to humans that it happens without any conscious effort. While this remarkable ability depends in part on the human brain's immense processing power, the fast response of the extraocular muscles and the eyeball's light weight are also vital. Even a small point and shoot camera mounted on a servo is typically too heavy and slow to move with the agility of the human eye. How, then, can we give a computer the ability to track movement quickly and responsively?

Thanks to recent progress in camera miniaturization, small, easily manipulable cameras are now readily available. In this project, we use a first person view (FPV) camera intended for use on model airplanes. The camera is mounted on servo motors which can aim the camera with two degrees of freedom. The entire assembly weighs only 32 grams, only slightly more than a typical human eyeball. Coupled with a GPU-based tracking algorithm, the FPV camera allows the computer to robustly track a wide array of patterns and objects with excellent speed and stability.

The above video clip shows a short demonstration. We built a simple camera tracking system using the FPV camera. The video demonstrates how the tracking camera snaps to a person moving in front of it. We show both the view captured by the tracking camera (the smaller video), and the view from a different camera that shows the movement of the tracking camera (the larger video).

How to build it

Parts List (links to parts included)

Note: the software (downloadable below) requires a PC with CUDA-capable graphics hardware (GPU).

The Camera
We used a 420-line pan-tilt camera manufactured by Fat Shark. The camera is mounted on two servo motors, which allow for about 170° of rotation on the yaw axis and 90° of rotation on the pitch axis. The camera produces composite video in PAL format. An NTSC version of the camera is available as well, but it was out of stock when we ordered our parts.

Power
Because the video transmitter requires the 12 volts, we power the camera with a 12V rechargeable Lithium battery. We use a voltage regulator to provide 5 volts for the camera and the servos. We added capacitors before and after the regulator to eliminate any voltage fluctuations. Both the video and servo cables from the camera connect to headers on the voltage regulator circuit, which provide the regulated 5V power supply. The servo control signals and the video output are passed through to a second set of headers, which connect to the frame grabber and the Arduino, respectively. The video output header additionally provides a 12V power supply for the video transmitter.


A picture and the schematic of the voltage regulator circuit.


Digitizing the Video

We used a USB frame-grabber manufactured by StarTech to read the video into the host PC. The frame grabber supports both NTSC and PAL composite video, so the NTSC camera could be used without any hardware changes. We used a video cable sold by Digital Products Company to connect the frame grabber to the video output header on the voltage regulator circuit. The cable also has a power jack, which provides 12 volts to the video transmitter.



The camera is connected to the frame grabber (L), and the Arduino is connected to the servos (R).

The frame-grabber provides 640 x 480 interlaced video at 25 FPS. For efficiency, we downsample the video to half resolution for tracking. Our downsampling filter discards the even lines to eliminate errors due to combing artifacts. We display the video at full resolution, after eliminating combing artifacts with a standard deinterlacing filter.


Controlling the Servos

We used an Arduino Diecimila to generate the control signal for the servos. The Arduino receives the desired pulse widths for the servos over its serial port. Each pulse width is encoded as a 16-bit integer, with 1 bit reserved to select one of the two servos. We use the servo library included with the Arduino software to generate the PWM signals.

Wireless Operation

We can eliminate the wired connections to the host PC with a wireless transmitter and receiver for the video, and a wireless RF link for the servos.




The RF Link

The wireless RF link transmits the servo angles digitally, with a range of up to 500 feet. Both the transmitter and receiver connect to Arduinos running the Virtual Wire library. The transmitter Arduino (connected to the host PC) broadcasts each 2-byte angle, followed by a byte of all zeros to keep the transmitter and receiver in sync. The receiver Arduino updates the servo angles when an angle is transmitted correctly (that is, all 3 bytes are received). Because Virtual Wire is incompatible with the Arduino's servo library, we use the Software Servo library to control the servos.

Wireless Video

The wireless video transmitter broadcasts NTSC or PAL video with a range of up to 500 meters. The transmitter plugs directly into the RCA jack from the camera. The 12 volt line on the video output header powers the transmitter. The RCA jack on the receiver plugs directly into the frame grabber. The receiver can be powered by a generic 12 volt power adapter.

Software

The software for the project, including source code and build files, is available via the links below.   The tracking software is based upon an algorithm developed jointly by the UMass Computer Vision and UMass Computer Graphics Labs. It is written for an NVidia GPU using the CUDA specification. Running on the GPU is necessary to get the real time rates we show in the video. Our implementation has only been tested in Windows 7, but we do not foresee major difficulties in porting it to other versions of Windows. The software can also be built for Linux (Ubuntu), but we did not have a frame grabber that worked under Linux, so we have not built a complete system under Linux. Still, we have tested each of the components of our system under Linux and they all work properly. More details about the software and how to build it are given in the attached README files.

Download Software

58 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. That footage has been stabilised in post. You can see the edge of the frame swimmong around all over the shop. Why didn't you mention that in the article?

    ReplyDelete
  3. @Dusan We used this camera because it is considerably lighter than any USB webcam we have seen, which allows it to move with much greater agility.

    @index.html The video shows the unaltered output of our tracking software. Our software always keeps the tracking target centered in the frame, so if the camera lags slightly, or if the target moves beyond the cameras range of motion, the video will shift. This is explained in the documentation for the software.

    ReplyDelete
  4. Can it lock and track a single target if more than one target is present and moving.

    ReplyDelete
  5. Yes, we have tested the software with multiple objects moving in front of the camera, and even with the camera itself moving.

    ReplyDelete
  6. Hi - We'd like to write about this in NVIDIA's CUDA newsletter this week. Ok with you? What are some of the potential real-world applications? - Calisa Cole (ccole@nvidia.com)

    ReplyDelete
  7. Instead of trying to move the camera, why not bounce off a mirror and move that instead. You can get precise fast movable mirrors from laser systems, cabable of scanning hundreds of times faster that a moving head arrangement like this, and readily available from the entertainment industry.

    ReplyDelete
  8. I purchased the EasyCAP USB frame grabber. I can't seem to get mine to work properly. My system is more than qualified with a GeForce GTX 295, but I have not been able to get the frame grabber to work in any program. I know that the camera works because i tested it on my TV. I've tried using RCA things that I know work, such as my xBox with the frame grabber card, and still no luck. I'm currently trying to run it on windows 7 64-bit but i also tried it on win xp 32 with the same result

    I was wondering if you could post how you installed and set up the USB frame grabber. The more details the better.

    Thanks,
    ~Alex

    ReplyDelete
  9. Chelfyn: Using a mirror would present some engineering issues that we do not have to consider with a moving camera, but it would certainly be an interesting idea for a future project.

    Alex: We used a driver made by another company which seems to work with our frame grabber. Here's a link:
    http://www.sabrent.com/drivers/USB-AVCPTSetup_For_Win7.zip

    ReplyDelete
  10. Hey..congrats..its very good project..
    i m going to make it..can you send me more details about coding,software using & connections..since i m just starting now with such project kind of work ..
    my email address is..

    kanadaditya@yahoo.co.in

    thank you

    ReplyDelete
  11. thx a lot for the fun !

    cameraTrack works well :
    - with an old black and white cmos sensor (PAL output)

    - on debian, but its a pain to get all necessary libs ...
    - with easycap60 usb digitizer on ion platform (zotac, atomN330)
    and
    - with an old bt848 video capture board on nvidia 9800 (athlon64x2@3000)

    tc

    ReplyDelete
  12. Great project,

    I am looking to create a similar device that can motion track with a Pan and Tilt camera but relay the information back to a main unit that is robotic and follows the same actions as the smaller wireless unit. Do you think I could speak to someone about helping me with this? i am on email at zak@swat.tv

    Any help would be appreciated

    Thanks

    Zak

    ReplyDelete
  13. Hi, It is a good project, I like this project I will try to set up by myself to see how it works.
    Thanks for bring an idea.

    ReplyDelete
  14. Hello Anytime I try clink on any of the links it brings me to the main Blogger page even though I'm already logged in.
    Am I missing something?

    Thanks

    F

    Links below bring me to this page --> https://www.google.com/accounts/ServiceLogin?service=blogger&continue=https%3A%2F%2Fwww.blogger.com%2Floginz%3Fd%3Dhttp%253A%252F%252Fwww.blogger.com%252Fpost-edit.g%253FblogID%253D6138463946195121546%2526postID%253D2501459672138242714%26a%3DADD_SERVICE_FLAG&passive=true&alinsu=0&aplinsu=0&alwf=true&hl=en&ltmpl=start&skipvpage=true&rm=false&showra=1&fpui=2&naui=8#s01



    Video Transmitter link
    http://www.blogger.com/post-edit.g?blogID=6138463946195121546&postID=2501459672138242714#transmitter

    Frame Grabber link
    http://www.blogger.com/post-edit.g?blogID=6138463946195121546&postID=2501459672138242714#frame_grabber

    Arduino link
    http://www.blogger.com/post-edit.g?blogID=6138463946195121546&postID=2501459672138242714#arduino

    Virtual Wire library link
    http://www.blogger.com/post-edit.g?blogID=6138463946195121546&postID=2501459672138242714

    Software Servo library link
    http://www.blogger.com/post-edit.g?blogID=6138463946195121546&postID=2501459672138242714

    ReplyDelete
  15. I'm working on a short film that would benefit heavily if this setup could be configured with a larger camera (http://www.dpreview.com/news/1002/10020806canoneos550d.asp). With a $1000 budget, how could this setup be modified so that the final result is a full 1080p image with tracking as accurate (if not moreso) than in your previous project? I've already spoken with a major film equipment distrubutor here in Los Angeles. If this can be done, they're interested in buying the design. Email me here: lreed7950@gmail.com

    ReplyDelete
  16. could please share the algorithm , so i can understand how it works ?

    ReplyDelete
  17. Guys.
    The links isn't working longer...
    Annyone has it?
    Please e-mail me .zip
    andre@oriontech.no

    -Andy

    ReplyDelete
  18. The download link is available now. Sorry that the link was broken for a while due to server issue.

    ReplyDelete
  19. My friends and I are working on this for our final project for ITT Tech. Would it be possible to provide us some more details including schematics and pictures (if possible) to help us build this awesome project. We just received all the parts this week and want to make sure we don't do anything wrong. Feel free to contact me directly jessexcruz@gmail.com

    Thanks in advance.

    Jesse

    ReplyDelete
  20. I will try to port to ArduPilotMega for camera control
    Thanks for a great head start.

    ReplyDelete
  21. We were finally able to figure out how to get the software working and here are some videos we filmed of this project

    http://youtu.be/Gc6N31J6mtE?hd=1

    http://youtu.be/-e4iM21I5a4?hd=1

    Thanks for putting all the info and code on the internet for anyone to build.

    -Jesse

    ReplyDelete
  22. Hi awesome job i would like to assemble this and i noticed the camera is now discontinued can any one hlp or if possible i would like to buy this assembled please help...

    ReplyDelete
  23. The pan tilt camera has apparently been replaced with this: http://www.dpcav.com/xcart/product.php?productid=16598&cat=0&page=2

    We have not tested our system with the new model.

    ReplyDelete
  24. Hello I have build camera tracking system but compile is error

    Help me .... Thank you very much.

    Compiling with CUDA Build Rule...

    Linking...

    Camera.obj : error LNK2019: unresolved external symbol _cvCreateCameraCapture referenced in function "public: __thiscall Camera::Camera(int const &)" (??0Camera@@QAE@ABH@Z)

    Camera.obj : error LNK2019: unresolved external symbol _cvReleaseCapture referenced in function "public: __thiscall Camera::~Camera(void)" (??1Camera@@QAE@XZ)

    ReplyDelete
  25. Help me compile is error.


    1>------ Build started: Project: cameraTrack, Configuration: Debug Win32 ------
    1>Linking...
    1>Camera.obj : error LNK2019: unresolved external symbol _cvCreateCameraCapture referenced in function "public: __thiscall Camera::Camera(int const &)" (??0Camera@@QAE@ABH@Z)
    1>Camera.obj : error LNK2019: unresolved external symbol _cvReleaseCapture referenced in function "public: __thiscall Camera::~Camera(void)" (??1Camera@@QAE@XZ)
    1>Camera.obj : error LNK2019: unresolved external symbol _cvFlip referenced in function "public: struct _IplImage * __thiscall Camera::getNextFrame(void)" (?getNextFrame@Camera@@QAEPAU_IplImage@@XZ)
    1>Camera.obj : error LNK2019: unresolved external symbol _cvQueryFrame referenced in function "public: struct _IplImage * __thiscall Camera::getNextFrame(void)" (?getNextFrame@Camera@@QAEPAU_IplImage@@XZ)
    1>cameraTrack.cu.obj : error LNK2001: unresolved external symbol __imp____glewBufferDataARB
    1>cameraTrack.cu.obj : error LNK2001: unresolved external symbol __imp____glewBindBufferARB
    1>main.obj : error LNK2001: unresolved external symbol __imp____glewBindBufferARB
    1>cameraTrack.cu.obj : error LNK2001: unresolved external symbol __imp____glewGenBuffersARB
    1>main.obj : error LNK2019: unresolved external symbol _cvReleaseImage referenced in function "void __cdecl cleanUp(void)" (?cleanUp@@YAXXZ)
    1>main.obj : error LNK2019: unresolved external symbol _cvCreateImage referenced in function _main
    1>main.obj : error LNK2019: unresolved external symbol __imp__glewInit referenced in function _main
    1>main.obj : error LNK2019: unresolved external symbol _cvGetCaptureProperty referenced in function "public: int __thiscall Camera::width(void)const " (?width@Camera@@QBEHXZ)
    1>../../bin/win32/Debug/cameraTrack.exe : fatal error LNK1120: 11 unresolved externals
    1>Build log was saved at "file://c:\Program Files\NVIDIA Corporation\NVIDIA GPU Computing SDK\C\src\cameraTrack\Debug\BuildLog.htm"
    1>cameraTrack - 13 error(s), 0 warning(s)
    ========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

    ReplyDelete
  26. Hi. Nice work.
    What if there are two moving objects (two teachers)? Will this work too?
    --Aadeesh

    ReplyDelete
  27. This comment has been removed by the author.

    ReplyDelete
  28. Have you guys ever tried star/planet/moon/sun/ISS tracking?

    ReplyDelete
  29. can you contact me on iluzionist@gmail.com

    ReplyDelete
  30. Not necessarily a Gay sizegenetics means sex, you'll find sizegeneticss done by gay men for gay clients (men). The sizegeneticss are attractive and good built and they would waste no time and make no bones to satisfy every tingling need which you just as a client may ask them to provide. By removing a look online, likely find a good sizegeneticss displaying their profiles and ads and would be happy to find you serviced over a offers and demands you create with terms and conditions for the Gay sizegenetics in your head. .
    http://buysizegeneticsonline.tumblr.com/

    ReplyDelete
  31. www.security360cameras.com provides affordable surveillance equipment for business and homes that can be remotely monitored with any PC, smart phones, i-pad, i-phone,mac or laptop. Clients do not have to pay a monthly fee. The apps are free and easy to use. The Stand Alone DVR's are easy to use . One can add as much Hard drive as they wish, We provide special AV( Audio Video) specially made by western digital for this purpose wireless camera.


    ReplyDelete
  32. non trovo il codice...arduino code software link not disponible ?? best regards for link cotecorsaire@gmail.com

    ReplyDelete
  33. Hi there, great work, is the software source stil available?

    ReplyDelete
  34. hello

    first of all I would like felicitarls for their great achievements in this project so interesting. I am currently undertaking the creation of a vision system on arduino as a classroom project with undergraduate students in engineering, and I would seem interesting to evaluate and learn about its development because I think a good implementation.

    unfortunately the source code available on the homepage of the project http://umassgv.blogspot.com/2010/07/realtime-tracking-with-pan-tilt-camera.html
    presents the link broken, could you kindly provide me appreciate some link where you can access it, but for the educational purpose that would give that code to give a new approach to classes.

    very grateful towards their care.

    Att William Wheats. malliwi88@hotmail.com

    ReplyDelete
  35. Hi, awesome work!! may i take your source? link broken...
    armada741@naver.com

    ReplyDelete
  36. Hi awesome job i would like to assemble this and i noticed the camera is now discontinued can any one hlp or if possible i would like to buy this assembled please help...

    ReplyDelete
  37. Great article ...Thanks for your great information, the contents are quiet interesting. I will be waiting for your next post.

    Cctv Price

    ReplyDelete
  38. hello,
    great article. i was wondering if you could provide the source code as the link is broken. i would really appreciate it.
    my email id is suraj.mehta@ymail.com
    thanks

    ReplyDelete
  39. Hello, my name Daniel

    that's great project, i like to combine with my RC Car

    could you provide the source code ?..
    i try to get the link but error..
    i would really appreciate it

    my email ; danieldwi69@gmail.com
    thanks

    ReplyDelete
  40. This comment has been removed by the author.

    ReplyDelete
  41. Hi, very nice project.Could you please share the software too?.My email address is sainer.s@gmail.com

    ReplyDelete
  42. Hi, very nice project.Could you please share the software too?.My email address is sainer.s@gmail.com

    ReplyDelete
  43. Hi, very nice project.Could you please share the software too?.My email address is sainer.s@gmail.com

    ReplyDelete
  44. Yes wireless camera concept is very nice. very useful information is provided here in this blog. I'm using this Wireless camera for better viewing.

    ReplyDelete
  45. very interesting and nice implementing your plan thank you for the beautiful information. συναγερμοι σπιτιων

    ReplyDelete
  46. A Professionally managed storage system, storage solutions, material holding, established in 1980, in UAE. Our expertise has a decades of experience indulged in major shelving projects in Emirates and East African countries. It provides mezzanine flooring and different shelving systems/shelving units like metal shelves, boltless shelving, mobile shelving, slotted Angle shelving, storage pallets, heavy duty pallet racks, storage racks, pallet racking systems and also a plastic pallet supplier especially in UAE. All our products are manufactured from high quality steel and finished with a powder coated finish for a long lasting durabilty.

    To ensure that you get maximum space utility, our qualified storage space advisors utilize modern computer programming techniques. We design, Supply and Install Storage Systems from renowned and ISO certified manufacturers from Malaysia, India and Italy.

    ReplyDelete
  47. Hi, I am not a daily blogger but I like to read article or blogs, mainly I read all blogs, it is my habit to collect information where ever I can, it will never waste ,today I caught your nice blog and it is written nicely with good content.....I appreciate it.....

    Heavy duty racking in dubai

    ReplyDelete
  48. HI NICE PROJECT I AM TRYING FOR IT FROM 3 months if you can share the software too to my mail

    dnagasuman@gmail.com

    ReplyDelete
  49. Hi, very nice project.Could you please share the software too?.My email address is riadi.zakaria@gmail.com

    ReplyDelete
  50. Hi, amizing project.Could you please share the software too?.My email address is
    eli_sayag@hotmail.com

    ReplyDelete
  51. hi
    this is the prefect project i need could you please email me the source codes
    thank you so much

    vj_yadav@hotmail.co.uk

    ReplyDelete
  52. Great information on the real time GPS tracker. Really it helps to know about the GPS tracker.

    ReplyDelete
  53. Hello, this project has exactly what I am trying to work on. I need to track Black helium balloons up to 3000-4000 feet in the air. I would be adding a 3-Axis sensor unit to give me the inclination angle and compass bearing to the inflight balloons. Would you please email me the source codes. Thanks very much.

    my email is aeronautdw@gmail.com

    ReplyDelete