Detection and Motion Analysis
MENTOR: Nicolas Saunier
There are 3 related projects here: improve OpenCV's background segmentation; develop good feature-based trackers, and add motion trajectory analysis.
- You must know C++ well
- You should have had a computer vision course and understand signal processing
- Knowledge of sqlite is an asset (trajectory analysis)
- Knowledge of a distributed version control software, preferably mercurial, is a strong asset
PROJECT: Feature-based Tracker
Project Description
Demo: run the tracker on a traffic video and overlay the trajectories over the video
Short term: put the pieces available in OpenCV together to implement one of the algorithm described in the references
Medium goal: implement at least two feature-based tracking methods with a similar interface, and clean options / configuration structures. The tracker should track reasonably well a couple of objects moving without occlusion
Long goal: track all moving objects in a complex scene despite partial and complete occlusion
Milestones/Timeline
What we want
- There is currently no implementation of a feature-based tracker in OpenCV. The building blocks are available (with the two functions cvGoodFeaturesToTrack and cvCalcOpticalFlowPyrLK) and there are a few well published algorithms (see the references). The goal of this project is to implement at least one version of a feature-based tracker, in a modular way so that alternative algorithms may be later easily added.
Work is being done on this at Willow Garage, so you should take in account (see the bottom of the page) http://pr.willowgarage.com/wiki/PluggableDescriptors
- There is also a visual surveillance demo (the demo is blobtrack.cpp in samples/c, while most of the code is in src/cvaux/vs). One possibility is to integrate the new feature-based tracker in this framework, while refactoring it if necessary (to a more straightforward architecture).
References:
Beymer, D.; McLauchlan, P.; Coifman, B. & Malik, J. A Real-time Computer Vision System for Measuring Traffic Parameters Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), IEEE Computer Society, 1997, 495-501
Kanhere, N. & Birchfield, S. Real-Time Incremental Segmentation and Tracking of Vehicles at Low Camera Angles Using Stable Features IEEE Transactions on Intelligent Transportation Systems, 2008, 9, 148-160
Saunier, N. & Sayed, T. A feature-based tracking algorithm for vehicles in intersections Third Canadian Conference on Computer and Robot Vision, IEEE, 2006
- Kim, Z. Real time object tracking based on dynamic feature grouping with background subtraction Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2008, 1-8
PROJECT: A New Interface for Background Modeling
Project Description
Demo: run the background modeling and display the foreground objects and best background estimate at each frame (for different models)
Short term: propose a common class design for a generic background model and re-factor the two existing background model according to this model
Medium goal: implement at least a simple background model and the original Mixture of Gaussians algorithm and allow the extraction of foreground objects
Long goal: extensive tests of the most common background models with recommendations for specific applications (traffic monitoring, dealing with sudden light changes, slight camera motion...), including recommended parameter values
Milestones/Timeline
What we want
- There are currently two core background models in OpenCV (described in cvaux.h).
- CV_BG_MODEL_FGD: latest and greatest algorithm, described in Foreground Object Detection from Videos Containing Complex Background. Liyuan Li, Weimin Huang, Irene Y.H. Gu, and Qi Tian. ACM MM2003 9p
- CV_BG_MODEL_FGD_SIMPLE: A code comment describes this as a simplified version of the above, but the code is in fact currently identical
CV_BG_MODEL_MOG: “Mixture of Gaussians”, older algorithm, described in P. KadewTraKuPong and R. Bowden, Proc. 2nd European Workshp on Advanced Video-Based Surveillance Systems, 2001. http://personal.ee.surrey.ac.uk/Personal/R.Bowden/publications/avbs01/avbs01.pdf
- Unfortunately, these background models do not have currently a common interface. Preliminary new code was committed by Vadim Pisarevsky very recently (in cvaux.hpp). The goal of this project is to refactor the existing background models (using an abstract base class for example). A very desirable feature is to allow to extract the foreground objects without updating the background model and implement other models. The current “Mixture of Gaussians” is a slightly modified version of one of the most famous algorithm described in the first reference below. It would be a good idea to add it (e.g. by modifying the existing similar model (CV_BG_MODEL_MOG)), as well as more simple ones like median filtering. Another approach relies on adaptive kernel density estimation (See the three last references below).
References:
- Learning patterns of activity using real-time tracking. C Stauffer and W Grimson August 2000. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8):747-757.
- Efficient adaptive density estimation per image pixel for the task of background subtraction. Zivkovic, Zoran and van der Heijden, Ferdinand. Pattern Recognition Letters, ISSN: 01678655, Vol: 27, Issue: 7, Date: May, 2006, Pages: 773-780
- Motion-Based Background Subtraction Using Adaptive Kernel Density Estimation. Mittal, A. Paragios, N. IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2004, VOL 2, pages II-302-309
"Efficient adaptive density estimation per image pixel for the task of background subtraction", Z. Zivkovic , F. van der Heijden, Pattern Recognition Letters, vol. 27, no. 7, pages 773-780, 2006 (code from http://staff.science.uva.nl/~zivkovic/DOWNLOAD.html)
PROJECT: Trajectory Management and Analysis
Project Description
Demo: input a trajectory (either by tracking motion in a video, or from the mouse) and return the most similar trajectories from a database
Short term: propose a design of the data structure to store trajectories, of the database and implement a prototype
Medium goal: implement the trajectory distances/similarities (Euclidean distance, dynamic time warping and the longest common sub-sequence)
Long goal: run benchmarks of the various components, such as reading / writing large amounts of data, random access in the database and queries for similar trajectories (see the demo)
Milestones/Timeline
What we want
- Trajectories, i.e. sequences of positions in time, are a very common output of vision systems.
- Yet, there is no simple tool ready to use in OpenCV, or other open source library to my knowledge, that offers storage and analysis methods for trajectories.
- The goal of this project is to add such functionalities to OpenCV:
- 1) reading and writing of trajectories (e.g. using sqlite),
- 2) visualize trajectories, either in highgui in image space, or using another tool (e.g. python and matplotlib),
- 3) analyse trajectories, in particular using distances like dynamic time warping and the longest common sub-sequence (and provide examples using classifiers in the OpenCV machine learning library such as k-means and k nearest neigbours).
- The code should accommodate large numbers of trajectories (e.g. storing all the feature trajectories in a feature-based tracker).