OpenCV

I can't imagine why I didn't use this before. OpenCV is Intel® Open Source Computer Vision Library. It is a collection of C functions and C++ classes that implement some popular algorithms of Image Processing and Computer Vision. Look at this:
Motion Analysis and Object Tracking Reference
--------------------------------------------------------------------------------
Accumulation of Background Statistics
---Acc
---SquareAcc
---MultiplyAcc
---RunningAvg
Motion Templates
---UpdateMotionHistory
---CalcMotionGradient
---CalcGlobalOrientation
---SegmentMotion
Object Tracking
---MeanShift
---CamShift
---SnakeImage
Optical Flow
---CalcOpticalFlowHS
---CalcOpticalFlowLK
---CalcOpticalFlowBM
---CalcOpticalFlowPyrLK
Estimators
---Kalman
---CreateKalman
---ReleaseKalman
---KalmanPredict
---KalmanCorrect
---CreateConDensation
---ReleaseConDensation
---ConDensInitSampleSet
---ConDensUpdateByTime

Isn't it amazing? It is terrific actually.