This study presents a hybrid algorithm incorporating Particle Swarm Optimization (PSO) and Particle Filter (PF) for multiple-object tracking based mainly on gray-level histogram model. To start with, the hybrid object tracker uses PSO to search the objects in the beginning, taking advantage of the PSO for global optimization. Once the objects have been successfully found by PSO, the hybrid object tracker then switches to PF to continuously track the objects. To avoid the varying-size problem of the objects, Speeded Up Robust Features (SURF) is used to detect the object around its neighborhood in the video sequence for defining the real image size of the object for remodeling the target object by histogram. As a result, tracking speed can be maintained by the hybrid tracker using simple histogram model while circumventing the varying-size problem of the objects during the tracking process.