In video surveillance, tracking-based approaches are very popular especially for detecting abandoned objects in public areas. Once the object has been tracked, the object status can be further classified as removed or abandoned. However, some shortcomings were found on tracking-based approaches, e.g. illumination changes and occlusion. Therefore, in this paper, an alternative approach to detect abandoned objects is proposed by incorporating background modeling and Markov model. In addition the shadow removal is employed to rectify detected objects and obtain more accurate results. The experimental results show that the proposed scheme is better than other methods in terms of accuracy and correctness.