This paper presents a detection-based method to subtract abandoned object from a surveillance scene. Unlike tracking-based approaches that are commonly complicated and unreliable on a crowded scene, the proposed method employs background (BG) modelling and focus only on immobile objects. The main contribution of our work is to build abandoned object detection system which is robust and can resist interference (shadow, illumination changes and occlusion). In addition, we introduce the MRF model and shadow removal to our system. MRF is a promising way to model neighbours' information when labeling the pixel that is either set to background or abandoned object. It represents the correlation and dependency in a pixel and its neighbours. By incorporating the MRF model, as shown in the experimental part, our method can efficiently reduce the false alarm. To evaluate the system's robustness, several dataset including CAVIAR datasets and outdoor test cases are both tested in our experiments.