Real-time face recognition is a necessary feature in advanced portable surveillance systems. However, the high complexity of available algorithms to objects segmentation and recognition makes it impossible to include such a feature into a portable device. In this paper, we aim at designing a portable surveillance system which can take MPEG audio/video currently with recognizing human faces based on a digital camera platform. The proposed flow fully utilizes the available intermediate data passing from standard video compression flow in a digital camera. An efficient face recognition approach based on adaptive feature extraction and support vector machines (SVMs) are proposed to address the performance issues. The experimental result shows that the recognition speed running on a commercial digital camera platform can achieve 1.216 seconds/frame.