In today's world, video surveillance appears everywhere in our life. The challenge for a video surveillance system is to recognize the object of interest for analysis. The convolutional neural network (CNN) models achieve high accuracy on image recognition, but the models require powerful calculations. The models cannot be applied to most smart surveillance systems directly. In this paper, we propose a low-cost CNN design for the application of surveillance systems. Instead of using GPUs, we use a hardware accelerator called Neural Compute Stick (NCS) accompanied with the Rock64 to build the system. The NCS is a low-cost and low-power USB device, which has the advantages in the high-speed calculation of images. As a result, we use the NCS to load the Single Shot MultiBox Detector (SSD) network for human detection. Our system can get each detected image in 0.15 sec. It is six times faster than other single-board surveillance systems. Furthermore, the cost of building the real-time surveillance system is less than 100. Therefore, our system can achieve a low-cost and high-performance intelligent surveillance system.