Human-machine interaction is a popular research filed recently. Microsoft Kinect can provide us an economical way for tracking the skeleton of the human body. In this study, we present an efficient Kinect-based method for a potential application of TV remote controller. The proposed method consists of two parts: four direction (up/down/left/right) recognition and handwritten digit recognition. The developed recognitions include five main steps: initialization, tracking skeleton, judging the start condition, recording the path, and recognition. We use an intuitive way to start the event and achieve high accurate recognition for the users. We apply the least square method for the four direction recognition and a support vector machine library for the handwritten digit recognition. Experimental results show that the proposed system has great potential on future human-machine development.