A predicting model based on long-short-term-memory (LSTM) and gated recurrent unit (GRU) is proposed to assist autonomous vehicles (AVs) to drive safely. To understand the behaviors of surroundings under a mixed scene of vehicles, bicycles, and pedestrians, the proposed model can predict the future trajectory of each object with models constructed by GRU. Since different objects have diverse behaviors, this paper applies different models to different categories for vehicles, pedestrians, and cyclists. For each object, the proposed model considers three observed trajectories with different time steps as the input data to predict a more accurate future trajectory. The proposed model is verified and compared with LSTM and GRU on KITTI dataset in the conducted experiments.