Image data captured from outdoor visual devices are usually degraded by turbid media, such as haze, smoke, fog, rain, or snow. Therefore, weather conditions would usually disrupt or degrade proper functioning of vision-assisted transportation systems or ADAS (advanced driver assistance systems), as well as several other outdoor surveillance-based systems. To cope with these problems, removal of weather effects (or deweathering) from images has been important and received much attention. Hence, it is important to provide a preprocessing stage to automatically determine the weather condition for an input image, and then the corresponding proper deweathering operations (e.g., removals of haze, rain, or snow) would be correctly triggered accordingly. This paper presents a deep learning-based weather image recognition framework by considering the three most common weather conditions, including hazy, rainy, and snowy, in outdoor scenes. For an input image, our method automatically classifies the image into one of the three categories or none of them (e.g., sunny or others). Extensive experiments based on both well-known deep networks, GoogLeNet and AlexNet, are conducted on open weather image dataset to evaluate the proposed method and the feasibility has been verified.