Although it is of great importance to recognize weather conditions automatically, this task has not been explored thoroughly in practice. Generally, most approaches in the literature simply treat it as a common image classification task, i.e., assigning a certain weather label to each image. However, there are significant differences between weather recognition and common image classification, since several weather conditions tend to occur simultaneously, like foggy and cloudy. Obviously, a single weather label is insufficient to provide a comprehensive description of the weather conditions. In this case, we propose to utilize auxiliary weather-cues, e.g., black clouds and blue sky, for comprehensive weather description. Specifically, a multi-task framework is designed to jointly deal with the weather-cue segmentation task and weather classification task. Benefit from the intrinsic relationships lying in the two tasks, exploring the information of weather-cues can not only provide a comprehensive description of weather conditions, but also help the weather classification task to learn more effective features, and further improve the performance. Besides, we construct two large-scale weather recognition datasets equipped with both weather labels and segmentation masks of weather-cues. Experiment results demonstrate the excellent performance of our approach. The constructed two datasets will be available at https://github.com/wzgwzg/Multitask_Weather. © 2019 Elsevier Ltd