Debris flows commonly cause tremendous damage to buildings in mountainous areas. The identification of buildings susceptible to debris flows is vital for settlement risk management. The efficient identification method is a major issue limiting the targeted regional policy setting. By combining geographic information system (GIS) and Deep Encoding Network (DE-Net) methods, we proposed an automatic identification method for buildings highly susceptible to debris flows with large-scale digital elevation data and high-resolution remote sensing imagery. The judgment criteria were based on a vulnerability matrix containing different threshold values of the horizontal distance (HD) and vertical distance (VD) between buildings and channels obtained from the statistics of 362 buildings destroyed by 23 debris flows and the maximum debris flow depths of 26 events, respectively. Five steps, which are debris flow channel extraction, building extraction, building cluster segmentation, distance calculation, and building classification, were implemented in the method. A case study in Puge County, Sichuan Province, demonstrated the high identification potential of the method for buildings susceptible to debris flows in large areas with only scarce information available. The identification results provide valuable information regarding high-risk residential areas to governments and facilitate targeted measure design in these areas in the initial planning stage.