While doing summarization, human needs to under-
stand the whole document, rather than separately understanding
each sentence in the document. However, inter-sentence features
within one document are not adequately modeled by previous
neural network-based models that almost use only one layer
recurrent neural network as document encoder. To learn high
quality context-aware representation, we propose a shortcut-
stacked document encoder for extractive summarization. We use
multiple stacked bidirectional long short-term memory (LSTM)
layers and add shortcut connections between LSTM layers to
increase representation capacity. The shortcut-stacked document
encoder is built on a temporal convolutional neural network-
based sentence encoder to capture the hierarchical structure
of the document. Then sentence representations encoded by
document encoder are fed to a sentence selection classifier for
summary extraction. Experiments on the well-known CNN/Daily
Mail dataset show that the proposed model outperforms several
recently proposed strong baselines, including both extractive
and abstractive neural network-based models. Furthermore, the
ablation analysis and position analysis also demonstrate the
effectiveness of the proposed shortcut-stacked document encoder.