Previous functional magnetic resonance imaging (fMRI) studies have shown that brain responses to nociceptive pain, non-nociceptive somatosensory, visual, and auditory stimuli are extremely similar. Actually, perception of external sensory stimulation requires complex interactions among distributed cortical and subcortical brain regions. However, the interactions among these regions elicited by nociceptive pain remain unclear, which limits our understanding of mechanisms of pain from a brain network perspective. Task fMRI data were collected with a random sequence of intermixed stimuli of four sensory modalities in 80 healthy subjects. Whole-brain psychophysiological interaction analysis was performed to identify task-modulated functional connectivity (FC) patterns for each modality. Task-modulated FC strength and graph-theoretical-based network properties were compared among the four modalities. Lastly, we performed across-sensory-modality prediction analysis based on the whole-brain task-modulated FC patterns to confirm the specific relationship between brain patterns and sensory modalities. For each sensory modality, task-modulated FC patterns were distributed over widespread brain regions beyond those typically activated or deactivated during the stimulation. As compared with the other three sensory modalities, nociceptive stimulation exhibited significantly different patterns (more widespread and stronger FC within the cingulo-opercular network, between cingulo-opercular and sensorimotor networks, between cingulo-opercular and emotional networks, and between default mode and emotional networks) and global property (smaller modularity). Further, a cross-sensory-modality prediction analysis found that task-modulated FC patterns could predict sensory modality at the subject level successfully. Collectively, these results demonstrated that the whole-brain task-modulated FC is preferentially modulated by pain, thus providing new insights into the neural mechanisms of pain processing.