The problem of traffic signal control is essential but remains unsolved. Some researchers use online reinforcement learning, including the off-policy one, to derive an optimal control policy through interaction between agents and environments in simulation. However, it is difficult to deploy the policy in real transportation systems due to the gap between simulated and real traffic data. In this paper, we consider an offline reinforcement learning method to tackle the problem. First, we construct a realistic traffic environment and obtain offline data based on a classic actuated traffic signal controller. Then, we use an offline reinforcement learning algorithm, namely conservative Q-learning, to learn an efficient control policy via offline datasets. We conduct experiments on a typical road intersection and compare the conservative Q-learning policy with the actuated policy and two data-driven policies based on off-policy reinforcement learning and imitation learning. Empirical results indicate that in the offline-learning setting the conservative Q-learning policy performs significantly better than other baselines, including the actuated policy, but the other two data-driven policies perform poorly in test scenarios.