Favorable isolation performance in optical transmitting-receiving (T/R) channels is key to establishing full-duplex bi-directional laser communication links when designing mono-wavelength laser communication terminals. This study proposes a novel design scheme for T/R channels that use a single laser wavelength. The system includes measures such as the deflection at λ/4 wave plate angles, light trap installation at a polarization beam splitter, and removal of polarization analyzers in optimized systems. A mathematical model describing isolation in the optical T/R channels is presented. Three types of factors that affect the isolation of optical T/R channels were analyzed and simulated in detail. The present simulation results show that the roughness of the λ/4 wave plate and the solid angle of the detector field of view are exponentially related to T/R isolation, which can thus be regarded as the most significant factors affecting isolation. According to the present design scheme, the prototype T/R channel was developed for measuring isolation. The measured value (77.8 dB) agrees well with the result determined using the established mathmatical model (77.3 dB). Conclusively, the present theoretical model can provideguidance for optimizing isolation in a mono-wavelength optical channel. Thus, it can be applied extensively in laser communication systems, including inter-satellite networks. ? 2018 Elsevier GmbH