High-power fiber lasers have been widely explored in engineering and science, and improving the beam quality restricted by transverse mode instability (TMI) is waiting to be solved for extra high-power applications. Here, we theoretically propose a phase transition model for understanding TMI in a fiber oscillator. A general dynamics model describing TMI is established by modifying the heat equation, and then a special dynamics model of a two-mode fiber (TMF) oscillator is obtained by applying this general dynamics model to TMF oscillator case. Theoretical analysis shows that there is a reversible phase transition point in this TMF oscillator model, which can well explain the sudden and reversible change of TMI. Based on linear stability analysis near the phase transition point, an analytical threshold formula of TMI is given to calculate the TMI threshold in the TMF oscillator. The calculated results are consistent with the reported experimental results. Furthermore, the relationship between the TMI threshold and several parameters was also discussed in detail such as laser wavelength, pump wavelength, core radius, cladding radius, etc. This theoretical model will be useful to understand and suppress the TMI in fiber oscillators. © 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement