The instability of the pillar was discussed based on the potential energy principle and the cusp catastrophe theory, and a simplified mechanical model of the pillar was established considering the mining effect. The necessary-sufficient conditions, the jump value of displacement of pillar and the released energy expressions were deduced. The results show that the instability of the pillar is related to the properties of the rock, the external force and the relative stiffness of the elastic area to the plastic area. The instability of system is like to occur with the enlarging of the softening area or the decreasing of E/λ. The calculation done shows that the estimated results correspond to practical experience.