Abstract: The deformation stability of rock test system is studied based on rigidity criterion and nonlinear catastrophe theory.The rock deformation stability is dependent on the environment rigidity and the equivalent rigidity of rock.The rock instability occurs at a position after the peak value in the constitutive curve.An increase in the environment rigidity will be beneficial to the system stability and rigid loading will generally not lead to a strong instability-caused destruction.The formula for the deformation value corresponding to the instability position of the test system is found based on the cusp catastrophe model and rigidity criterion.

Key words: Rock mechanics, Instability criterion, Constitutive model, Stability, Catastrophe, Environment rigidity