Mining-induced deformation and failure of overlying strata may trigger roof collapse, surface subsidence, and abnormal gas emissions. To address this, this study proposes a distributed optical fiber monitoring method for large-scale rock mass deformation, leveraging the high sensitivity and distributed sensing advantages of Brillouin optical time-domain analysis (BOTDA) technology. A physical similar model experiment (geometric similarity ratio 1:100) was conducted to investigate overlying strata deformation characterization, with the introduction of the Average Frequency Shift Variation (AFSV) as a core indicator. The findings indicate: (1) AFSV ≥ 20 MHz serves as a reliable threshold for identifying significant rock mass deformation and failure, correlating with mine pressure behavior; (2) The optical fiber frequency shift curve exhibits a step-like pattern, enabling accurate characterization of the overburden three-zone structure (caving zone, fractured zone, bending subsidence zone) with a sampling interval of 10 mm; (3) Cross-validation with Fiber Bragg Grating (FBG) sensors and total stations shows that the AFSV-based characterization results have a linear fitting coefficient R2 ≥ 0.99 with FBG wavelength drift data. This high level of agreement demonstrates the accuracy and reliability of the proposed method. This study demonstrates that AFSV can effectively reflect mine pressure behavior patterns, providing a quantitative and reliable dynamic monitoring basis for overlying strata zoning, roof stability evaluation, and gas extraction optimization in mining engineering.