The forest-water relationship is a hot and important issue in ecohydrology studies. China has implemented many large-scale reforestation programs in the past decades to address growing soil erosion and desertification, but that also caused serious water yield reductions. In this study, based on the simple water balance equation, we made statistical and graphic analyses on the long-term hydrological data of the 42 watersheds in the rocky mountain area of northern China, and then we were able to analyze the impact trend of forest on water yield. Our results show that the impact of forest on runoff is very weak in the lower-precipitation regions (in this study <500 mm). In the higher-precipitation regions (>500 mm) the impact of forest on reducing water yield is different with the increasing forest coverage (f), and runoff shows an impact trend of “weak-big-severe”; the largest absolute MAR is about 225 mm, and the largest MAR ratio is about 35%. We highlight the importance of dividing different forest-coverage phases in analyzing the effect of a forest on water yield.