Livestock grazing has shaped grassland ecosystems around the world. Previous studies indicated grazing showed various impacts on small rodents; however, most studies were conducted over 1–2 years without controlling for confounding factors such as immigration/emigration and predation in rodents. Brandt’s voles (Lasiopodomys brandtii) are generally recognized as pests because of food overlap with domestic herbivores, but are also important for biodiversity conservation because they provide nests or food to many birds. Fully understanding the ecological relationship between domestic herbivores and small mammals is essential to making ecosystem management decisions. To address these needs, we carried out a field experiment during the period 2010–2013 to assess the effects of sheep grazing on vegetation and the population density of Brandt’s voles along a gradient of three grazing intensities by using 12 large-scale enclosures. Responses of Brandt’s voles to livestock grazing varied with grazing intensity and year. As compared to the control group, sheep grazing had no effect on vole abundance in the first year but an overall negative effect on vole abundance in the following 3 years. Successive grazing caused decreases in survival and male body mass of voles, but had no significant effect on fecundity. Negative effects of grazing were associated with a grazing-induced deterioration in both food quantity (reflected by biomass and cover of less-preferred plants), and food quality (measured by tannin and total phenol content). Our findings highlight the urgent need for more flexible management of yearly rotational grazing to optimize livestock production while maintaining species diversity and ecosystem health.