Objectives Cross-sectional studies have found some built environmental attributes to be associated with residents’ lower levels of mobility (functional capacity to walk outside the home). However, less is known about what environmental attributes are related to mobility decline. This longitudinal study examined area-level associations of specific environmental attributes with mid-to-older aged adults’ changes in walking mobility. Methods Data collected from 4,088 adults (aged 46–71 years at baseline) who participated in a cohort study in Brisbane, Australia were used. The outcome was the change in self-reported mobility score (SF-36) from 2013 to 2016, which were aggregated at the neighborhood (N = 156) and suburb (N = 99) levels, due to the known lack of sensitivity in SF-36 subscales to individual changes. Linear regression analysis examined associations of mobility change with seven environmental attributes measured at baseline (residential density, intersection density, land use mix, density of walking/bike paths, park density, bus stop density, density of social incivilities), adjusting for confounding variables. Results Participants on average reported 4% of mobility decline during the 3-year study period. It was found that greater land use diversity was consistently associated with less decline in walking mobility, while greater density of social incivilities was associated with more decline in walking mobility. The latter finding was significant only at the neighborhood level. No consistent associations were observed for residential density, intersection density, density of walking/bike paths, park density, and bus stop density. Discussion Our findings suggest that mid-to-older aged adults who live in areas with lower land use diversity and more social incivilities may be at risk of developing mobility limitations. Recommended policies to slow residents’ mobility decline and to achieve aging in place include improving these environmental attributes where needed and advising older adults to relocate to safer, mixed-use neighborhoods.