Background: Epidemiological studies suggest physical activity (PA) slow or prevent both cognitive decline and age-related atrophy in frontal and hippocampal gray matter volumes. However, much of this evidence is based on self-reported measures of PA. Methods: PA was measured objectively with a SenseWear™ Armband to examine the cross-sectional associations between the duration of light, moderate and vigorous intensity PA with gray matter volume in the dorsolateral prefrontal cortex (DLPFC) and hippocampus in 167 (female: 43%) cognitively healthy older adults aged 73 to 78. Results: The duration of objective moderate to vigorous intensity physical activity (MVPA) was associated with a greater volume of the right DLPFC (β = 0.16; p = 0.04). In addition, objective moderate-intensity PA alone was also associated with greater volume of the left (β = 0.17; p = 0.03) and right (β = 0.19; p = 0.01) DLPFC after controlling for covariates and adjustment for multiple comparisons. In contrast, there were no significant associations between light- or vigorous-intensity PA and gray matter volumes (all p > 0.05). No associations between PA and cognitive performance were detected, and self-reported PA was not associated with any of the outcomes investigated. Conclusions: These findings suggest that an intensity-dependent relationship may exist, whereby a greater duration of MVPA, perhaps driven by moderate-intensity PA, is associated with preserved gray matter volume in frontal regions of the brain. Future research should investigate the mechanisms of this dose-effect and determine whether greater brain volumes associated with objective PA convey protective effects against cognitive decline.