Objective: This study aimed to investigate cross-sectional and longitudinal associations between fat mass (i.e., body mass index [BMI], waist circumference [WC], and waist to hip ratio [WTHR]) and hippocampal volumes. Methods: UK Biobank participants (N = 20,395) aged 40 to 70 years (mean follow-up = 7.66 years), were included and categorized into one of four groups, which represented their baseline fat mass status and trajectory of change by follow-up assessment: normal weight to overweight/obesity, overweight/obesity to normal weight (ON), normal weight stable (NS), or overweight/obesity stable (OS). Regression models used NS (WC < 80 cm in women and < 94 cm in men; WTHR < 0.85 in women and < 0.90 in men; BMI < 25 kg/m2 in women and men) as the reference group. Hippocampal volumes were automatically segmented using the FMRIB Software Library. Results: Compared with NS, OS (BMI: B = −62.23 [SE = 16.76]; WC: B = −145.56 [SE = 16.97]; WTHR: B = −101.26 [SE = 19.54]) and ON (BMI: B = −61.1 [SE = 30.3]; WC: B = −93.77 [SE = 24.96]; WTHR: B = −69.92 [SE = 26.22]) had significantly lower hippocampal volumes. Conclusions: The detrimental effects of overweight/obesity may extend beyond the duration of overweight/obesity itself.