Background: The link between blood lipids and cardiovascular disease (CVD) is complex. Our aim was to assess the differential effect of blood lipids on CVD risk according to age, sex, body weight, diet quality, use of lipid-lowering drugs and presence of hypercholesterolemia. Methods: In this secondary analysis of the ATTICA prospective cohort study, serum blood lipids, i.e., total cholesterol (TC), low-density lipoprotein cholesterol (LDL-C), high-density lipoprotein cholesterol (HDL-C), triglycerides (TG) and liproprotein(a) [Lp(a)], and sociodemographic, anthropometric, lifestyle and clinical parameters were evaluated at baseline (2001/2002) in 2020 CVD-free men and women. CVD incidence was recorded at the 10-year follow-up (2011/2012). Results: All blood lipids assessed were univariately related to CVD risk; however, associations remained significant only for HDL-C and TG in multivariate models adjusted for age, sex, body mass index, smoking, Mediterranean Diet Score, physical activity, presence of hypercholesterolemia, hypertension and diabetes mellitus, use of lipid-lowering drugs, and family history of CVD [RR per 1 mg/dL (95% CI): 0.983 (0.967, 1.000) and 1.002 (1.001, 1.003), respectively]. In stratified analyses, TC and LDL-C predicted CVD risk in younger subjects, normal-weight subjects, and those not on lipid-lowering drugs, while HDL-C and TG were significant predictors in older subjects, those with low adherence to the Mediterranean diet, and hypercholesterolemic subjects; a significant effect on CVD risk was also observed for TG in males, overweight participants and lipid-lowering medication users and for Lp(a) in older subjects and females (all p ≤ 0.050). Conclusions: The impact of blood lipids on CVD risk differs according to several biological, lifestyle and clinical parameters.