Purpose: This study explores factors that are associated with the severity of breast cancer (BC) at diagnosis. Methods: Interviews were conducted among women (n = 3326) aged 20–79 diagnosed with BC between 2011 and 2013 in Queensland, Australia. High-severity cancers were defined as either Stage II–IV, Grade 3, or having negative hormone receptors at diagnosis. Logistic regression models were used to estimate odds ratios (ORs) of high severity BC for variables relating to screening, lifestyle, reproductive habits, family history, socioeconomic status, and area disadvantage. Results: Symptom-detected women had greater odds (OR 3.38, 2.86–4.00) of being diagnosed with high-severity cancer than screen-detected women. Women who did not have regular mammograms had greater odds (OR 1.78, 1.40–2.28) of being diagnosed with high-severity cancer than those who had mammograms biennially. This trend was significant in both screen-detected and symptom-detected women. Screen-detected women who were non-smokers (OR 1.77, 1.16–2.71), postmenopausal (OR 2.01, 1.42–2.84), or employed (OR 1.46, 1.15–1.85) had greater odds of being diagnosed with high-severity cancer than those who were current smokers, premenopausal, or unemployed. Symptom-detected women being overweight (OR 1.67, 1.31–2.14), postmenopausal (OR 2.01, 1.43–2.82), had hormone replacement therapy (HRT) < 2 years (OR 1.60, 1.02–2.51) had greater odds of being diagnosed with high-severity cancer than those of healthy weight, premenopausal, had HRT > 10 years. Conclusion: Screen-detected women and women who had mammograms biennially had lower odds of being diagnosed with high-severity breast cancer, which highlighted the benefit of regular breast cancer screening. Women in subgroups who are more likely to have more severe cancers should be particularly encouraged to participate in regular mammography screening.