Objectives: To evaluate the strength of the evidence for, and the extent of, overdiagnosis in noncancer conditions. Study Design and Setting: We systematically searched for studies investigating overdiagnosis in noncancer conditions. Using the ‘Fair Umpire’ framework to assess the evidence that cases diagnosed by one diagnostic strategy but not by another may be overdiagnosed, two reviewers independently identified whether a Fair Umpire—a disease-specific clinical outcome, a test result or risk factor that can determine whether an additional case does or does not have disease—was present. Disease-specific clinical outcomes provide the strongest evidence for overdiagnosis, follow-up or concurrent tests provide weaker evidence, and risk factors provide only weak evidence. Studies without a Fair Umpire provide the weakest evidence of overdiagnosis. Results: Of 132 studies, 47 (36%) did not include a Fair Umpire to adjudicate additional diagnoses. When present, the most common Umpire was a single test or risk factor (32% of studies), with disease-specific clinical outcome Umpires used in only 21% of studies. Estimates of overdiagnosis included 43–45% of screen-detected acute abdominal aneurysms, 54% of cases of acute kidney injury, and 77% of cases of oligohydramnios in pregnancy. Conclusion: Much of the current evidence for overdiagnosis in noncancer conditions is weak. Application of the framework can guide development of robust studies to detect and estimate overdiagnosis in noncancer conditions, ultimately informing evidence-based policies to reduce it.