Traditionally rectal symptoms following pelvic/prostate radiotherapy are correlated to the dosimetry of the anorectum or a substructure of this. It has been suggested that the perirectal fat space (PRS) surrounding the rectum may also be relevant. This study considers the delineation and dosimetry of the PRS related to both rectal bleeding and control-related toxicity. Initially, a case–control cohort of 100 patients from the RADAR study were chosen based on presence/absence of rectal control-related toxicity. Automated contouring was developed to delineate the PRS. 79 of the 100 auto-segmentations were considered successful. Balanced case–control cohorts were defined from these cases. Atlas of Complication Incidence (ACI) were generated to relate the DVH of the PRS with specific rectal symptoms; rectal bleeding and control-related symptoms (LENT/SOM). ACI demonstrated that control-related symptoms were related to the dose distribution to the PRS which was confirmed with Wilcoxon rank sum test (p < 0.05). To the authors knowledge this is the first study implicating the dose distribution to the PRS to the incidence of control-related symptoms of rectal toxicity.