In this study, a comprehensive experimental program was developed to determine the microstructural, mechanical, and fracture features of calcium aluminate cement concrete (CACC) after exposure to high temperatures. The residual fracture parameters, including fracture toughness, fracture energy, and characteristic length, were obtained based on RILEM recommendations. In addition, XRD and SEM were used to evaluate microstructural changes after exposure to different temperatures. Mechanical properties such as the residual compressive, tensile, and flexural strengths, as well as the elastic modulus, were also assessed. The SEM images revealed that the voids among the particles increased due to increasing internal vapor pressure, indicating that the number of pores in the concrete structure increased. Furthermore, the results showed that after subjecting the specimens to high temperatures, the concrete became more ductile, which may be due to the increase in the number of pores after water evaporation. The residual fracture energy of CACC was observed to increase with increasing temperature. However, the residual fracture toughness and flexural strength decreased as temperatures increased. In addition, the results demonstrated that, above 400 °C, the weight loss in the concrete is mainly due to the evaporation of chemically bound water and decomposition of cement hydration compounds, which are chemical processes.