PurposeRadiation necrosis (RN) is an important complication of radiation therapy (RT) and is challenging to assess radiographically because lesions are prone to inconsistent delineation, often necessitating intracranial surgery to obtain a diagnosis. This study evaluated how established deep learning segmentation models can be retrained on an institutional RN cohort, assessed external generalization, and integrated the best components into RN-D3, an end-to-end pipeline for RN detection, differentiation, and delineation.Methods and materialsWe trained models on a cohort of 52 patients with RN after proton beam RT treated at Massachusetts General Hospital (2004–2016) and tested these models on an external cohort of 29 patients with 39 RN lesions from the MOLAB brain metastasis dataset (photon-based RT, five Spanish institutions, 2005–2021). Six architectures were evaluated: four convolutional neural networks (nnU-Net, STU-Net, TRIAD PlainConv, and Spark3D) and two transformers (SwinUNETR and TRIAD SwinB). Training strategies included training from scratch, supervised pretraining, self-supervised pretraining, and foundation model initialization. Matched-encoder pairs enabled direct comparison of pretraining effects. Performance was evaluated by Dice similarity coefficient (DSC), surface Dice at 2 mm, and lesion detection rate. A post-processing pipeline combining encoder features and shape radiomics was assessed for reducing false positives. The best segmentation model and this classifier were combined into RN-D3, an end-to-end pipeline evaluated on 115 brain metastasis (BM) lesions from MOLAB for RN-versus-non-RN differentiation.ResultsSpark3D and STU-Net achieved the highest external DSC (median 0.69 [IQR 0.50–0.84] and 0.65 [0.29–0.88]), followed by baseline nnU-Net (0.61 [0.39–0.83]). For Spark3D and STU-Net, large lesions (≥1 mL, n = 21) were detected at 100% (median DSC 0.77 and 0.73) and small lesions (<1 mL, n = 18) at 78 and 67% (DSC 0.60 and 0.58). The remaining models detected 17–22% of small lesions with a median DSC near zero. False-positive filtering removed 2/13 and 7/14 false positives for Spark3D and STU-Net, retaining all true detections for Spark3D and 85% for STU-Net. RN-D3 achieved an end-to-end sensitivity of 0.90 and a specificity of 0.88 on 115 BM lesions for RN-versus-non-RN differentiation. The two transformer-based architectures evaluated showed larger drops from internal to external cohorts (ΔDSC −0.36 to −0.42 vs. − 0.12 to −0.20 for CNNs), though all architectures degraded externally.ConclusionRN-D3 integrates a CNN segmentation backbone with an RN-versus-non-RN classifier into an end-to-end pipeline addressing detection, differentiation, and delineation of RN. On external data, RN-D3 achieved high end-to-end sensitivity and high specificity in patients with brain metastases, with detection of small lesions the main determinant of missed cases. Small-lesion detection remains the primary translational bottleneck.