The purpose of this work was to develop a framework for 3D fusion of CT coronary angiography (CTCA) and whole-heart dynamic 3D cardiac magnetic resonance perfusion (3D-CMR-Perf) image data-correlating coronary artery stenoses to stress-induced myocardial perfusion deficits for the assessment of coronary artery disease (CAD). Twenty-three patients who underwent CTCA and 3D-CMR-Perf for various indications were included retrospectively. For CTCA, image quality and coronary diameter stenoses > 50% were documented. For 3D-CMR-Perf, image quality and stress-induced perfusion deficits were noted. A software framework was developed to allow for 3D image fusion of both datasets. Computation steps included: (1) fully automated segmentation of coronary arteries and heart contours from CT; (2) manual segmentation of the left ventricle in 3D-CMR-Perf images; (3) semi-automatic co-registration of CT/CMR datasets; (4) projection of the 3D-CMR-Perf values on the CT left ventricle. 3D fusion analysis was compared to separate inspection of CTCA and 3D-CMR-Perf data. CT and CMR scans resulted in an image quality being rated as good to excellent (mean scores 3.5 ± 0.5 and 3.7 ± 0.4, respectively, scale 1-4). 3D-fusion was feasible in all 23 patients, and perfusion deficits could be correlated to culprit coronary lesions in all but one case (22/23 = 96%). Compared to separate analysis of CT and CMR data, coronary supply territories of 3D-CMR-Perf perfusion deficits were refined in two cases (2/23 = 9%), and the relevance of stenoses in CTCA was re-judged in four cases (4/23 = 17%). In conclusion, 3D fusion of CTCA/3D-CMR-Perf facilitates anatomic correlation of coronary lesions and stress-induced myocardial perfusion deficits thereby helping to refine diagnostic assessment of CAD.