Automated nonlinear registration of coronary PET to CT angiography using pseudo-CT generated from PET with generative adversarial networks

Coronary 18F-sodium-fluoride (18F-NaF) positron emission tomography (PET) showed promise in imaging coronary artery disease activity. Currently image processing remains subjective due to the need for manual registration of PET and computed tomography (CT) angiography data. We aimed to develop a novel fully automated method to register coronary 18F-NaF PET to CT angiography using pseudo-CT generated by generative adversarial networks (GAN). A total of 169 patients, 139 in the training and 30 in the testing sets were considered for generation of pseudo-CT from non-attenuation corrected (NAC) PET using GAN. Non-rigid registration was used to register pseudo-CT to CT angiography and the resulting transformation was used to align PET with CT angiography. We compared translations, maximal standard uptake value (SUVmax) and target to background ratio (TBRmax) at the location of plaques, obtained after observer and automated alignment. Automatic end-to-end registration was performed for 30 patients with 88 coronary vessels and took 27.5 seconds per patient. Difference in displacement motion vectors between GAN-based and observer-based registration in the x-, y-, and z-directions was 0.8 ± 3.0, 0.7 ± 3.0, and 1.7 ± 3.9 mm, respectively. TBRmax had a coefficient of repeatability (CR) of 0.31, mean bias of 0.03 and narrow limits of agreement (LOA) (95% LOA: − 0.29 to 0.33). SUVmax had CR of 0.26, mean bias of 0 and narrow LOA (95% LOA: − 0.26 to 0.26). Pseudo-CT generated by GAN are perfectly registered to PET can be used to facilitate quick and fully automated registration of PET and CT angiography.


INTRODUCTION
Coronary positron emission tomography (PET) has shown promise for the non-invasive assessment of atherosclerotic plaque. 1 By targeting processes directly involved in plaque progression and rupture (including inflammation and microcalcification) PET has broadened our understanding of plaque biology. 2 Importantly, it has recently been demonstrated that beyond pathophysiological insights 18 F-sodium fluoride (18F-NaF) provides prognostic implications with the coronary microcalcification activity (CMA) acting as a strong independent predictor of myocardial infarction. 3,4 While 18 F-NaF emerged as a promising tool for risk stratification in CAD patients, wider adoption of this imaging modality remains challenging. 2 18 F-NaF PET requires co-registered computed tomography (CT) angiography images for precise anatomical localization of 18 F-NaF activity within coronary plaques. Although typically the CT angiography is acquired on a hybrid PET/CT scanner during the same imaging session as PET, in order to allow for patient repositioning and the respiratory phase at which the CT angiography is acquired the reading physician has to carefully coregister both datasets. 5 This important step is necessary for precise quantification of PET activity, which needs to be guided by the anatomical information derived from CT angiography. 6,7 Currently this step is time consuming, subjective, and requires great operator expertise, adding to the complexity of coronary PET protocols. In view of the already existing tools for 18 F-NaF PET quantification, CT angiography and PET co-registration emerge as the final obstacle for near-full automation of post-acquisition data processing and analysis which could facilitate widespread use of this promising imaging modality.
In the current study we aimed to develop and evaluate a novel, fully automated method for co-registering coronary PET and CT angiography datasets using a conditional generative adversarial network (GAN) 8,9 and a diffeomorphic nonlinear registration algorithm. 10,11 GANs are a type of deep learning algorithms where two neural networks are trained simultaneously, one responsible for generating images and the other classifying whether the generated images are realistic. The two networks are trained against each other, thus ''adversarial'', to generate new realistic data. The networks are prominently used for generation and translation of image data with the objective of learning the underlying distribution of source domain to generate indistinguishable target realistic data samples. In medical imaging, GANs are often used for denoising, 12 low dose to high-dose translation 13 , and increasing samples in medical imaging training datasets. 14 In our study we employ GAN to generate ''pseudo-CT'' from corresponding non-attenuation corrected (NAC) PET data. The generated pseudo-CT is perfectly aligned to PET as it is derived from the PET image, unlike the actual noncontrast CT, which is acquired during the same imaging session but is prone to misalignment due to patient motion. The pseudo-CT is then nonlinearly registered to CT angiography using diffeomorphic registration algorithm called demons which iteratively computes the displacement for each voxel in a computationally efficient manner. The resulting transformation is used for the final PET to CT angiography registration.

Patient population
One hundred sixty-nine patients with established coronary artery disease undergoing hybrid coronary 18 F-NaF PET and contrast CT angiography at the Edinburgh Heart Center within the investigator-initiated, doubleblind, randomized, parallel-group, placebo-controlled DIAMOND (Dual Antiplatelet Therapy to Reduce Myocardial Injury) trial (NCT02110303) were included in the current study. 15 All patients underwent a comprehensive baseline clinical assessment and hybrid 18 F-NaF PET imaging alongside coronary CT calcium scoring and coronary CT angiography. The study was approved by the local institutional review board, the Scottish Research Ethics Committee (REC reference: 14/SS/0089), and it was performed in accordance with the Declaration of Helsinki. All patients provided written informed consent before any study procedures were initiated.

Acquisition
All patients underwent 18 F-NaF PET on a hybrid PET/CT scanner (128-slice Biograph mCT, Siemens Medical Systems, Knoxville, Tennessee) 60 minutes after intravenous 18 F-NaF administration. During a single-imaging session, we acquired a non-gated noncontrast CT attenuation correction (AC) scan for attenuation correction purposes, followed by a 30-minute PET emission scan in list mode. The electrocardiogram (ECG)-gated list mode dataset was reconstructed using a standard ordered expectation maximization algorithm with time-of-flight and point-spread function correction. Using 4 cardiac gates, the data were reconstructed on a 256 9 256 matrix (with 75 slices using 2 iterations, 21 subsets, and 5-mm Gaussian smoothing). After the PET scan, based on standard protocol, a gated low-dose noncontrast ECG-gated CT for calculation of the coronary artery calcium score was performed. Subsequently, a contrast-enhanced, ECG-gated coronary CT angiogram was obtained in mid-diastole on the same PET/CT system without repositioning the patient. The ECGgated non-contrast and contrast CT were not used in automatic registration method.

Manual image registration
We used a dedicated software package for coronary PET image analysis (FusionQuant, Cedars-Sinai Medical Center, Los Angeles, California). 16 PET and CT angiography reconstructions were reoriented, fused, and systematically co-registered in 3 orthogonal planes. 5 18 F-NaF uptake in the sternum, vertebrae, blood pool in the left, and right ventricle served as key points of reference. Subsequently, PET activity in the ascending aorta and aortic arch was aligned with the non-contrast CT AC and final refinement of co-registration was performed according to landmarks around the coronary arteries as well as the aortic and mitral valves ( Figure 1).

Automatic deep learning-based registration
We developed a fully automated PET and CT angiography registration framework using conditional generative adversarial network (GAN) 9 and nonlinear diffeomorphic (demons) registration. 10,11 In the first step, we generated CT images (pseudo-CT) from NAC PET images ( Figure 2). We choose NAC PET for our registration framework as it is immune to potential misregistration of emission data and non-contrast CT AC images which can affect PET AC reconstructions. These pseudo-CTs are in perfect alignment with the original PET images. In the second step, these pseudo-CTs were used to register PET to CT angiography ( Figure 3).
Pseudo-CT generation GANs 8 are a type of deep learning networks which use generative modeling to synthesize realistic images from a given input, consisting of two key components, generator and discriminator. The generator learns the mapping between two types of images, in our case source (PET) and target (non-contrast CT AC), generated on the condition of the target image corresponding to input source image 9 ( Figure 2). The second component is the discriminator which tries to classify the generated images as real or fake (generated from source). These two networks were trained in an adversarial fashion, 8 competing with one another, until the discriminator network was unable to distinguish between the real and generated cases of CT.
In our implementation, the generator used was a modified UNet 17 with skip connections and attention gates. 18 The UNet is a popular encoder-decoder network commonly used in biomedical imaging to learn the information present in the input image and encapsulate it. The input to the generator was a 2D NAC PET slice and the output was the generated pseudo-CT slice. The encoder part consisted of repeated convolution layers each followed by a batch normalization 19 and a rectified linear unit (ReLU) 20 followed by a maxpool operation for dimension reduction. The decoder part upsampled the information learnt in a meaningful representation based on the condition of same slice of real-CT AC image. Skip connections were used with attention gates to connect layers of encoder with corresponding layers of decoder to localize high-level salient features present in PET, often lost in downsampling. Attention gates help to suppress irrelevant background noise without the need of segmenting heart regions and preserve features relevant for CT generation. The output of attention gates was concatenated with the corresponding upsampling block.
The discriminator used was a deep convolutional neural network (CNN) which had two inputs: the pseudo-CT which was the output of the generator and the corresponding CT AC image slice. The network took a 70 9 70 patch of both the inputs and estimated a similarity metric. The discriminator repeated this process for the entire slice, averaging the similarities for each patch, to provide a single probability of whether the pseudo-CT slice was real or fake when compared to the CT AC image slice.
Image preprocessing CT images were resampled and resized to PET dimensions, per patient. For generation of pseudo-CT, each slice of PET and CT was normalized by subtracting with mean and dividing with the voxel range of Gaussian-smoothed patient data. 21 The images were cropped using the largest CT slice by automatically obtaining the bounding box using the boundary of CT scan. The same bounding box was applied to the corresponding PET scan. Slices of output pseudo-CT were combined and overlayed with the real-CT AC image to obtain the background.
Training of pseudo-CT generation The patient population was randomly divided into train and test sets of 139 and 30 patients. The generator model was trained using a combination of adversarial loss for discriminator and a mean-squared loss between input PET and generated pseudo-CT slice. The adversarial loss was optimized to ensure the generator produces realistic pseudo-CT slices and the discriminator was unable to distinguish between real and pseudo-CT slices. The conditional GAN was trained for 250 epochs with a learning rate of 0.0001, an input batch size of 8 NAC PET slices of 256 9 256 voxels. The output pseudo-CT slices were consolidated per patient and used as reference for the final registration.
Nonlinear diffeomorphic registration The pipeline for the end-to-end registration of PET to CT angiography is shown in Figure 3. We first registered the generated pseudo-CT scan to the non-contrast CT AC using a nonlinear diffeomorphic registration algorithm called demons. 10,11 The non-contrast CT AC is subsequently registered rigidly to CT angiography. Both these transforms were applied by integrating the 2 motion vector fields from the first step with the second step to PET thus registering PET to CT angiography using the generated pseudo-CT.

Image quantification
On co-registered PET and CT angiography images, we measured 18 F-NaF uptake within coronary plaque activity using automatically extracted whole-vessel tubular and tortuous 3-dimensional volumes of interest from CT angiography datasets using FusionQuant. 6,7,16 These encompass all the main native epicardial coronary vessels and their immediate surroundings (4-mm radius), enabling both per-vessel and per-patient uptake quantification. Within these volumes of interest, we measured maximum standardized uptake values (SUV max ) (Figure 4) and target-to-background (TBR) valuescalculated by dividing the coronary SUV max by the blood-pool activity measured in the right atrium (mean SUV in cylindrical volumes of interest at the level of the Figure 1. Methodology for manual co-registration of PET and CT angiography. PET and CT angiography were manually co-registered by first aligning the blood pool (left) using key points of reference. The PET activity was subsequently aligned in the aorta to the CT angiography (middle). The final refinement of co-registration was performed according to landmarks around the coronary arteries as well as the aortic and mitral valves. right coronary artery ostium: radius 10 mm and thickness 5 mm). 6,7 We also measured the coronary microcalcification activity (CMA) which quantifies 18 F-NaF activity across the entire coronary vasculature, is highly reproducible, and acts as an independent predictor of myocardial infarction. 3 CMA was defined as the integrated activity in the region where standardized uptake value exceeded the corrected background bloodpool mean standardized uptake value ? 2 SDs. The perpatient CMA was defined as the sum of the per-vessel CMA values. The same analysis was repeated after automatic co-registration utilizing the same region of interests. Translation vectors in each of the 3 directions for observer and automatic registered PET were exported from observer-marked vessels per patient using FusionQuant.

Statistical analysis
Categorical variables are presented as frequencies (percentages) and continuous variables as medians (interquartile range). Variables were compared using a Pearson v 2 statistic for categorical variables and a Wilcoxon rank sum or Kruskal-Wallis test for continuous variables. We assessed the distribution of data with the Shapiro-Wilk test. Figure 2. Generation of pseudo-CT from NAC PET. The GAN consisted of two deep learning networks, the generator and discriminator which were trained together to generate realistic pseudo-CT images. The input to the UNet-based generator was a 2D NAC PET slice and the output was the corresponding pseudo-CT slice. The encoder part of the generator consisted of 4 contracting convolution blocks each. Each convolution block was followed by a 2 9 2 max pool operation for dimension reduction. The decoder block consisted of 3 repeated upsampling blocks each doubling the number of feature channels. Batch normalization normalized inputs from a layer and stabilized the learning process. ReLU is used to set all negative inputs to zero and pass all positive inputs to introduce nonlinearity in the network. Skip connections were used with attention gates connecting the encoder to the decoder. The discriminator with convolution blocks had inputs of generated pseudo-CT and corresponding real-CT AC slice. The output of the network was an averaged similarity between the two inputs. The generated pseudo-CT slices were consolidated per patient and input to the diffeomorphic registration pipeline. AC, attenuation correction; CT, computed tomography; NAC, non-attenuation corrected; PET, positron emission tomography; ReLU, rectified linear unit.
The performance of the proposed method was evaluated through the coefficients of reproducibility (CR) with observer manual registration using Bland-Altman plots with 95% limits of agreement (LOA) and difference in displacement vector fields. Analyses were performed using R studio version 1.4.1717 (RStudio, Boston, MA).
Twenty-five (83%) of the 30 patients in the testing set had poor registration between NAC PET and noncontrast CT AC, according to expert observer. To achieve perfect alignment of the PET and CT images all datasets in the training dataset required adjustments made by the interpreting physician. These were most prominent in the z-axis reflecting the discordance in the diaphragm position-which is a result of breathing (while CT AC data can be acquired during a breath-hold the PET scan last for 30 minutes).
A case example of GAN-based nonlinear diffeomorphic registration in Figure 4 shows similar registration and SUV max in the coronary arteries compared to expert observer. Activity of 88 vessels in the 30 patients, was assessed using TBR max , SUV max , and CMA. TBR max had a coefficient of repeatability (CR) of 0.31, mean bias of 0.92, and narrow limits of agreement (LOA) (95% LOA -0.29 to 0.33) ( Figure 5A). SUV max had excellent CR of 0.26, mean bias of 0, and narrow limits of agreement (95% LOA -0.26 to 0.26) Fig. 3. Overview of nonlinear diffeomorphic registration pipeline. The GAN-generated pseudo-CT from NAC PET was registered to the corresponding non-contrast CT attenuation correction image using diffeomorphic (demons) non-rigid registration (Transform 1). The non-contrast CT AC was then registered rigidly to CT angiography image of the same patient (Transform 2). These two transforms (Transforms 1 and 2) were applied to AC PET image, registering PET to CT angiography automatically. AC, attenuation corrected; CT, computed tomography; NAC, nonattenuation corrected; PET, positron emission tomography.
( Figure 5B). Between observer and GAN-based registration, CMA had CR of 0.57, mean bias of 0.07, and narrow limits of agreement (95% LOA -0.54 to 0.60) ( Figure 6). Difference in displacement motion vectors between GAN and observer-based registration (Figure 7) was 0.8 ± 3.02 mm in the x-direction, 0.68 ± 2.98 mm in the y-direction, and 1.66 ± 3.94 mm in the z-direction. The overall time for the GAN-based registration with analysis was 84 seconds on a CPU workstation and 27.5 seconds on a GPU workstation. The overall registration and analysis time for the experienced observer was 15 ± 2.5 minutes.

DISCUSSION
We propose a fully automated deep learning-based framework to register 18 F-NaF PET to CT angiography images. A conditional GAN was used to synthesize pseudo-CT from coronary NAC PET images. The perfectly registered pseudo-CT provides an input to a nonlinear registration pipeline which transforms PET alignment to match CT angiography. We trained the proposed method with 139 pairs of coronary PET/CT angiography images. The evaluation in a separate cohort of 30 patients demonstrated excellent correlation of TBR max , SUV max and CMA between observer and our method. The proposed method runs automatically in 27.5 seconds, approximately 33 times faster than expert observer and has great potential to streamline timeconsuming manual rigid registration which is necessary for coronary PET/CT angiography data. Our approach is the first to use pseudo-CT generated from GAN for nonlinear diffeomorphic registration of coronary PET and CT angiography images. This development paves the way for more widespread utilization of coronary PET. Since acquisition protocols have been already validated across multiple centers and image analysis has been automated, by providing automatic co-registration an 18 F-NaF CMA uptake value can become available as soon as image reconstruction is completed and CT angiography coronary centerlines are available. 18 F-NaF PET has emerged as a promising noninvasive imaging tool for assessment of active calcification processes across a wide range of cardiovascular conditions. 16,[22][23][24][25][26][27] In coronary artery disease, 18 F-NaF uptake provides an assessment of disease activity and prediction of subsequent disease progression and clinical events. 1,3 Over the past years multiple technical refinements for 18 F-NaF PET have been introduced. [28][29][30][31] Motion correction techniques address heart contractions, tidal breathing, and patient repositioning during the prolonged PET acquisitions. [32][33][34] Novel uptake measures such as CMA enable a patient-level assessment of disease activity which is guided by centerlines derived from contrast-enhanced CT angiography. 6,7 Dedicated software tools, optimized timing of the acquisition, and dedicated reconstruction algorithms further streamline 18 F-NaF coronary imaging. 16,35 In view of all these refinements to date, the need for manual co-registration of the CT angiography and PET datasets remains the last step, which still requires advanced cardiac imaging expertise for image quantification and is associated with subjectivity. Our current study addresses this important aspect of 18 F-NaF coronary PET imaging. By leveraging AI, we were able to develop and test a fully automated tool aligning the CT angiography and PET datasets.
GAN AI methods 8,9 have recently become popular in medical imaging for image-to-image translation tasks. 36 By learning the mapping from one type of the image to the to another, they are often used for denoising, 13,37,38 segmentation, 39,40 low dose to highdose reconstruction 41 , and registration 42-44 tasks. Dong et al 45 have used GANs for generation of CT from PET that has been used for attenuation correction. In this study we propose to use GANs for the challenging task of fully automated coronary PET and CT angiography image registration. Automatic registration of PET and CT angiography is difficult to accomplish due to nonlinear respiratory and cardiac motion displacement between the two modalities and limited anatomical information provided by coronary PET. We leverage the fact that the generated pseudo-CT is perfectly registered to PET, which is the input to GAN, unlike the noncontrast attenuation correction image acquired in the same imaging session. The latter misalignment occurs due to patient motion and the lengthy PET acquisition. By registering the pseudo-CT to the actual CT and then subsequently to CT angiography with nonlinear diffeomorphic registration, these issues are overcome, and we obtain the transform to register PET to CT angiography.
Automatic GAN-based nonlinear diffeomorphic registration of PET and CT angiography can be employed for accurate alignment of images. This method facilitates automated analysis of 18 F-NaF coronary uptake with the user input limited to careful inspection whether extra-coronary activity does not corrupt the coronary 18 F-NaF uptake measurements. In view of the already available tools for quantification of 18 F-NaF activity on a per-vessel and per-patient level, 6,7 by developing automatic PET to CT angiography registration we have paved the way for more widespread use of coronary PET. This development could further simplify the complex processing protocols needed for clinical application of coronary PET imaging.
Several studies have attempted automatic cardiac PET to CT registration. In previous studies, Nakazato et al 46 have evaluated a non-AI-based method to register myocardial perfusion 13 N-ammonia PET to CT angiography. Yu et al 47 proposed an AI-based framework using convolution neural networks to nonrigidly register PET/ CT images. However, 18 F-NaF coronary PET imaging does not visualize the myocardium and so direct PET to CT angiography registration is not feasible. Our proposed solution overcomes the difficulties in image registration of images with different visual appearances.
Our study has limitations. It is a post hoc analysis of single-center data and was acquired on one hybrid PET/ CT scanner. The observer registration was performed rigidly using a summed PET scan registered to the diastolic gate; however, the pseudo-CT was generated using summed NAC PET data and CT angiography was registered nonrigidly using corresponding AC PET scan. The CT AC misregistration may occur between the PET and CT affecting the quality of the PET images. This causes difference in observer registration, which was performed using potentially incorrect attenuation corrected PET and automatic registration, which was performed with NAC pet images. Further studies could be performed utilizing the generation of pseudo-CT for optimizing attenuation correction of the PET signal. However, to date no clinical studies have utilized such corrections. The user correction was limited to rigid translation of the vessel in contrast to automatic nonlinear alignment by our method. Nevertheless, there is currently no other suitable standard to evaluate the misalignment and the vessel-based rigid co-registration is the basis of the current clinical analysis.

CONCLUSION
Pseudo-CT generated by GAN from PET, which are perfectly aligned with PET, can be used to facilitate rapid and fully automated nonlinear diffeomorphic registration of PET and CT angiography. We show that our method has excellent agreement and is approximately 33 times faster as compared to expert observer registration.