Comparison of Imaging Biomarkers in the Alzheimer Disease Neuroimaging Initiative and the Mayo Clinic Study of Aging FREE

Imaging plays an important role in the study of Alzheimer disease (AD). Imaging biomarkers can track disease progression,¹ detect changes early in the phase of mild cognitive impairment (MCI),^2- 3 and help predict which subjects may later develop AD.^4- 5 Imaging measures have become common outcome measures in clinical treatment trials because they may reduce sample size.^6- 7 Increasing interest in the use of imaging in clinical trials led to the development of the Alzheimer Disease Neuroimaging Initiative (ADNI), which aimed to improve methods for clinical trials and validate imaging and other biomarkers.^8- 9 The ADNI is an observational study of MCI and AD that used identical recruitment mechanisms as typical trials, including advertising and recruitment from memory clinics. Therefore, the ADNI is based on a highly selected convenience sample. Because ADNI data are freely available, a large number of studies are published each year using these data. However, it is unclear to what extent subjects recruited through these mechanisms are representative of the general population and hence whether results are generalizable.

We aimed to determine whether imaging measures would differ in ADNI participants compared with the population-based cohort of the Mayo Clinic Study of Aging (MCSA). We assessed hippocampal volume and rates of decline in hippocampal volume because they are established and widely studied biomarkers of AD.^6,10- 11 Because results could be influenced by differences in inclusion characteristics and demographics, we compared the cohorts before and after matching for specific demographic and cognitive features.

Subjects with a clinical diagnosis of amnestic MCI (aMCI) and cognitively normal (CN) subjects who had been recruited into the MCSA (and had agreed to undergo magnetic resonance imaging [MRI]) or the ADNI underwent analysis.

The MCSA is a longitudinal epidemiologic study of normal aging and MCI in Olmsted County, Minnesota. The recruitment mechanisms have been reported in detail previously.¹² Briefly, all Olmsted County residents aged 70 to 89 years on October 1, 2004, were identified using the medical records linkage system of the Rochester Epidemiology Project.^13- 14 The population was also resampled in 2008 and 2009 to replenish the cohort. Subjects were randomly selected from this enumeration. Subjects received a letter of invitation giving them the opportunity to refuse participation by returning a letter of refusal. Subjects who did not return the letter then received a follow-up telephone call inviting them to participate. Magnetic resonance imaging was performed in all subjects who agreed to participate and did not have any contraindications to MRI. S ubjects who underwent imaging in the MCSA have demographic characteristics very similar to those who did not undergo imaging (Table 1). Subjects were characterized as CN by consensus^12,15 and when their age-adjusted neuropsychological test scores were consistent with normative data developed in this community.¹⁶ Diagnostic criteria for MCI were as follows¹⁷: (1) cognitive concern by the subject, an informant, a nurse, or a physician; (2) impairment in 1 or more of the 4 cognitive domains (from the cognitive battery); (3) essentially normal functional activities (using the Clinical Dementia Rating Scale¹⁸ and Functional Activities Questionnaire); and (4) absence of dementia (defined by the DSM-IV).¹⁹ Subjects were categorized as having aMCI if their memory was impaired. The diagnosis of aMCI was made on clinical grounds without the use of rigid cutoffs on psychometric scores.

The ADNI is a longitudinal multisite observational study of CN subjects and subjects with aMCI and AD (http:// www.ADNI-info.org).⁸ Subjects were recruited using local AD research centers, memory clinics, newspaper advertisements, radio, and other public media campaigns. Diagnostic criteria for the ADNI were largely the same as for the MCSA. Criteria for CN subjects included (1) Mini-Mental State Examination (MMSE)²⁰ score between 24 and 30 inclusive; (2) no memory complaints; (3) objective memory performance in the normal range; and (4) a Clinical Dementia Rating Scale score of 0 and memory box score of 0. Diagnostic criteria for aMCI included (1) memory complaint verified by an informant; (2) objective memory impairment measured by the educational level–adjusted score on the Wechsler Memory Scale–Revised, Logical Memory II; (3) MMSE scores between 24 and 30 inclusive; (4) a Clinical Dementia Rating Scale score of 0.5 and memory box score of at least 0.5; and (5) preservation of general cognition and functional activities of daily living. Subjects enrolled in the ADNI were aged 55 to 90 years. The ADNI participants with AD were not included in our analysis because the MCSA does not follow up participants with AD.

Informed consent was obtained from all subjects. The MCSA was approved by the Mayo Clinic institutional review board, and the ADNI was approved by the institutional review board at each site.

We selected 2 samples of subjects. The first was a random sample of all available MCSA and ADNI subjects, and the second sample applied matching for demographic and cognitive variables. Cross-sectional and longitudinal samples were selected. The first available MRI was used for the cross-sectional analysis and as baseline for the longitudinal analysis. Two serial MRIs were used for the longitudinal analysis for each subject. Scan interval was approximately 12 months for the ADNI and 15 months for the MCSA (the routine follow-up interval in the MCSA).

For the cross-sectional analysis, the total number of available CN subjects was 229 in the ADNI and 1283 in the MCSA. The total number of aMCI subjects available was 397 in the ADNI and 179 in the MCSA. To obtain comparable sample sizes between the ADNI and MCSA, we took a simple random sample of the MCSA CN subjects, resulting in 229 subjects. Similarly, we took a simple random sample of the ADNI aMCI subjects, resulting in 179 subjects. Because of the random subsampling strategy, the samples used for our analyses were representative (within sampling error) of the parent cohorts from which they were drawn (eTable). For the longitudinal analysis, 206 ADNI CN subjects, 686 MCSA CN subjects, 347 ADNI aMCI subjects, and 92 MCSA aMCI subjects had serial scans available for analysis. Again, to obtain comparable group sizes, we took a random sample of the MCSA CN subjects and ADNI aMCI subjects, resulting in 206 MCSA CN subjects and 92 ADNI aMCI subjects.

In sample 2, the ADNI and MCSA subjects were frequency matched by age, sex, educational level, apolipoprotein E (APOE) genotype, and MMSE score. All variables were dichotomized into strata by age (70-79 and 80-90 years), sex (men and women), educational level (6-13 and 14-20 years), and MMSE score (24-28 and 29-30 for CN subjects; 22-25 and 26-30 for aMCI subjects). The ADNI and MCSA subjects were matched with a one-to-one frequency by taking a random sample within each of the 32 strata of the larger study group to match the number of subjects in the smaller study group. We matched the CN and aMCI subjects separately. Subjects who could not be matched were excluded. For the cross-sectional analysis, 212 CN subjects and 97 aMCI subjects were selected for the ADNI and the MCSA. For the longitudinal analysis, 191 CN subjects and 65 aMCI subjects were selected for the ADNI and the MCSA.

Subject demographics for the 2 cross-sectional and longitudinal samples are shown in Tables 2, 3, 4, and 5. The samples used for analysis differ slightly from those reported in the preceding paragraph because some subjects were excluded owing to poor quality of the imaging.

Protocols for MRI acquisition were very similar for MCSA and ADNI subjects, although MCSA subjects underwent 3.0-T and ADNI subjects underwent 1.5-T MRI. The ADNI collects 1.5-T MRIs in all subjects and 3.0-T images in only 25% of the sample; therefore, ADNI 1.5-T MRI scans were used for this study. To ensure that field strength did not bias our results, we compared hippocampal volumes at 1.5 T and 3.0 T in ADNI subjects who underwent scanning at both field strengths. Similar to findings of a previous study,²¹ hippocampal measurements were comparable across field strengths (Figure 1).

The MCSA subjects underwent imaging with a 3-dimensional, magnetization-prepared, rapid-acquisition gradient echo sequence developed at the Mayo Clinic for the ADNI.⁹ The sequence was acquired in the sagittal plane with a repetition time of 2300 milliseconds, echo time of 3 milliseconds, inversion time of 900 milliseconds, flip angle of 8°, 26-cm field of view, and a 256 × 256 in-plane matrix with a phase field of view of 0.94 and section thickness of 1.2 mm. The ADNI is a multisite study, and minor variations in the MRI protocol are based on the specific hardware/software configuration on each scanner. The nominal variables in the sagittal plane of the ADNI magnetization-prepared rapid-acquisition gradient echo included repetition time of 2400 milliseconds, echo time of 3 milliseconds, inversion time of 1000 milliseconds, flip angle of 8°, 24-cm field of view, a 192 × 192 in-plane matrix, and a section thickness of 1.2 mm.⁹

All images were corrected for gradient nonlinearity²² and intensity inhomogeneity.²³ Hippocampal volumes were measured using Freesurfer software, version 4.5.0 (Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital).²⁴ The cross-sectional analysis pipeline was used to calculate hippocampal volumes for the cross-sectional sample, and the longitudinal analysis pipeline was used to assess rates of hippocampal change for the longitudinal sample. Hippocampal measurements calculated using Freesurfer software have been previously validated against manual measurements.²⁵ Total intracranial volumes (TIVs) were measured using an algorithm developed in-house.²⁶

Statistical analyses were performed in R software, version 2.11.0 (http://www.r-project.org), and tests of statistical significance were conducted at the 2-sided α level of .05. For the cross-sectional analysis, we used hippocampal volume (in cubic centimeters) adjusted for TIV (in cubic centimeters). We fit a linear regression model of hippocampal volume (y) vs TIV (x) in all ADNI and MCSA CN subjects with available data (n = 1480) and then used the intercept (b₀) and slope (b₁) estimates from the model to calculate hippocampal volume adjusted for TIV (represented as HVa) as a residual using the following equation:

HVa = Hippocampal Volume − (b₀ + b₁ × TIV).

For the longitudinal analysis, the annual percentage of decline in hippocampal volume was calculated using unadjusted hippocampal volumes (represented as HV) in the following equation:

(Follow-up HV − Baseline HV)/(Baseline HV × Years Between Scans) × 100.

Wilcoxon rank sum and Mann-Whitney tests were used to test differences in continuous measures between the ADNI and MCSA groups, and χ² tests with continuity correction or Fisher exact test were used to test differences in categorical variables. We summarized group differences in imaging measures using the probabilistic index (corresponding to the area under the receiver operating characteristic curve).²⁷ The probabilistic index is a nonparametric estimate of groupwise differences or discrimination that measures the probability that the value from a randomly selected subject in one group is higher than the value from a randomly selected subject in the other group. A probabilistic index of 0.50 (or 50%) indicates no difference across groups.

Differences in demographic features across the MCSA and ADNI were similar for cross-sectional and longitudinal cohorts (Tables 2, 3, 4, and 5). In sample 1, MCSA subjects (aMCI and CN) were older and less educated and had worse performance on the MMSE than did ADNI subjects. The MCSA aMCI subjects included a lower proportion of APOE ϵ4 carriers than did ADNI subjects. No differences were observed in sex, educational level, or APOE genotype between the MCSA and ADNI subjects in sample 2. Despite frequency matching, age (cross-sectional sample only) and MMSE in the CN subjects still differed across the cohorts, although the median and interquartile ranges were similar. The ADNI CN subjects had a greater proportion of family history of AD and of racial and ethnic minorities across all samples, with a similar trend for aMCI in the cross-sectional sample.

In sample 1, hippocampal volume adjusted for TIV was significantly smaller in the MCSA CN subjects compared with the ADNI CN subjects, with no differences between the groups for the aMCI subjects (Figure 2A). After matching for age, sex, educational level, APOE genotype, and MMSE score in sample 2, no differences in hippocampal volume adjusted for TIV were observed between the MCSA and ADNI in the CN or aMCI subjects (Figure 2B).

In sample 1, the annual percentage of decline in hippocampal volume was greater in the ADNI compared with the MCSA for aMCI and CN subjects (Figure 3A). After matching for age, sex, educational level, APOE genotype, and MMSE score in sample 2, these differences across the ADNI and MCSA were still observed (Figure 3B).

This study highlights demographic differences in subjects recruited into the convenience-sample ADNI cohort compared with subjects recruited into the population-based MCSA cohort and demonstrates that imaging biomarkers from these 2 different recruitment mechanisms differ.

The most striking difference was that rates of decline in hippocampal volume were greater in the ADNI compared with the MCSA for both CN and aMCI subjects. This difference was observed even after matching for key demographic and cognitive variables. Increased rates of decline in hippocampal volume in CN subjects predict a faster rate of progression to dementia,²⁸ suggesting that the ADNI CN population includes a larger proportion of subjects on the path to AD dementia. Although it was somewhat unexpected that the proportion of APOE ϵ4 carriers was not higher among the ADNI CN subjects, our findings are consistent with the unusually high proportion (50%) of ADNI control subjects who showed amyloid pathologic changes as measured by Pittsburgh Compound B.²⁹ By contrast, the proportion of MCSA controls with positive findings for Pittsburgh Compound B was only 30%.³⁰ The pathologic diagnosis of AD was also more common in controls from a clinic vs a community setting in a previous study.³¹ The ADNI CN subjects were more highly educated than the MCSA CN subjects; therefore, cognitive reserve mechanisms may have protected them from clinical decline even though they are on a steeper downward trajectory of brain atrophy. Similarly, the higher rates of atrophy suggest that the ADNI aMCI group consists of a higher proportion of subjects with more aggressive disease than the MCSA aMCI group. Indeed, the ADNI aMCI subjects had a higher proportion of APOE ϵ4 carriers than those in the MCSA in sample 1. Again, the ADNI aMCI subjects had more education than MCSA aMCI subjects, suggesting that cognitive reserve mechanisms may have protected them from decline on the MMSE and progression to a clinical diagnosis of AD.

We hypothesize that this bias in the ADNI is a result of the recruitment mechanism. We can speculate that CN subjects who are worried about their cognition would be more likely to attend memory clinics and be more motivated to answer advertisements for the study. The CN and aMCI subjects with higher levels of education are also more likely to seek medical help at a memory clinic and become involved in observational studies. These highly educated subjects could have a more aggressive underlying disease but are able to compensate cognitively. Amnestic MCI subjects recruited through a population-based study are less likely to have sought medical care at a memory clinic and may have a broader spectrum of cognitive function. In addition, an important motivator for participation in the ADNI and other convenience studies could be the presence of a family history of dementia. Indeed, ADNI subjects had a higher proportion of family history compared with MCSA subjects. Although one may assume that similar biases would be observed in the MCSA subjects who agreed to undergo imaging, we have demonstrated that this is not the case, likely because less effort was required to agree to undergo imaging than to seek out participation in the ADNI.

The clinical inclusion criteria for CN and aMCI subjects differed slightly across the 2 cohorts. A diagnosis of CN in the MCSA was made by multidisciplinary consensus, which may be more conservative than the method used in the ADNI. Similarly, the diagnosis of aMCI in the MCSA is based on clinical grounds, whereas the ADNI relied more on a specific cutoff point on a memory test. The ADNI approach is likely to result in the recruitment of more impaired subjects. The reason this is not reflected in the MMSE scores could be that having a higher educational level provides a cognitive reserve, and the MMSE may be insensitive to subtle cognitive impairment. The ADNI also recruited younger subjects than the MCSA, which could also have resulted in the recruitment of subjects with more aggressive disease. Rates of atrophy have been found to be greater in younger aMCI subjects,³² possibly because they have purer and hence more aggressive AD pathologic changes compared with older subjects. Older subjects are more likely to have a mixture of pathologic findings,³³ including cerebrovascular disease.^34- 35 However, the trend for greater APOE ϵ4 carrier frequency, younger age, and higher educational levels in convenience samples compared with population-based samples has been observed in other cohorts,^36- 39 suggesting that this bias may be due to the general recruitment mechanism rather than the specific inclusion criteria used in the ADNI. Our findings suggest that CN and aMCI subjects in the ADNI are not representative of the general population and that subjects included in future preclinical prevention trials using the same recruitment mechanisms will also not be representative of the population. Finally, our results indicate that even rigorous demographic matching efforts are insufficient to correct for the selection bias.

The only difference observed in baseline hippocampal volumes between the ADNI and MCSA was in the CN subjects in sample 1, with larger hippocampal volumes observed in the ADNI. This difference is likely being driven by the younger age of the ADNI cohort because hippocampal volume has been shown to decrease with age.⁴⁰ After matching for demographic features, no differences in hippocampal volume were observed across cohorts. Cross-sectional hippocampal volumes also did not differ across the ADNI and MCSA within the aMCI subjects in sample 1 despite the observed differences in age, educational level, APOE genotype, and MMSE score. This finding could suggest that rates of decline in hippocampal volume are more sensitive markers of incident AD than cross-sectional hippocampal volume, perhaps because of the large degree of intersubject variability in hippocampal volume. Total intracranial volume also differed between the MCSA and ADNI. We suspect that MCSA subjects have larger TIVs because of the northern European heritage of many Minnesotan residents and the link between these nationalities and tall height.⁴¹

The strengths of this study include the large numbers of subjects and the use of 2 samples with and without restrictive correction for major demographic or cognitive confounders. A limitation, however, is that, although matching was performed on the major demographic factors, it may not eliminate other potential differences, such as other comorbidities, medication use, family history, and race and ethnicity, that may influence the imaging findings. The ADNI included a higher proportion of ethnic and racial minorities than did the MCSA. The MCSA and ADNI cohorts underwent imaging at different field strengths; however, we demonstrated excellent agreement between hippocampal volumes measured across field strengths (Figure 1). Scan intervals also differed between the ADNI and MCSA, although we adjusted for these differences. Although atrophy rates have been shown to accelerate over time in AD,³² the trajectory of change is likely to be approximately linear during these relatively short intervals. Finally, although the MCSA is a population-based study, some inherent participation biases may exist,¹² as is the case with any survey. However, the MCSA is representative of Olmsted County in Minnesota and of white individuals in the United States in general. The incidence of MCI and the demographic predictors of incident MCI in the MCSA are also similar to those reported in other population-based studies,^42- 44 including studies that have assessed other racial groups.⁴⁵

Overall, our findings show that subjects recruited into the ADNI are not representative of the general population and instead more closely resemble clinical populations. The imaging findings all point toward the ADNI including more CN subjects who are on the path to AD dementia and more aMCI subjects who have a pure and aggressive disease phenotype. Therefore, convenience clinical series may be limited by selection biases. These findings have important implications for the design of future treatment trials. If studies that assess power calculations and sample size estimates are performed in biased convenience samples, the high rates of atrophy will lead to smaller-than-appropriate sample size estimates, and therefore trials could be underpowered to detect treatment effects in the population. In addition, treatment trials that use a convenience sample will include a higher proportion of subjects with a pure and aggressive disease and hence are more likely to detect a treatment effect. However, the magnitude of the treatment effect is likely to be less than expected when the treatment is applied to an unbiased population, in which subjects are less likely to have pure AD. Care should also be taken when interpreting imaging studies from convenience samples such as the ADNI. Biomarkers identified from these highly selected convenience samples may not perfectly translate to the general population, and the findings will need to be validated in a population-based sample. This implication will be particularly important for studies seeking to validate new diagnostic criteria for AD in its various stages, in which imaging biomarkers play an important role.