A Neuropsychological Test Battery for Use in Alzheimer Disease Clinical Trials FREE

The Food and Drug Administration and the European Medicines Authority require Alzheimer disease (AD) treatments to demonstrate efficacy through cognitive outcome measures.¹ In practice, this requirement has most often been met by including the cognitive subscale of the Alzheimer's Disease Assessment Scale (ADAS-Cog) in clinical trials. However, this instrument has been widely recognized as suboptimal, particularly with respect to the measurement of areas of cognition that are compromised in patients with AD, including attention, planning, working memory, and executive function.^{2 - 3} To remedy this deficiency, further subtests have been added to the ADAS-Cog, including a delayed word recall task, a maze task,⁴ digit cancellation,⁴ and a subjective judgment included from the ADAS noncognitive subscale, which is designed to assess concentration and distractibility. However, the utility and sensitivity of these measures are unproved and the standard ADAS-Cog is still the most commonly used cognitive outcome measure in AD clinical trials.⁵ This is unfortunate, because the ability of pharmacological interventions to affect areas of cognition, such as executive function, offers much promise.

A further area of concern with the use of the ADAS-Cog is its lack of sensitivity for measuring cognitive change in patients with relatively high Mini-Mental State Examination (MMSE) scores. In planning clinical drug trials for patients with AD, it is assumed that patients receiving placebo will decline on the ADAS-Cog by approximately 7 points per year. However, several recently reported trials have suggested this may not be true. Wilcock et al⁶ reported that the 6-month placebo group decline on the ADAS-Cog in patients scoring 18 or higher on the MMSE was only slightly more than 1 point. Other recent studies have also reported similar modest changes in ADAS-Cog scores over time. For example, Seltzer et al⁷ report that the AD placebo group (mean baseline MMSE score, 24.3; SD, 1.3) in a 24-week study of donepezil hydrochloride showed less than a 1-point change. In addition, Kurz et al,⁸ in a study that examined treatment effects of rivastigmine by disease stage in AD, reported that the placebo group of 288 patients scoring 22 or higher on the MMSE at baseline changed by just 1.13 ADAS-Cog points in 26 weeks. The ADAS-Cog change from baseline scores in patients with mild cognitive impairment has been even more modest.^{7 ,9} These findings suggest that it would be prudent to identify cognitive outcome measures that are more sensitive to small changes in function in studies of patients with mild AD (ie, MMSE score > 20) and mild cognitive impairment.

In a previously reported, randomized, double-blind, placebo-controlled trial of AN1792(QS-21) (synthetic β-amyloid plus an adjuvant),¹⁰ a neuropsychological test battery (NTB), a novel combination of 6 well-known, validated, cognitive tests, yielding 9 measures of patient performance, was used. Scores for each of the individual NTB components from baseline to month 12 in the antibody responder (anti-AN1792 IgG [total] serum titer ≥ 1:2200 at any postinjection visit) and placebo groups were standardized into z scores and converted into an overall composite z score across all NTB components. The study found less worsening in the change from baseline NTB z scores at month 12 in the antibody responder group compared with z scores in the placebo group.¹⁰ The mean NTB z score improvement from baseline to month 12 was 0.03 (SD, 0.37) in the antibody responder group, while the NTB decline was 0.20 (SD, 0.45) in the placebo group. There were no significant differences in the decline on other traditional AD assessments (eg, ADAS-Cog, Disability Assessment Scale for Dementia [DAD], Clinical Dementia Rating Sum of Boxes [CDR-SOB], and Alzheimer Disease Cooperative Study Clinical Global Impression of Change [ADCS-CGIC]) between the placebo and responder groups. Concentration-response analysis indicated significant relationships between geometric mean titer and the NTB z score; greater improvements from baseline in the NTB z scores were associated with higher IgG antibody titers.¹⁰

The NTB components were selected because they indexed memory and/or executive function. Tests of visual and verbal memory were selected for inclusion. Delayed verbal recall has been repeatedly shown to be a sensitive measure of cognitive decline in patients with AD.¹¹ To maximize the chances of capturing delayed verbal memory effects, the Rey Auditory Verbal Learning Test was included in the NTB. However, to reduce the number of outcome variables to a manageable number, we combined total words recalled at delayed testing with total number of words correctly recognized. No correction for false-positive recognition was made in this analysis. Impaired executive function, defined as the inability to plan, organize, set, and adapt current and past knowledge to future behavior, is one of the core features of AD. In one study,¹² 64% of patients had executive dysfunction and these patients performed worse on tests of cognition, dementia severity, and activities of daily living compared with patients with normal executive function. Neuropsychological tests are typically classified according to the primary area of cognition believed to be necessary for successful completion of the task. However, this does not suggest that other cognitive skills are not required, because even a word recall task will require satisfactory attentional, working memory, language, and other skills.¹³ The NTB was also designed to incorporate tests of paired associative learning. Impairment on these tasks can be induced through scopolamine hydrobromide administration,¹⁴ and successful paired associative learning task performance is known to rely on the integrity of medial temporal lobe structures. These brain areas are reported to be among the first to experience degeneration in AD.¹⁵ This class of test has been shown to be a sensitive measure of cognitive decline in patients with AD.¹⁶

In this study, we performed a factor analysis of the individual NTB components to determine whether a “memory” factor and an “executive function” factor could be identified. We also characterized the internal consistency and test-retest reliability of the NTB and, to address the issue of clinical relevance, correlated the NTB with cognitive and functional measures. The concurrent validity of the NTB was also assessed by correlating the NTB z scores with scores from the ADAS-Cog, MMSE, and CDR-SOB. Finally, the sensitivity to change of the NTB was assessed in relation to MMSE baseline performance.

Eligible patients were aged 50 to 85 years and met the criteria for a diagnosis of probable AD as defined by the National Institute of Neurological and Communicative Disorders and Stroke–Alzheimer's Disease and Related Disorders Association¹⁷ ; magnetic resonance imaging of the brain was conducted to rule out other brain pathology to support the clinical diagnosis of AD. Additional inclusion and exclusion criteria and other details of patient disposition are described elsewhere.¹⁰

The NTB and other cognitive and functional measures were evaluated using data from a randomized, placebo-controlled, double-blind trial conducted at 28 centers in the United States and Europe between September 24, 2001, and December 18, 2002. A total of 372 patients with mild to moderate AD were randomized in a double-blind manner to receive treatment with AN1792, 225 μg (Elan Pharmaceuticals), and QS-21, 50 μg (Antigenics, Framingham, Massachusetts), containing 0.4% polysorbate 80 or isotonic sodium chloride solution in a 4:1 ratio. However, dosing was discontinued after reports of encephalitis in 6.0% of AN1792(QS-21)-treated patients.

The NTB is composed of “paper-and-pencil” assessments and takes on average approximately 40 minutes to complete. The NTB consists of 9 validated components: Wechsler Memory Scale visual immediate (score range, 0-18), Wechsler Memory Scale verbal immediate (score range, 0-24), Rey Auditory Verbal Learning Test (RAVLT) immediate (score range, 0-105), Wechsler Memory Digit Span (score range, 0-24), Controlled Word Association Test (COWAT), Category Fluency Test (CFT), Wechsler Memory Scale visual delayed (score range, 0-6), Wechsler Memory Scale verbal delayed (score range, 0-8), and RAVLT delayed (score range, 0-30).¹⁸ The RAVLT delayed measure is composed of delayed recall and recognition performance components that are summed to yield a score ranging from 0 to 30.

Three of the NTB tasks (visual paired associates, verbal paired associates, and digit span) were drawn from the Wechsler Memory Scale Revised.¹⁸ In addition to the digit span task, we also selected 2 tests of verbal fluency, the COWAT and the CFT. The former task examines the patient's phonological fluency skills and the latter examines the patient's semantic fluency (in this case, the ability to generate exemplars from the category “animals”). Key psychometric properties, such as the relative lack of ceiling and floor effects, sensitivity to cognitive decline in patients with mild AD, and robust test-retest reliability, have been previously established.^{13 ,19} Test-retest reliability for component tasks has been shown to range from 0.68 to 0.91.¹³ Reference to standard neuropsychological texts, such as that of Lezak,¹³ suggested that all 9 tasks were likely to be familiar to investigator site psychometricians.

The cognitive scales were administered by 2 blinded raters: the first administered the ADAS-Cog, DAD, and NTB, and the second administered the ADCS-CGIC and the CDR-SOB.

z Scores were calculated for each of the 9 NTB components using the sample baseline mean and standard deviation for all patients with baseline scores. The resultant z scores were averaged to obtain the NTB z score. An executive function z score was obtained by averaging the z scores from NTB components measuring executive function (CFT, COWAT, and Wechsler Memory Digit Span). The remaining 6 components, which measure memory, were averaged to obtain a memory z score. The NTB z scores were calculated for patients who had no missing data for at least 6 of the 9 component tests.

Using only the data from the placebo group, the test-retest reliability of the NTB was characterized by correlating the baseline NTB z score with the NTB scores obtained at months 6 and 12. The study was not designed to formally examine the test-retest reliability of the NTB and, therefore, although placebo group NTB z scores declined on average throughout the study, the 6- and 12-month z score correlations with the baseline z score provide the best available measure of test-retest reliability. Baseline scores on the components of the NTB were used to calculate the Cronbach α, a measure of the internal consistency of the NTB.

To characterize the concurrent validity of the NTB, baseline, month 6, and month 12 NTB z scores for all study participants were correlated with baseline, month 6, and month 12 scores on the MMSE, ADAS-Cog, CDR-SOB, and DAD. Furthermore, the NTB 6- and 12-month z score changes from baseline were correlated with available 6- and 12-month changes from baseline for the MMSE, ADAS-Cog, CDR-SOB, ADCS-CGIC, and DAD. All correlations were calculated using a Pearson product moment correlation coefficient.

A factor analysis was conducted with the 9 components of the NTB at baseline to determine whether a memory factor and an executive function factor could be identified. The Pearson product moment correlations between the baseline scores for each of the 9 NTB components were calculated and used in the factor analysis. The factor analysis was conducted using the principal components method of factor extraction with a varimax rotation.

Finally, the sensitivity to change of the NTB as a function of MMSE performance was determined by comparing the average 12-month NTB and ADAS-Cog changes from baseline for placebo group patients with high and low MMSE scores. A z score was also calculated for the ADAS-Cog using the sample baseline mean and standard deviation for all patients with baseline scores to allow a comparison between the average 12-month NTB and ADAS-Cog z score changes from baseline for placebo group patients with high and low MMSE scores.

A total of 372 patients were randomly assigned to receive study treatment, 300 received AN1792(QS-21) and 72 received placebo. Included patients were approximately 72 years old, with a slightly greater preponderance of women (55.1%), and had a mean MMSE score of 20. Of the patients, 83.3% were at steady-state use of acetylcholinesterase inhibitors at study enrollment, and inclusion criteria required that acetylcholinesterase inhibitor dose be unchanged for the duration of the study. No patient was withdrawn from the study because of changes in the doses of acetylcholinesterase inhibitor medication.

The NTB z scores for patients receiving placebo at baseline were correlated with their z scores at 6 and 12 months to assess test-retest reliability. The NTB had high test-retest reliability at 6 (r = 0.92; 95% confidence interval, 0.87-0.95) (Figure 1) and 12 (r = 0.88; 95% confidence interval, 0.79-0.93) months. The NTB also had high levels of internal consistency, with a Cronbach α of 0.84.

There was a correlation between the NTB z score and scores on traditional measures of cognition (MMSE and ADAS-Cog) and function (DAD and CDR-SOB) (P < .001) at baseline, month 6, and month 12 for patients receiving placebo and for all patients included in the study. Given the negligible differences between correlations calculated for all patients and for patients receiving placebo, only the correlations for all patients are presented in Table 1. In particular, correlations with the ADAS-Cog are strong at all 3 time points. The similarity of the correlations on each measure at baseline, month 6, and month 12 supports the assertion that the NTB has strong test-retest reliability. The correlations between the changes from baseline on the NTB and the changes from baseline for cognitive and functional measures for all patients were also different from 0 (P < .001) (Table 2). These correlations were smaller than correlations between scores obtained for each measure at baseline and 6 and 12 months (Table 1).

The factor analysis yielded 2 factors with eigenvalues greater than 1.0, indicating the presence of 2 factors. The factor loadings for both factors are given in Table 3 and are shown graphically in Figure 2. The factors for the Wechsler Memory Digit Span, COWAT, CFT, and RAVLT immediate tests load heavily on factor 2 (factor loadings, > 0.6). The rest of the NTB tests load primarily on factor 1. Of the 4 tests that primarily load on factor 2, 3 measure executive function, and all of the tests that primarily load on factor 1 measure memory. Thus, factor 1 can be viewed as a memory factor and factor 2 can be viewed as an executive function factor.

Finally, the average 12-month NTB and ADAS-Cog changes from baseline were examined for patients with mild and moderate AD receiving placebo. Patients with mild AD had MMSE scores of 21 to 26 at baseline, and patients with moderate AD had MMSE scores of 15 to 20 at baseline (Figure 3). The mean (SD) NTB change from baseline for patients with a high MMSE score (−0.21 [0.47]) was similar to that for patients with a low MMSE score (−0.18 [0.42]). In contrast, the mean ADAS-Cog change from baseline to month 12 in the placebo group was different for patients with high and low MMSE scores. Patients with high baseline MMSE scores declined by a mean (SD) of 0.63 (4.32) ADAS-Cog points, and patients with low baseline MMSE scores and more severe AD declined by a mean (SD) of 4.92 (7.61) points on the ADAS-Cog. As expected, the mean ADAS-Cog z score change from baseline to month 12 was also different for patients with high and low MMSE scores. Patients with high baseline MMSE scores declined by a mean (SD) of 0.07 (0.46) ADAS-Cog z score points, and patients with low baseline MMSE scores with more severe manifestations of AD declined by a mean (SD) of 0.52 (0.80) ADAS-Cog z score points.

Knowing the limitations of the ADAS-Cog, the primary aim of including the NTB in the AN1792(QS-21) phase 2 study¹⁰ was to establish a sensitive easy-to-administer test battery to assess cognitive change in patients with mild to moderate AD. The NTB z score composite assessing cognitive change in patients in the antibody responder group vs the placebo group yielded statistically significant evidence of reduced impairment in the treated group. This result was in contrast to the analyses conducted with the ADAS-Cog and other functional and cognitive measures, which did not find evidence of reduced impairment in the treatment group. Furthermore, a high correlation was found between performance on the NTB and traditional measures at baseline, month 6, and month 12, suggesting that the NTB exhibits high concurrent validity vs measures such as the ADAS-Cog and MMSE. In addition, we also observed statistically significant correlations with functional measures, such as the CDR-SOB and DAD. These correlations suggest that these scales predominately measure the same constructs. The correlations between the NTB change from baseline at 6 and 12 months and the other functional and cognitive measures were all different from 0.

Good test-retest reliability was a key requirement of the position paper published by the International Working Group on Harmonization of Dementia Drug Guidelines.² Further requirements included validity (the instrument must measure the intended disease-relevant cognitive functions), reliability (test-retest, also interrater and intrarater if scoring is subjective), appropriate sensitivity range (the absence of ceiling and floor effects), longitudinal data (information should be available on expected change during the trial), practice effects (determines the need for practice sessions before starting a trial), and equivalent forms (for repeated testing during the trial). Of these requirements, the most difficult to satisfy when selecting from the available cognitive tests are the requirements for practice effects and equivalent forms. Standard texts, such as that of Lezak,¹³ reveal that evidence regarding practice effects and the availability of genuinely equivalent cognitive tests is difficult to find. While the guidelines of Ferris et al² represent excellent guidance, it could be argued that cognitive tests with high levels of test-retest reliability are sufficiently psychometrically robust to preclude the necessity for parallel forms.

The NTB was initially designed to emphasize assessing cognition in the domains of memory and executive function. When all 9 NTB components were subjected to factor analysis, the model yielded evidence to support 2 clear factors. One factor is best characterized as memory, and the other factor represents executive function. However, while the NTB was designed to emphasize the memory and executive function domains, it also provides an index of global cognitive function by drawing on the many cortical areas required to support language (COWAT and CFT), attention (digit span forward), visual perception (Wechsler Memory Scale visual), verbal memory (Wechsler Memory Scale verbal and RAVLT), working memory (digit span backward), and list learning (RAVLT).

The outcome measures that contributed to the executive function factor were all derived from tests generally acknowledged²⁰ to index this cognitive domain. Further interpretation of the executive function factor is challenging largely because of the lack of a clear definition of the term. There are a number of definitions, and while some “family resemblance” between them is evident, some are couched in neuroanatomical terms, typically with reference to the prefrontal cortex,²¹ while others are more functional. An alternative view is that all cognitive tasks require some contribution from executive function; what varies from task to task is the extent to which this resource is taxed. Tasks with demanding working memory components seem to require significant contributions from executive function. Consequently, those tasks that require planning, the development of strategy, or the online monitoring of task components are all likely to make robust demands on executive function resources. Thus, given the need to search semantic memory for appropriate items in the NTB 2-word fluency tasks (CFT and COWAT), and the need to ensure that words are not repeated, it is not surprising that performance on these tests loads on the putative executive function factor. The need to maintain and order items in working memory would also explain why digit span loads on the executive function factor. The same need to utilize working memory when attempting to hold in memory the 15 items of the RAVLT could also account for why this metric seems to load on both factors. We suspect that executive function could be profitably indexed using other putative tests of this resource. Further studies might usefully use measures that index executive function through assessing set-shifting abilities and abstraction.

The measures of cognition that compose the NTB were selected to demonstrate sensitivity to cognitive deterioration and to exhibit robust psychometric properties. The NTB performed well, exhibiting a test-retest reliability correlation coefficient of 0.92 at 6 months and 0.88 at 12 months. This is among the higher levels of reliability for cognitive measures and suggests that the NTB is a highly reliable index of cognitive function and that it is well suited for repeated assessment. An analysis of the NTB's internal consistency also yielded satisfactory levels of performance. The observed Cronbach α of 0.84 represents high levels of consistency, exceeding the desirable threshold of 0.80 proposed by Kline.²²

The sensitivity of the NTB z score was compared with traditional measures, including the ADAS-Cog. The NTB seems to be sensitive to cognitive change independently of MMSE status when stratified by baseline score. The NTB showed similar decline in patients with mild (high baseline MMSE score [21-26]) and moderate (low baseline MMSE score [15-20]) AD. In contrast, performance on the ADAS-Cog shows systematic change depending on disease severity at baseline. The ADAS-Cog showed cognitive decline for patients with moderate AD but did not show decline in patients with mild AD. This is consistent with previous reports^{7 - 8} of comparatively small annual decline on the ADAS-Cog in those with mild AD.

The relative lack of decline in ADAS-Cog scores in patients with mild AD becomes problematic in the design of clinical trials for disease-modifying AD therapies. For trials that are powered to detect a reduction in the rate of decline over 1 year in the drug-treated patients compared with placebo, the number of subjects with mild AD required to detect a reduction in ADAS-Cog would be excessive. For example, sample size estimates were calculated to determine the number of patients with mild AD (MMSE score, 21-26) necessary to detect a drug-placebo difference of 50% of the annual placebo decline using the 12-month NTB and ADAS-Cog changes for patients with mild AD based upon the results of the AN1792(QS-21) study. To detect a change from baseline in the treatment group that is 50% less than that in the placebo group, with 80% power, a significance level of .05, and assuming equal sample sizes in the treatment and placebo groups, 1478 patients per group would be needed using the ADAS-Cog (mean placebo group decline, 0.63; SD, 4.32). In contrast, only 159 patients per group would be necessary to detect a change from baseline in the treatment group that is 50% less than that in the placebo group for the NTB (mean placebo group decline, 0.21; SD, 0.47). While the difference in sample sizes required for 80% power using the ADAS-Cog and NTB is accentuated by the particularly low rate of decline on the ADAS-Cog in the placebo group of the AN1792(QS-21) trial, the insensitivity of the ADAS-Cog is also apparent from other recent studies^{7 - 8} that found relatively small declines on this measure in patients with mild AD.

This study has several limitations. To our knowledge, this is the first study to formally evaluate the NTB as a composite measure for AD clinical trials. Performance on the NTB from other AD patient samples would provide additional support for the NTB as an alternative cognitive tool in AD clinical trials. The NTB could also be useful in evaluating patients with mild cognitive impairment, so future studies of the NTB might consider including these patients. Another limitation of this trial was that it was not designed to assess the test-retest reliability of the NTB. While the 6- and 12-month NTB scores allowed us to estimate test-retest reliability, examination of test-retest reliability over shorter periods would also be of interest to evaluate in future studies.

In summary, the NTB has exhibited excellent psychometric properties, especially in the areas of concurrent validity and test-retest reliability. In contrast to the ADAS-Cog, the NTB z score showed sensitivity to change across the range of MMSE scores in this trial and was the only measure of cognitive efficacy to yield a statistically significant effect in favor of the AN1792(QS-21) immunotherapeutic treatment. The NTB may represent a useful cognitive measure for clinical trials to assess cognitive change in patients with mild to moderate AD or potentially mild cognitive impairment and may be more effective than traditional measures in detecting cognitive changes in the clinical assessment of a therapy designed to reverse or halt disease pathological features.

Correspondence: John Harrison, C. Psychol, PhD; CPC Pharma Services, PO Box 3223, Warminster, Wiltshire BA12 8XA, England (johncpc@btinternet.com).

Accepted for Publication: January 24, 2007.

Author Contributions: Dr Harrison had full access to all of the data and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Harrison, Minassian, Jenkins, Black, Koller, and Grundman. Acquisition of data: Black. Analysis and interpretation of data: Harrison, Minassian, Jenkins, Black, and Grundman. Drafting of the manuscript: Harrison, Minassian, Jenkins, and Grundman. Critical revision of the manuscript for important intellectual content: Harrison, Minassian, Jenkins, Black, Koller, and Grundman. Statistical analysis: Harrison, Minassian, Jenkins, and Grundman. Obtained funding: Grundman. Administrative, technical, and material support: Harrison, Black, and Koller. Study supervision: Black, Koller, and Grundman.

Financial Disclosure: Dr Minassian is a salaried employee of and has been awarded stock options from Elan Pharmaceuticals; Dr Black is a full-time employee of and holds stock in Wyeth Research; and Drs Koller and Grundman are salaried employees of and have stock options and ownership in Elan Pharmaceuticals.

Funding/Support: The AN1792 clinical study was supported by Elan Pharmaceuticals and Wyeth Research; however, no support was provided for the analyses described herein.

Role of the Sponsor: Researchers from Elan Pharmaceuticals and Wyeth Research participated in the design, conduct, and analysis of the research described herein.

Disclaimer: The intent of the present study is not to provide the clinical results of the AN1792 trial (previously published) but to describe the properties of instruments used in that trial.