Don’t be late! Timely identification of cognitive impairment in people with multiple sclerosis: a study protocol

Abstract

Background

Cognitive impairment occurs in up to 65% of people with multiple sclerosis (PwMS), negatively affecting daily functioning and health-related quality of life. In general, neuropsychological testing is not part of standard MS-care due to insufficient time and trained personnel. Consequently, a baseline assessment of cognitive functioning is often lacking, hampering early identification of cognitive decline and change within a person over time. To assess cognitive functioning in PwMS in a time-efficient manner, a BICAMS-based self-explanatory digital screening tool called the Multiple Screener^©, has recently been developed. The aim of the current study is to validate the Multiple Screener^© in a representative sample of PwMS in the Netherlands. Additionally, we aim to investigate how cognitive functioning is related to psychological factors, and both work and societal participation.

Methods

In this cross-sectional multicentre study, 750 PwMS (aged 18–67 years) are included. To obtain a representative sample, PwMS are recruited via 12 hospitals across the Netherlands. They undergo assessment with the Minimal Assessment of Cognitive Functioning in MS (MACFIMS; reference-standard) and the Multiple Screener^©. Sensitivity, specificity, and predictive values for identifying (mild) cognitive impairment are determined in a subset of 300 participants. In a second step, the identified cut-off values are tested in an independent subset of at least 150 PwMS. Moreover, test–retest reliability for the Multiple Screener^© is determined in 30 PwMS. Information on psychological and work-related factors is assessed with questionnaires.

Discussion

Validating the Multiple Screener^© in PwMS and investigating cognition and its determinants will further facilitate early identification and adequate monitoring of cognitive decline in PwMS.

Background

Multiple sclerosis (MS) is a chronic inflammatory disease of the central nervous system characterized by demyelination and neurodegeneration [1]. In addition to physical limitations, 43–65% of people with MS (PwMS) develop cognitive symptoms that may severely affect daily life functioning and consequently health-related quality of life [2,3,4]. The most commonly and earliest affected cognitive domains are information processing speed, verbal memory, and visuospatial memory [3, 5].

The impact of cognitive impairment on daily life functioning can be significant, especially since most PwMS are relatively young at disease onset [3]. As such, cognitive impairment is one of the main reasons for unemployment in MS [6, 7]. About 43% of PwMS become unemployed within three years of diagnosis due to fatigue and physical impairment, but also cognitive impairment [8,9,10]. This early unemployment has a large impact on PwMS, their families, and on society in general [11]. However, by the time PwMS with self-perceived cognitive problems approach health care professionals, their cognitive deficits are often already advanced and potentially more difficult to treat, suggesting that early intervention might be promising [12, 13].

The need for early intervention is emphasized by a recent study showing that successful response to cognitive rehabilitation depends on the status of the brain’s functional network before the intervention [14]. PwMS with a functional connectivity that is more like that of healthy controls were able to benefit from a cognitive rehabilitation program (i.e., these participants significantly improved on neuropsychological tests and had better self-perceived cognitive functioning). However, those PwMS with a less efficient brain network at baseline (indicative of more MS-related pathology) were non-responsive, suggesting the existence of a small window of opportunity for intervention early in the development of the disease [14]. Additionally, other studies show that less MS-related brain damage (e.g., higher grey matter volume) are linked to better cognitive rehabilitation outcomes [15,16,17]. Therefore, it is crucial to identify PwMS at the earliest stages of cognitive impairment to allow for intervention when it is most effective to improve cognitive functioning.

Recent international recommendations for measuring and monitoring cognitive functioning in PwMS propose a baseline cognitive screening and annual follow-up [18]. However, in contrast to these recommendations, neuropsychological testing is not part of standard MS-care in current clinical practice in the Netherlands (and several other countries) [19]. Consequently, a good reference assessment of baseline cognitive performance is often lacking, hampering the detection of the first subtle changes in cognition. Detecting these early changes is particularly difficult when PwMS already experience difficulties in daily life functioning but still perform (above) average on neuropsychological assessments [3]. The main reason for not following the international recommendations is the lack of time and specialized personnel to assess cognitive functioning [19]. As digital assessment tools may lower the threshold for systematic assessment of cognitive functioning in PwMS, we recently developed a self-explanatory, time-efficient digital screening tool, the Multiple Screener^© [20].

The Multiple Screener^© consists of an adjusted version of the validated and recommended BICAMS (Brief International Cognitive Assessment for MS) paper-and-pencil assessment [21] and takes 15 min to complete. It assesses the most frequently impaired cognitive domains in MS: information processing speed via the Symbol Digit Modalities Test (SDMT [22]), verbal learning and memory via the Dutch version of California Verbal Learning Test Second Edition (CVLT-II [23,24,25]), and visuospatial learning and memory via the Spatial Recall Test (SPART [26]) [20]. In addition, the Multiple Screener^© also includes questionnaires on depression and anxiety [27], fatigue [28] and self-perceived cognitive symptoms [29], taking into account psychological factors when screening for cognitive deficits in MS. The main advantages of the Multiple Screener^© are that it does not require specialized personnel for administration (i.e., PwMS can perform the tests on their own), has automated scoring, and is time-efficient. The Multiple Screener^© has been tested in 236 healthy controls and normative data are available [20]. In healthy controls, the correlations between the Multiple Screener^© and the paper-and-pencil versions of the neuropsychological tests have been shown to be good to excellent [20]. However, a next essential step before the Multiple Screener^© can be used in clinical practice is to investigate its diagnostic accuracy especially in identifying PwMS with mild cognitive impairment according to a reference standard (the Minimal Assessment of Cognitive Function in Multiple Sclerosis, MACFIMS [30]), allowing for timely identification.

Objectives

As part of a larger research project (i.e., the Don’t be late! study, see Table 1) the primary objective of this study is to determine diagnostic accuracy of the Multiple Screener^© in a representative Dutch sample of PwMS. Specifically, we aim to determine how well the Multiple Screener^© can differentiate between PwMS with no cognitive impairment, mild cognitive impairment, and cognitive impairment according to the reference-standard (MACFIMS [30]). In a second step we aim to confirm the observed diagnostic accuracy of the Multiple Screener^© in differentiating between PwMS with no cognitive impairment, mild cognitive impairment, and cognitive impairment in an independent subset of PwMS. When reporting on the diagnostic accuracy of the Multiple Screener^©, the Standards for Reporting Diagnostic Accuracy guidelines from the Equator-Network (STARD 15, [31]) will be followed.

Table 1 Don’t be late study!

The study has the following secondary objectives:

• 1) To determine the test–retest reliability of the Multiple Screener^©;

• 2) To determine how cognitive, psychological, work-related, and health-related quality of life outcomes are related.

Methods

Design and setting

The present study is a cross-sectional multicentre study in which a representative sample of 750 PwMS will be included. In the 12 participating Dutch hospitals, demographical and medical information will be collected, and cognitive functioning of PwMS will be assessed with both the reference standard (MACFIMS) and the Multiple Screener^©. In line with international validation guidelines [32], the assessment of the Multiple Screener^© will be repeated within 3 weeks after the hospital visit in a small subset of participants (N = 30) in order to determine test–retest reliability. Finally, all participants will fill in several online questionnaires at home.

Participants

Recruitment and consent

We aim to recruit a representative sample of PwMS in the Netherlands that visit the neurologist in light of their MS. We will include PwMS with a variety in MS types (relapsing remitting, secondary progressive (85% of the population) and primary progressive (15%)), disease duration and age. All participating hospitals are asked to provide a patient information letter for a set period of time to all PwMS that visit the outpatient clinic, independent of cognitive status, employment status, disease status and meeting the in- and exclusion criteria. Contact details of PwMS that give permission to be approached about participation are shared with the researchers from the Amsterdam UMC, Vrije Universiteit Amsterdam. After at least one week, they contact the potential participant to provide additional information if requested (or refer to an independent physician) and to ask whether they would like to participate in the study. When PwMS decide to participate, the researcher will screen the subjects for eligibility via telephone (see below for inclusion and exclusion criteria) such that an unnecessary hospital visit will be avoided when a subject is not eligible. In case of a positive screening outcome, a visit for the assessment will be scheduled at which written informed consent will be obtained.

Inclusion criteria

To be eligible to participate in this study, people must fulfil the following criteria: a confirmed MS diagnosis according to the McDonald 2017 criteria [33], age between 18 and 67 years, no changes in disease modifying therapy within the last 3 months, and no relapse or steroid treatment six weeks prior to the study visit.

Exclusion criteria

Participants will be excluded from participation in this study if they have other neurological or psychiatric comorbidities that can potentially influence cognitive functioning, a current or history of drug or alcohol abuse, have insufficient vison or hearing, or are unable to speak or read Dutch. The reasons for excluding participants from the current study will be documented.

Ethical approval

The study will be conducted according to the principles of the Declaration of Helsinki (2013) and in accordance with the Dutch Medical Research Involving Human Subjects Act (WMO). The Medical Ethical Committee (METC) of the Amsterdam UMC, Vrije Universiteit Amsterdam has approved this study (METC 2021.0707) on 4 May 2022.

Measures and procedures

Demographic and clinical characteristics

During the assessment at the participating hospitals, information on demographical and clinical characteristics will be collected from participants and their medical file. The following characteristics will be collected: age in years, sex, educational level (Dutch Verhage scale), work status, date of diagnosis, MS subtype, MS severity assessed with the telephone version of the Expanded Disability Status Scale (EDSS) [34], medication usage, medical history, and comorbidities.

Neuropsychological assessment

Participants will undergo an extensive neuropsychological assessment (120–150 min) at the location of the participating site. The assessment consists of the Multiple Screener^©, the MACFIMS test battery [30], a social cognition test, performance validity tests and an assessment of awareness of cognitive functioning. Parallel versions will be used for the tests that are overlapping between the MACFIMS and Multiple Screener^© and the order of administration will be counterbalanced to minimize learning effects and influence of fatigue.

Multiple Screener^©

The Multiple Screener^© is a digital tool aiming to assess cognitive functioning in PwMS. It is a digital, self-explanatory version of the validated and recommended BICAMS [21] and takes 15 min to complete. It includes the following three tests:

• Digital version of the CVLT-II [23,24,25]: Verbal learning and memory. The ability to learn 16 auditory presented semantically related words is examined over five trials. After each trial participants are asked to type the remembered words (direct recall). The total number of the correctly remembered words is calculated.

• Digital version of the SDMT [22]: Processing speed and working memory. Nine pairs of digits and symbols are visually presented. Participants are asked to type the numbers associated with the paired symbols as fast as possible. The total number of correct answers within 90 s is calculated.

• Digital version of the SPART [26]: Visuospatial memory. A 6 × 6 grid with 10 black checkers is displayed three times for ten seconds. After each time, an empty grid is displayed with ten black checkers next to it. Participants must swipe the black checkers to the correct places in the empty grid to match what they observed. The total number of correctly placed checkers is calculated. See Fig. 1 for an illustration of the SDMT and the SPART.

Fig. 1

The digital version of the Digit Modalities Test (SDMT) (left) and the Spatial Recall Test (SPART) (right) in the Multiple Screener^© application. The Dutch version of the California Verbal Learning Test–second edition (CVLT-II) is not depicted as this test has an auditory format

The software of the Multiple Screener^© is produced by the manufacturer Sherpa B.V. In accordance with the legislation of the Medical Device Directive, the software is qualified as a medical device, classified in risk class I (low risk), reported to FARMATEC-CIBG-VWS, and CE-certified by the manufacturer.

A subset of participants (n = 30) will be invited to return to the hospital within 3 weeks after the initial assessment to complete The Multiple Screener^© for a second time to determine the test–retest reliability.

MACFIMS

The MACFIMS is an internationally renowned and well-validated, 90-min, paper-and-pencil test battery that is commonly used to determine cognitive impairment in MS. It consists of tests for verbal and visuospatial learning and memory and information processing speed (cf. the Multiple Screener^©) and in addition tests for language and working memory, visuospatial orientation, and executive functioning [30]. The tests and corresponding cognitive domain(s) are summarized in Table 2.

Table 2 Minimal assessment of cognitive functioning in MS test battery

Performance validity

The Amsterdam Short Term Memory Test (ASTM) [39] will be used to assess performance validity in all participants. In case the ASTM indicates underperformance (cut-off of ≤ 84; [40]) the Rey 15-item Test (higher specificity compared to the ASTM) [41] will additionally be performed.

Social cognition

Social cognition and in particular affective theory of mind (i.e., the ability to recognise the thought or feelings of others) will be measured with the revised version of the Reading the Mind in the Eyes Test [42].

Awareness of cognitive functioning

Finally, to assess (online) awareness of global cognitive functioning, a subset of the participants (N = 200) will be asked to estimate their own performance immediately before and after completion of the MACFIMS battery. More specifically, they will be asked to estimate what percentile score they believe that they would receive for the overall test battery if compared with a randomly selected demographically matched peer group. A normal distribution including brief explanations of percentiles scores (inspired by Rothlind et al. [43]) will serve as a visual aid for participants.

Questionnaires

To reduce the burden on the day in the hospital, participants will fill out several online questionnaires at home (for an overview of the questionnaires see Table 3). Participants will be asked to complete the questionnaires within one week after the hospital visit to ensure that the collected data most closely resembles the status of the participant during the hospital visit. The researcher will send reminders if the questionnaires have not been returned.

Table 3 Questionnaires on MS-related and psychological factors, work and societal participation

Outcomes

Primary outcome

The primary outcome measures for the first study objective are sensitivity, specificity, negative and positive predictive value, and the receiver-operating characteristic of the Multiple Screener^©.

Secondary outcomes

A secondary outcome measure is the test–retest reliability (i.e., intraclass correlation coefficients) of the Multiple Screener^©. Additionally, secondary outcome measures include the relationships between cognitive functioning (as measured with the Multiple Screener^© and the MACFIMS test battery [30]) with the following measures (the hypothesized directions of these relationships are summarized in Table 4):

• 1. Psychological measures: self-perceived cognitive functioning [29], awareness of cognitive functioning [43], physical and psychological impact of MS [45], mood [27], fatigue [28], personality traits [50, 52], stressful life events [53], resilience [48] and mastery [49]

• 2. Patient-reported health-related quality of life [44]

• 3. Work-related measures: MS-related work difficulties [55], work productivity and activity impairment [54], negative work events and work accommodations [57]

• 4. Health and lifestyle measures: Physical exercise, smoking, alcohol, diet, sleep, BMI, household composition, and social activities.

Table 4 Hypothesized direction of correlations between cognitive scores with health-related quality of life, psychological, and work-related, health and lifestyle measures

Power calculation

Because the Multiple Screener^© is aimed at assessing cognitive decline, especially sensitivity to detect (mild) cognitive impairment should be high, while a relatively lower degree of specificity can be tolerated. Based on accuracy values from the paper–pencil version of the Multiple Screener^© (BICAMS) in MS and comparable cognitive screening instruments frequently used in people with Parkinson’s disease, we aim for sensitivity values of at least 0.80 and specificity values of at least 0.70 [58, 59]. As reported by Amato et al. [60] we expect that approximately 50% of PwMS at the outpatient clinics will classify as having no cognitive impairment (i.e., cognitively preserved; CP), 30% will classify as having mild cognitive impairment (MCI), and 20% will classify as having overt cognitive impairment (CI). Based on these prevalence estimations, a minimum sample size of 100 will be required to detect sensitivity values of at least 0.80 with a power of 80% and alpha threshold of 0.05 [61]. In total 198 participants will be required to detect specificity values of at least 0.70 with identical power and significance level. However, a sample size of 300 is recommended to reliably evaluate accuracy values of screening tools [61]. As such, for our primary objective we aim to include 300 PwMS. Moreover, we aim to confirm the accuracy of the Multiple Screener^© in an independent sample of at least 150 PwMS (i.e., another subset of our sample).

This study is part of a larger research project and a subset of participants from the current study (i.e., participants with mild cognitive impairment) will be selected for the intervention study of the second work package (see Table 1). Therefore, the overall required sample size (N = 750) is based on the power calculation for the intervention study. For additional information, the reader is referred to Aarts et al. [62].

Statistical analysis

Data will be analysed using R Studio software (at least version 4.2.1; [63]) and IBM SPSS Statistics (at least version 28 [64]). In case of non-normality, data will be presented as median and inter-quartile range and transformed for further analyses if appropriate or non-parametric tests will be applied. Participants with missing data and outliers will be excluded for that particular analysis. A p-value of 0.05 will be considered as statistically significant for all analyses.

Primary study parameters

For our primary objective we will determine sensitivity, specificity, positive and negative predictive values of the Multiple Screener^© as compared to the MACFIMS. Based on previous definitions for cognitive impairment among PwMS [65], participants will be divided into three subgroups depending on their severity of cognitive impairment. Participants scoring at least 2 standard deviations (SDs) below the mean normative values on at least 2 out of 6 cognitive domains assessed with MACFIMS will be classified as having CI. Participants who score 1 to 1.99 SDs below the mean normative values on at least 1 cognitive domain and/or at least 2 SDs below the mean normative values on 1 cognitive domain (not fulfilling the CI criteria) will be classified as having MCI. The remaining participants will be defined as CP [65]. For the Multiple Screener, participants scoring at least 2 SDs below the mean normative values on at least 1 of the 3 tests will be classified as CI. Participants scoring 1 to 1.99 SDs below the mean normative values on at least 1 of the 3 tests will be classified as MCI. The remaining participants will be defined as CP. Overall, regression-based norms adjusted for age, sex, and education will be used for individual cognitive tests before determining cognitive status.

Participants' cognitive status will be determined via the MACFIMS and will be investigated in relation to the scores detected with Multiple Screener^©. All accuracy values will be calculated separately for the detection of MCI and CI (one against all approach for multiclass classification) and will be presented as percentages. Additionally, receiver-operating characteristic (ROC) analyses will be performed to determine overall accuracy and optimal cut-off scores of the Multiple Screener^© for detecting MCI and CI in people with MS. Once we have determined accuracy values and optimal cut-off scores in 300 participants, we will test these in another subset of at least 150 participants to confirm their correctness.

The Multiple Screener^© will be considered a sufficiently adequate screening instrument for the detection of (M)CI if its overall sensitivity values are at least 0.80 and specificity values at least 0.70. However, if one of the individual tests does not meet these criteria, we will determine accuracy values of the two other tests over and above that of all three tests combined.

Secondary study parameters

Test–retest reliability

Test–retest reliability of the Multiple Screener^© will be determined by calculating intraclass correlation coefficients (ICCs) for absolute agreement, using a two-way mixed model. Based on the 95% confidence interval of the ICC estimate, values will be considered to reflect poor reliability (< 0.5), moderate (0.5 -0.75), good (0.75–0.9), and excellent (> 0.90) [66]. The coefficients will be calculated separately for the SDMT, CVLT-II, and the SPART.

Relationships between cognition and psychological, work-related, and patient-reported health-related quality of life measures

Cross-sectional associations between cognition and psychological, work-related, and health-related quality of life measures will be analysed using Pearson’s or Point-Biserial correlations and linear regression analyses (including stepwise procedures) in both subsets and the overall sample. Correlations coefficients of less than 0.3, between 0.3 and 0.7, and greater than 0.7 will be considered weak, moderate, and strong, respectively [67]. An overview of the hypothesized correlations can be found in Table 4.

Additionally, logistic regression analyses will be used to identify the predictive value of demographical and disease characteristics (such as sex, MS subtype, medication, comorbidities) on cognitive functioning. Additionally, differences between groups (CP, MCI and CI) in demographic and clinical characteristics and other outcome measurements (e.g., psychological, work-related and health-related quality of life measures) will be analysed using independent samples t-tests, Mann–Whitney U tests and Pearson’s chi-square tests. For particular analyses, confounding variables (such as age, sex, education, EDSS score, disease duration, mood, fatigue etc.) will be inserted. Bonferroni corrections will be applied to correct for multiple comparisons within each objective.

Safety reporting

We will not collect information on (serious) adverse events due to the observational and non-interventional nature of this study.

Study status

The first participant was included on 19 July 2022. Currently 216 participants have been enrolled in the study (December 2023).

Discussion

Cognitive impairment is common in PwMS and can severely affect health-related quality of life. In order to intervene timely, a baseline assessment and frequent monitoring of cognitive functioning seems crucial. However, in the Netherlands, neuropsychological assessment is not (yet) integrated into standard care due to the time-consuming nature of cognitive testing and limited availability of trained personnel [19]. The current study will validate a digital screening tool with the primary objective to enable early identification of cognitive decline in PwMS. The validation of the Multiple Screener^© within a representative sample of PwMS that visit a neurologist, will lay the foundation for implementing a cognitive screening tool for annual testing in clinical practice in the near future. This study will further help raise awareness among health care professionals about cognitive impairment in MS and its significance within the broader scheme of priorities in MS-care. The fact that 12 hospitals in the Netherlands are interested in participating in the study further emphasizes the need for such screening methods. In addition, this study will also contribute to the development of practical guidelines for Dutch professionals regarding the screening and subsequent monitoring of cognitive decline in MS.

Moreover, with the present study we are collecting one of the largest datasets on cognition and its determinants in PwMS which will provide us with a wealth of data that can be used to answer multiple relevant related research questions. Specifically, it will enhance our understanding of the relationship between cognition and relevant confounders, ranging from cognitive self-awareness to fatigue and mood problems.

To conclude, the validation of the Multiple Screener^© will facilitate early identification of cognitive impairment in PwMS; ultimately enabling better management of cognitive symptoms in this population. Additionally, the study's comprehensive dataset will allow new insights into factors related to cognition in PwMS, thus informing future research and clinical practices. Finally, timely identification of cognitive impairment is a crucial step for initiating early interventions, an important aspect that will be explored in subsequent phases of the larger Don't be late! study.