- Research
- Open access
- Published:
Validation and comparison of the stroke prognosis instrument (SPI-II) and the essen stroke risk score (ESRS) in predicting stroke recurrence in Asian population
BMC Neurology volume 23, Article number: 287 (2023)
Abstract
Background
Currently, there are limited data on the accuracy of available risk scores to predict stroke recurrence in the Asian population.
Method
A single-center, retrospective cohort study was conducted among patients with acute ischemic stroke during January 2014 - December 2018. Longitudinal data with three years of follow-up among these patients were collected and validated through both electronic and manual chart review. The area under the receiver-operating curve (AUROC) method or C-statistic and calibration plot were used to evaluate and compare the Stroke Prognosis Instrument II (SPI-II) and the Essen Stroke Risk Score (ESRS) in predicting the long-term risk of recurrent ischemic stroke. The predictive performances of the two scores were compared using DeLong’s method.
Results
The study cohort consisted of 543 patients, including 181 and 362 patients with and without recurrent events. There were no significant differences in mean age and gender between the two groups. Recurrence cases tended to have significant more risk factors compared to those without events. Among cases with recurrent events, 134 (74.03%) and 65.74% (119) cases were classified as high-risk based on SPI-II and ESRS, respectively. The AUROC curve of the SPI-II and ESRS score was 0.646 (95% CI, 0.594–0.697) and 0.614 (95%CI, 0.563–0.665), respectively (p = 0.394). Based on the calibration plot, the SPI-II and ESRS scores showed similar moderate predictive performance on recurrence stroke with a C statistic (95% CI) of 0.655 (95% CI: 0.603–0.707) and 0.631 (95% CI 0.579–0.684), respectively.
Conclusion
Both ESRS and SPI-II scores had moderate predictive performance in Thai population.
What is already known about this subject:
- Currently, there is limited data regarding the accuracy of risk score to predict stroke recurrence in Asian population.
What this study adds:
- The Stroke Prognosis Instrument (SPI-II) and the Essen Stroke Risk Score (ESRS), a commonly used risk score, developed based on Western population cohort, possessed moderate predictability in predicting long-term stroke recurrence in Asian population.
Introduction
Stroke is among the leading causes of death and disability in low and middle-income countries (LMIC) [1]. Thailand is a middle income country in Southeast Asia with a population of 65 million. Data from Thailand’s national health statistic indicated that there were more than 355,000 stroke cases in the year 2019 [2]. Stroke was also the major contributor to disability adjusted life years (DALYs) lost in 2017 globally [3]. Despite such a magnitude of the problem, stroke care remains poor, especially in developing countries, partly due to a less advanced health system and suboptimal public health literacy of the public [4]. As a result, stroke recurrence seems to be much higher in developing countries or less mature health systems compared to developed countries [5, 6]. A recent study in China showed that 41% of stroke survivors had recurrent stroke at 5 years while data from a recent registry of developed countries showed only 9.5% of recurrence in 5 years [7, 8]. With multiple recurrence, mortality and disability rates are therefore much higher in developing countries [1, 3].
To better predict risk of future recurrence, a number of risk scores have been developed to assist clinicians in identifying patients who are at high risk of stroke recurrence [9]. Among these risk scores, the Stroke Prognosis Instrument II (SPI-II)[10] and the Essen Stroke Risk Score (ESRS)[11] have been developed and validated in various populations showing potential utility in a diverse group of patient population. These two scores are commonly used due to their relative ease of use and readily available variables for the models. These risk scores however were developed in mostly Caucasian population and in developed healthcare systems. Previously, both SPI-II and ESRS scores have been evaluated and compared on the predictive accuracy in Asian population [12, 13]. Results of these studies indicated that these scores had moderate predictive performance in Asian population. However, little data exists on the predictive performance of these risk scores in developing Southeast Asian countries including Thailand. Therefore, this study aimed to evaluate and compare the predictive performance of SPI-II and ESRS scores in predicting long-term stroke recurrence among Thai ischemic stroke survivors. The results of this study may provide useful information on the possibility of adopting these risk scores to identify and streamline limited healthcare resources to care for patients at high risk of recurrence.
Methods
Study design and setting
A retrospective observational study was conducted among patients who experienced and survived acute ischemic stroke during January 2014 - December 2018. The study site was the Siriraj Hospital, a 2,500 bed, university-affiliated, tertiary care hospital in Bangkok, Thailand. The hospital is equipped with a 17-bed stroke unit. There were 21 full-time staff neurologists with board certification in neurology who provided care for patients under the supervision of the Neurology Department. In addition, the hospital embraces multidisciplinary approach in stroke care with a multidisciplinary team of physicians, nurses, pharmacists and other allied health workers who work together both for outpatient and acute care settings. The study was approved by the Institutional Review Board of Siriraj Hospital (IRB Number: Si642/2019, date of approval: September 17, 2019) and followed the principles of the Declaration of Helsinki. The informed consent of patients was waived by the Institutional Review Board of Siriraj Hospital due to the retrospective nature of the study.
Study population
All adult patients who experienced and survived acute ischemic stroke (International Classification of Disease Tenth Revision or ICD-10 of 1630, 1631, 1632, 1633, 1635, 1638 and 1639) during January 2014 - December 2018 were screened. Patients who had regular follow-up for at least three years at Siriraj Hospital were included. Exclusion criteria included (1) cerebral embolism, (2) presence of atrial fibrillation, (3) missing critical data for risk score calculation, (4) referral patients with-out longitudinal follow-up information, and (5) loss to follow-up. Two-step approaches were used to identify patients including case identification by the ICD‐10 and manual chart review for case confirmation and data collection.
Data collection
Data were collected manually from medical chart and hospital database where appropriate and transferred into a standardized, case record form by two clinical pharmacists. Demographic data including age, gender, smoking status and alcohol use were collected. Comorbidities were obtained from clinical diagnosis from the medical chart and/or the International Classification of Diseases 10th Revision (ICD-10) coding from the hospital database. Stroke related information consisted of types of stroke, duration of stroke, level of disability were also collected. All cases of stroke were confirmed by computed tomography (CT) and/or magnetic resonance imaging (MRI).
Risk scores and recurrent stroke
Details on the components and weight of each component for both the SPI-II and ESRS scores are shown in Table 1. Generally, while both scores used age and presence of risk factors and comorbidities, the differences are the presence (and absence) of some variables, including smoking, stroke, peripheral arterial disease and heart failure as variables [10, 11]. There are also differences in the weighting of common variables in the scores. In addition, these scores used different risk categories. SPI-II classifies patients into low (0–3 points), moderate (4–7 points), and high risk (8–15 points) groups, while ESRS classifies patients into either low (0–2 points) or high risk (3–9 points) groups. These differences reflect different study designs, study population and the methods used to construct the models and identify risk thresholds [10, 11]. For recurrent events, two-step approaches were used to identify patients with stroke recurrence including case identification by ICD‐10 and manual chart review for case confirmation.
Statistical analysis
Continuous variables were reported as numbers and percentages and calculated as the mean ± standard deviation (SD) or median (interquartile range, IQR) depending on the distribution detected by normality tests. Stroke outcomes by the ESRS and SPI-II scores were computed as the rate of events per 100 patient-years. Independent two-proportion sample were compared using either the chi-square or Fisher’s exact test as appropriate. The significance of continuous variable as a two-independent sample was assessed with Independent t-test or Mann Whitney U test following normality assumption. The ROC curve was performed by discriminant threshold for the ESRS and SPI-II score to obtain the optimum values with the highest sensitivity and specificity compared to stroke recurrence. Cox proportional hazards models were used to determine the risk for each threshold through the hazard ratio and the 95% confidence interval (CI), respectively. The area under the receiver operating curve (AUROC) method or C-statistics was used to test the diagnostic accuracy for stroke risk classification of ESRS and SPI-II scores. The predictive performances of the two scores were compared using DeLong’s method [14]. In addition, the calibration plots with score of 0, 1, 2, 3, ≥ 4 for both ESRS and SPI-II was evaluated. Statistical significance was defined as p < 0.05. All statistical analyses were performed by using SPSS version 18.0 and STATA version 14.1.
Results
There were a total of 543 patients who met the inclusion criteria, 181 patients had recurrent stroke during the follow-up period. The baseline characteristics of the study population are shown in Table 2. Mean ages of those with and without recurrent stroke were not significant different (65.95 \(\pm\) 13.66 vs. 65.41 \(\pm\) 13.81 years for those with recurrent stroke and those without recurrent events, p = 0.667). No difference in gender distribution existed. For stroke subtypes, 46.4%, 24.9% and 28.7% were large vessel disease, small vessel disease and stroke of undetermined etiology, respectively. A history of prior stroke or TIA was more common in patients with recurrent events compared to those without recurrent events. In addition, patients with recurrence tended to have higher rates of cardiovascular risk factors including hypertension, impaired fasting glucose, peripheral arterial disease, prior myocardial infarction, and other cardiovascular diseases. Among 181 cases with recurrent events, 134 (74.03%) and 65.74% (119) cases were classified as high-risk based on the SPI-II and ESRS, respectively. Risk of recurrent stroke after ischemic stroke stratified by Stroke Prognosis Instrument II (SPI-II) and Essen Stroke Risk Score (ESRS) are shown in Table 3.
Comparison of SPI-II vs. ESRS scores
Overall, both the SPI-II and ESRS scores showed moderate predictive performance in recurrent stroke with AUROC (95% confidence interval [CI]) of 0.646 (0.594–0.697) and 0.614 (0.563–0.665) as shown in Fig. 1(A) and Fig. 1(B), respectively. There was no statistical significance between the two scores (p = 0.394). SPI-II score of ≥ 4 showed 63% sensitivity and 58% specificity with positive predictive value (PPV) of 43% (95% CI: 39.07–47.09) and negative predictive value (NPV) of 76% (95% CI: 72.06–79.74). ESRS score of > 3 showed 65% sensitivity and 51% specificity with PPV of 40% (95% CI: 36.80-44.06) and NPV of 75% (95% CI: 70.52–78.88).
The results of the Cox proportional hazards models showed that patients with a SPI-II score of ≥ 4 (high risk group) had a significantly higher risk of recurrent events compared to those with a SPI-II score of < 4 (HR: 2.12; 95% CI: 1.562–2.862, p < 0.001) as shown in Fig. 2(A). Patients with an ESRS score of > 3 (high-risk group) also had a significantly higher risk of recurrent events compared to those with an ESRS score of < 3 (HR: 1.85; 95% CI: 1.359–2.507, p < 0.001), as shown in Fig. 2(B) The risk of a recurrent event significantly increased at the beginning at one year after the index event.
Predictive performance of SPI-II and ESRS scores
The predictive performance of the SPI-II and ESRS scores was evaluated using calibration plots. The C index and the slope of the SPI-II score were 0.655 (95% CI: 0.603–0.707) and 0.250 (95% CI 0.169–0.332), respectively. The C index and slope of the ESRS score were 0.631 (95% CI 0.579–0.684) and 0.321 (95% CI 0.190–0.452), respectively. Based on the calibration analysis, both the SPI-II and ESRS scores had a moderate ability to identify those who experienced recurrent events and those who did not. (Figure 3A and B).
The predictive performance of both scores was evaluated using calibration plots. Horizontal axis shows rate of recurrent stroke in the study cohort, whereas vertical axis indicates observed rate of recurrent stroke in original cohort per 100 person-years.
Discussion
In this study, both the SPI-II and ESRS were shown to have moderate ability to identify those who were at risk of recurrent events among survivors of ischemic stroke in Thailand. Compared to previous studies in the Asian population, the SPI-II and ESRS appeared to perform well in our study population compared to previous reports. Meng et al. evaluated the performance of both scores in 11,384 Chinese patients who were prospectively followed at 132 hospitals in China. The AUROCs of SPI-II and ESRS were 0.59 (95% CI, 0.58–0.61) and 0.59 (95% CI, 0.58–0.60), respectively [12]. In fact, the AUROC of both scores in our study was quite similar to those reported by Weimar et al.[15] While this is an interesting finding, more studies preferably with larger sample size are needed to confirm our results. However, based on the finding of our study, both SPI-II and ESRS scores appear useful to be used in Thai population who experienced previous acute ischemic stroke.
It is important to note that the risk factors listed in both risk scores have been shown to have biological relationship with the pathophysiology of ischemic stroke. Age, hypertension, diabetes and smoking are well-known, independent risk factors for ischemic stroke partly due to endothelial dysfunction that is the ultimate consequence of these risk factors [6]. History of atherosclerotic vascular diseases in other organs has also been shown to predict the future risk of stroke [7]. This is most likely a reflection of systemic vascular dysfunctions that affects all major organs including brain vasculature [16].
Predicting future recurrent stroke has important public health value, especially in developing countries where healthcare resources are limited [9]. Stroke has become such a grave public health burden especially among developing countries in Asia, both due to the large number of patients affected and the clinical and economic consequences that follow [1, 4]. However, stroke care in these countries remains far from ideal [4]. Therefore, a clinically reliable tool to identify high-risk patients which may allow prioritization of resource utilization can help improve care for those who are in the greatest need. Risk prediction score is therefore useful in optimizing resource utilization. With ease of use and low cost for application for both SPI-II and ESRS, these scores are therefore useful for resource-limited countries. Nevertheless, both scores are still with moderate predictive performance. This is partly due to the fact that recurrent stroke can be driven by a myriad of factors. Recently, artificial intelligence (AI) has been adopted in the stroke risk prediction model [17, 18]. More studies are coming to show whether this AI-based prediction model can be used in real clinical practice, especially in developing countries.
There are several limitations in our study. First, our sample size was relatively small compared to previous studies, due to the necessity to perform manual chart review. As a result, large studies are needed to confirm our findings. Second, since the study design was a retrospective study, missing events may potentially occur. However, stroke was a serious illness and almost always led to hospitalization, the chance of missing events was low. Third, we calculated the risk scores at the initial follow-up visit. In reality, dynamic changes in risk may occur over time. A large change in risk score may potentially affect our findings. Certain issues not captured by the risk scores (i.e., drug non-compliance, drug-drug or drug-herb interactions, genetic polymorphism) may also affect the finding. Fourth, we investigated the performance of these risk scores only in ischemic stroke, therefore results cannot be applied in other stroke types. It should be emphasized that the prognosis of stroke recurrence is different in ischemic stroke subtypes. For example, cognitive impairment has been shown to be a frequent finding in patients with multiple lacunar infarction recurrences [19]. For cardioembolic stroke, early recurrent embolization is the most important predictor of in-hospital mortality [20]. Lastly, although we found that both the SPI-II and ESRS can be used in the Thai population, more studies from other developing countries in Asia are needed.
Conclusion
Both the SPI-II and ESRS scores have similarly moderate predictive performances in predicting recurrent stroke among Asian who experienced and survived acute ischemic stroke in a developing health system. As a result, both risk scores are useful to clinicians practicing in developing countries where stroke recurrence is higher than in developed countries. Further studies with larger sample size from other developing countries in the region should be conducted to confirm our findings.
Data Availability
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
References
Global regional. National age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980–2017: a systematic analysis for the global burden of Disease Study 2017. Lancet. 2018;392(10159):1736–88.
Health Statistics. National Statistical Office. Ministry of Digital Economy and Society. http://statbbi.nso.go.th/staticreport/page/sector/th/05.aspx. Access verified on August 20, 2022.
Global regional. National disability-adjusted life-years (DALYs) for 359 diseases and injuries and healthy life expectancy (HALE) for 195 countries and territories, 1990–2017: a systematic analysis for the global burden of Disease Study 2017. Lancet. 2018;392(10159):1859–922.
Venketasubramanian N, Yoon BW, Pandian J, Navarro JC. Stroke epidemiology in South, East, and South-East Asia: a review. J Stroke. 2017;19(3):286–94.
Chin YY, Sakinah H, Aryati A, Hassan BM. Prevalence, risk factors and secondary prevention of stroke recurrence in eight countries from south, east and southeast asia: a scoping review. Med J Malaysia. 2018;73(2):90–9.
Flach C, Muruet W, Wolfe CDA, Bhalla A, Douiri A. Risk and secondary Prevention of Stroke recurrence: a Population-Base Cohort Study. Stroke. 2020;51(8):2435–44.
Chen Y, Wright N, Guo Y, Turnbull I, Kartsonaki C, Yang L, et al. Mortality and recurrent vascular events after first incident stroke: a 9-year community-based study of 0·5 million chinese adults. Lancet Glob Health. 2020;8(4):e580–e90.
Amarenco P, Lavallée PC, Monteiro Tavares L, Labreuche J, Albers GW, Abboud H, et al. Five-year risk of stroke after TIA or minor ischemic stroke. N Engl J Med. 2018;378(23):2182–90.
Chaudhary D, Abedi V, Li J, Schirmer CM, Griessenauer CJ, Zand R. Clinical risk score for Predicting Recurrence following a cerebral ischemic event. Front Neurol. 2019;10:1106.
Kernan WN, Viscoli CM, Brass LM, Makuch RW, Sarrel PM, Roberts RS, et al. The stroke prognosis instrument II (SPI-II): a clinical prediction instrument for patients with transient ischemia and nondisabling ischemic stroke. Stroke. 2000;31(2):456–62.
Weimar C, Diener HC, Alberts MJ, Steg PG, Bhatt DL, Wilson PW, et al. The Essen stroke risk score predicts recurrent cardiovascular events: a validation within the REduction of atherothrombosis for continued health (REACH) registry. Stroke. 2009;40(2):350–4.
Meng X, Wang Y, Zhao X, Wang C, Li H, Liu L, et al. Validation of the Essen stroke risk score and the stroke prognosis instrument II in chinese patients. Stroke. 2011;42(12):3619–20.
Yang F, Yan S, Wang W, Li X, Chou F, Liu Y, et al. Recurrence prediction of Essen Stroke Risk and Stroke Prognostic Instrument-II scores in ischemic stroke: a study of 5-year follow-up. J Clin Neurosci. 2022;104:56–61.
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.
Weimar C, Benemann J, Michalski D, Müller M, Luckner K, Katsarava Z, et al. Prediction of recurrent stroke and vascular death in patients with transient ischemic attack or nondisabling stroke: a prospective comparison of validated prognostic scores. Stroke. 2010;41(3):487–93.
Amarenco P, Goldstein LB, Sillesen H, Benavente O, Zweifler RM, Callahan A 3, et al. Coronary heart disease risk in patients with stroke or transient ischemic attack and no known coronary heart disease: findings from the Stroke Prevention by Aggressive reduction in cholesterol levels (SPARCL) trial. Stroke. 2010;41(3):426–30.
Lip GYH, Genaidy A, Tran G, Marroquin P, Estes C, Sloop S. Improving stroke risk prediction in the General Population: a comparative Assessment of Common Clinical Rules, a New Multimorbid Index, and machine-learning-based algorithms. Thromb Haemost. 2022;122(1):142–50.
Li X, Wang Y, Xu J. Development of a machine learning-based risk prediction model for cerebral infarction and comparison with nomogram model. J Affect Disord. 2022;314:341–8.
Arboix A, Font A, Garro C, García-Eroles L, Comes E, Massons J. Recurrent lacunar infarction following a previous lacunar stroke: a clinical study of 122 patients. J Neurol Neurosurg Psychiatry. 2007;78(12):1392–4.
Arboix A, Oliveres M, Massons J, Pujades R, Garcia-Eroles L. Early differentiation of cardioembolic from atherothrombotic cerebral infarction: a multivariate analysis. Eur J Neurol. 1999;6(6):677–83.
Acknowledgements
We would like to acknowledge the support of the Division of Neurology, Department of Medicine, Pharmacy Department of Siriraj Hospital, and Siriraj Medical Research Center which made this study possible.
Funding
The authors declare that no funds, grants, or other support was received during the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
J.K., A.S., C.K. and S.N. contributed to the conception and design. J.K. and A.S. performed data collection. J.K., A.S., C.K. and S.N. performed data analysis and data interpretation. A.S. prepared all figures. S.N. wrote the manuscript. All authors reviewed the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The study was approved by the Institutional Review Board of Siriraj Hospital (IRB Number: Si642/2019, date of approval: September 17, 2019) and followed the principles of the Declaration of Helsinki. The informed consent of patients was waived by the Institutional Review Board of Siriraj Hospital due to the retrospective nature of the study.
Consent for publication
Not applicable.
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Kongwatcharapong, J., Sornkhamphan, A., Kaveeta, C. et al. Validation and comparison of the stroke prognosis instrument (SPI-II) and the essen stroke risk score (ESRS) in predicting stroke recurrence in Asian population. BMC Neurol 23, 287 (2023). https://doi.org/10.1186/s12883-023-03329-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12883-023-03329-w