- Research
- Open access
- Published:
Machine learning characterization of a rare neurologic disease via electronic health records: a proof-of-principle study on stiff person syndrome
BMC Neurology volume 24, Article number: 272 (2024)
Abstract
Background
Despite the frequent diagnostic delays of rare neurologic diseases (RND), it remains difficult to study RNDs and their comorbidities due to their rarity and hence the statistical underpowering. Affecting one to two in a million annually, stiff person syndrome (SPS) is an RND characterized by painful muscle spasms and rigidity. Leveraging underutilized electronic health records (EHR), this study showcased a machine-learning-based framework to identify clinical features that optimally characterize the diagnosis of SPS.
Methods
A machine-learning-based feature selection approach was employed on 319 items from the past medical histories of 48 individuals (23 with a diagnosis of SPS and 25 controls) with elevated serum autoantibodies against glutamic-acid-decarboxylase-65 (anti-GAD65) in Dartmouth Health’s EHR to determine features with the highest discriminatory power. Each iteration of the algorithm implemented a Support Vector Machine (SVM) model, generating importance scores—SHapley Additive exPlanation (SHAP) values—for each feature and removing one with the least salient. Evaluation metrics were calculated through repeated stratified cross-validation.
Results
Depression, hypothyroidism, GERD, and joint pain were the most characteristic features of SPS. Utilizing these features, the SVM model attained precision of 0.817 (95% CI 0.795–0.840), sensitivity of 0.766 (95% CI 0.743–0.790), F-score of 0.761 (95% CI 0.744–0.778), AUC of 0.808 (95% CI 0.791–0.825), and accuracy of 0.775 (95% CI 0.759–0.790).
Conclusions
This framework discerned features that, with further research, may help fully characterize the pathologic mechanism of SPS: depression, hypothyroidism, and GERD may respectively represent comorbidities through common inflammatory, genetic, and dysautonomic links. This methodology could address diagnostic challenges in neurology by uncovering latent associations and generating hypotheses for RNDs.
Background
Of more than 6,000 rare diseases that affect about 6% of the global population [1, 2], at least half are life-threatening or chronically debilitating dysfunctions of the central or peripheral nervous system [3]. However, due to the medical communities’ limited expertise in unusual disease presentations and insufficient diagnostic tools, rare neurologic diseases (RNDs) are frequently detected with a delay. Despite the socioeconomic burden of such diseases, the disorders’ rarity hinders the accrual of sufficient statistical power appropriate for systematic exploration of their pathological and clinical features, which are crucial for optimizing diagnostic accuracy and precision [4].
Recent advancements in artificial intelligence and machine learning (ML) have introduced novel strategies to study the pathologies and treatment outcomes for RNDs. Applying ML to electronic health records (EHR) is demonstrating increasing promise in the accurate detection of rare diseases and is gaining traction as decision support systems due to its strengths in extracting knowledge and identifying otherwise elusive associations [5,6,7]. Such capabilities are offering opportunities to mine clinical data, even retrospectively, to augment the diagnostic clarification and confirmation of rare diseases and validate co-occurring characteristics amongst affected individuals [8]. These insights bolster the case for ML’s increasingly larger role in decoding rare diseases and perhaps even inspiring fresh hypotheses based on newly unearthed correlations.
Stiff person syndrome (SPS) is an RND characterized by axial muscle stiffness and painful spasms, affecting only one to two in a million annually [9, 10]. While hallmark electromyography findings, symptomatic alleviation from benzodiazepine therapy, and the presence of autoantibodies against glutamic-acid-decarboxylase-65 (anti-GAD65) suggest SPS, diagnosis still relies on subjective clinical judgment, given the condition’s variable and poorly understood presentation [11, 12]. This ambiguity often leads to misdiagnosis and significant delays in diagnostic confirmation (mean delay of 6.2 years) and treatment initiation. Like many other RNDs, the rarity of SPS leads to insufficient statistical power, challenging the identification of relevant disease interrelationships. In this retrospective proof-of-principle study, we showcased the utility of ML models on EHR to overcome the constraints of conventional statistical tests and highlight the capability of such models to identify potential clinical features that may bear comorbid associations in the pathology of SPS.
Methods
Subject characteristics
To date, there exists no consensus on the ‘ideal’ sample size in ML analyses of EHR-based clinical data [13]. While higher sample sizes minimize type II errors, rare diseases are prone to mislabeling which can exacerbate type I errors, which are equally, if not more worrisome than insufficiencies in statistical power [14]. To that end, in this single-center retrospective study, we maintained a strict inclusion criterion with carefully selected patients with SPS and controls based on positive serum antibody titers to avoid any mislabeling of data and preserve model performance. Thus, our retrospective cohort consisted of 48 patients who were seen at Dartmouth Health from January, 2010 to January, 2020 and had elevated serum anti-GAD65 titers (≥ 0.03 nmol/L). Of these 48 patients, 23 were diagnosed with SPS by neurologists. All subjects’ past medical history (problem list) items from the EHR were binarized (1 if present; 0 otherwise), resulting in 319 unique items. This study was approved by the Committee for the Protection of Human Subjects at Dartmouth Health (STUDY02000166) and followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.
Data analysis and feature selection
Patient demographics as well as relevant binarized clinical characteristics based on previous literature on comorbidities of SPS (e.g., autoimmune processes) were evaluated via t-tests and Fisher’s exact tests in SPSS [15, 16].
To identify the key features that best characterize SPS among the 319 binarized items, we applied a contribution selection algorithm (CSA), an iterative process that optimizes a classifier’s performance by progressively evaluating and reranking features [17]. We adopted a backward elimination approach with a support vector machine (SVM) classification model featuring a Gaussian kernel for its superior versatility relative to linear or polynomial kernel functions [18].
Beginning with 319 features, each iteration of the CSA performs three tasks: 1) computes the SHapley Additive exPlanation (SHAP) values, which reflect the relative contribution of individual features, 2) assesses classifier performance, and 3) eliminates the feature with the lowest SHAP value. Derived from cooperative game theory, a feature’s SHAP value signifies the feature’s impact on the model’s prediction, reflecting the shifts in prediction in response to the presence or absence of that feature [19].
Using the “shap” package and custom scripts in Python3, we implemented our algorithm, iterating through 319 models of progressively fewer features [19]. We overcame our small sample size limitation by rigorously training our Gaussian kernel-based SVM model with both SPS and non-SPS patient cases and validating its predictions via a repeated-stratified fourfold validation scheme. Performance metrics and 95% confidence intervals for each model were calculated from 200 rounds of bootstrapping. The features employed by the highest-performing model were subsequently identified, and their respective impact on the models’ decision was visualized. The overall architecture for our feature identification pipeline is shown in Fig. 1.
Architectural overview of support vector machine-based contribution selection algorithm (CSA) pipeline. After binarization of 319 medical history items from all subjects, each iteration of CSA computes the SHapley Additive exPlanation (SHAP) values for each feature, assesses the model’s performance in classifying patients with stiff person syndrome vs controls, and eliminates feature with the lowest SHAP value. Features in the best-performing model are identified
Results
Subject characteristics
Baseline demographics and relevant clinical features of 23 patients with SPS and 25 anti-GAD65 controls were analyzed. No statistically significant differences were detected in terms of demographics, serum anti-GAD65 titer, or clinical/psychiatric characteristics/comorbidities (p > 0.05) (Table 1).
Identification of relevant features
For each step of CSA, an SVM model was trained using the remaining features and validated using a stratified fourfold cross-validation scheme. Generally, filtering out non-discriminatory features from the models increased prediction metrics, as expected.
The model's performance was enhanced through iterative removal of non-discriminatory features, as depicted in Fig. 2A-C. Area under the curve (AUC) reached its highest peak of 0.808 (95% CI, 0.791 to 0.825) when using 5 features: depression, joint pain, hypothyroidism, gastroesophageal reflux disease (GERD), and asthma. Sensitivity of 0.766 (95% CI, 0.743 to 0.790), F1-score of 0.761 (95% CI, 0.744 to 0.778), and accuracy of 0.775 (95% CI, 0.759 to 0.790) were achieved with the subsequent feature set, after excluding asthma. Finally, precision reached a peak of 0.817 (95% CI, 0.795 to 0.840), with the following feature set, after excluding joint pain. For model interpretation, we focused on the model involving 4 features (depression, joint pain, hypothyroidism, and GERD), as it achieved the highest number of best performing metrics.
Determination of Ideal Feature Set Through Evaluation of Model Performance. Model performance was calculated at every iteration of the backward elimination feature selection process. A F1-score and sensitivity are optimized at four features with 0.761 (95% CI, 0.744 to 0.778) and 0.766 (95% CI, 0.743 to 0.790), respectively, while precision reached a peak with three features with 0.817 (95% CI, 0.795 to 0.840). B Area under the curve (AUC) is optimized at five features with 0.808 (95% CI, 0.791 to 0.825). C Accuracy is optimized at four features with 0.775 (95% CI, 0.759 to 0.790)
Model interpretation
Throughout the feature selection process, the SHAP values of each feature were calculated. A positive SHAP value of a feature indicates a shift in the model’s decision to characterize an individual as having SPS. To visualize the impact of identified features (depression, joint pain, hypothyroidism, GERD) on the optimal model’s decision to characterize patient as having SPS or not, their SHAP values were plotted in a beeswarm plot (Fig. 3A). It is noteworthy that the absence of each of these four items consistently influenced the classifier's decision toward anti-GAD controls. Figure 3B shows bar graphs calculated from averaged SHAP values indicating the absolute impact of the features, regardless of their directional contribution. Depression emerged as the final remaining feature, highlighting its dominant influence in characterizing SPS.
Beeswarm plot and bar graph summarizing SHapley Additive explanation (SHAP) values of the optimal model. A The beeswarm plot presents the SHAP values for key features—namely depression, hypothyroidism, GERD, joint pain—ranked according to their relative valence. This set of features was selected from the model with the highest performance indices. Each dot represents a predicted case, and color reflects the original value of a feature (red = feature present, blue = feature absent). For each prediction, a positive SHAP value on the x-axis indicates an increase in the model’s tendency to characterize the patient as having stiff person syndrome (SPS) and vice versa. Notably, the absence of any of these features tends to steer the classifier's prediction towards anti-GAD65 controls. B The bar graph demonstrates the overall absolute impact of the selected features in the models’ decision-making process. Depression emerges as the feature with the highest impact on the optimized model’s decision. Collectively, the plots visually underscore the discriminative power of these features in SPS diagnosis
Discussion
We leveraged EHR and employed a data-driven feature selection approach (i.e. CSA) to identify medical history items that optimally characterize SPS. By using ML models to classify patients with SPS and anti-GAD65 controls, our methodology allowed us to determine clinical features that may be linked with the diagnosis of SPS: depression, hypothyroidism, GERD, and joint pain. This study, to the best of our knowledge, is the first attempt to identify features and disease interrelationships from otherwise underutilized health information in a primary RND.
Our findings align with results from comorbidity studies and case reports pertaining to SPS [16, 20, 21]. Depression emerged as the most characteristic feature, which corroborates the established higher risk of depression and other psychiatric comorbidities in patients with SPS compared to the general population [16]. Given the chronic nature of SPS, the persistent pain these patients endure may render them more susceptible to depression [22]. Hence, characterizing the quality and progression of depression patients with SPS experience (e.g. depressed mood from chronic pain, anhedonia, sleep difficulties, etc.) may help guide the diagnosis of SPS. Furthermore, there is growing evidence that inflammatory neurologic disorders, like multiple sclerosis, may increase the risk of affective dysregulation [23, 24]. Given the presence of depression as the strongest characterizer of SPS, studying the relationship between neuro-inflammation and depression in SPS patients may lead to more targeted interdisciplinary therapies.
The primary utility of our approach is less to develop a diagnostic algorithm for immediate use, but to generate new hypotheses for RNDs even when the pathological mechanisms are not yet fully characterized. This is exemplified by our identification of hypothyroidism as one of the final discriminative features of SPS. This not only supports the suspected autoimmune pathogenesis of SPS [12], but also prompts further inquiry to examine a potential genetic association between SPS, type I diabetes mellitus, and autoimmune thyroid disorders (e.g. Hashimoto thyroiditis) through a common HLA subtype. Similarly, our identification of GERD, which has been reported to be significantly associated with depression [21], as one of the final characteristic features of SPS highlights the utility of our framework in uncovering poorly understood extra-axial dysautonomic manifestation of SPS involving the gastrointestinal tract [25, 26]. As the recognized role of autoantibodies in multiple gastrointestinal dysmotility disorders, including anti-GAD65 in diseases implicated in GERD [26, 27], continues to increase, further research may shed light on a potential pathophysiology that is common in both central and autonomic nervous system dysfunctions of SPS.
RNDs are one of the most frequently studied groups of rare diseases [28]. However, patients with a broad spectrum of RNDs continue to experience delays (approximately 4 years) in accurate diagnosis following symptom onset, which is particularly concerning for illnesses where early intervention improves prognosis and quality of life [3]. ML is transforming medicine by unearthing previously unrecognized disease associations and enriching physicians’ fund of knowledge [7]. And together with artificial intelligence, such innovation synergistically complement physicians’ diagnostic precision and accuracy whilst optimizing their anticipation of relevant comorbidities. Our feature identification strategy via CSA represents a scalable application of ML in maximizing the utility of EHR to inform future research for RNDs, thereby circumventing the constraints of statistical underpowering. Together with hypothesis-driven investigations, the integration of ML can accelerate translational research and accommodate personalized care for patients with RNDs.
One limitation of our study is the high features-to-instances ratio intrinsic to the epidemiology of SPS, which amplified the complexity of feature selection. As a proof-of-principle study, we sought to mitigate this constraint, which also affects traditional statistical models, as well as the absence of an external validation cohort by implementing a repeated-stratified validation scheme within our available cohort. Some forms of SPS may be linked with other immune processes (e.g., myasthenia gravis or progressive encephalomyelitis with rigidity and myoclonus) or represent a paraneoplastic process, such as in the context of positive anti-amphiphysin antibodies [29,30,31]. Our sample size limitation restricts our ability to identify such associations. Future work should aim to optimize this features-to-instances ratio and externally validate the ML models by applying the identified feature set on more comprehensive data formed by multiple institutions and networks. Another shortcoming of our study pertains to the heterogeneity inherent to EHR. We addressed this by categorizing and, in some instances, combining some symptoms and diagnoses. For example, due to the paucity of cases under each category, depressive symptoms and major depressive disorder were consolidated into a single category: “depression.” Future interdisciplinary studies that incorporate multimodal data including neuropsychiatric questionnaires and electromyography findings will provide a more complete understanding of the individual subjects.
Conclusions
Our ML-based feature selection approach enabled the processing of underutilized EHR and identified pertinent medical items that warrant further research to fully characterize SPS: depression, hypothyroidism, and GERD may reflect comorbid conditions of SPS linked by inflammation, genetics, and the autonomic nervous system. This data-driven approach can complement more targeted investigations to explore RNDs and refine their diagnostic criteria.
Availability of data and materials
The data that support the results of this study are available from the corresponding author upon reasonable request.
Abbreviations
- RND:
-
Rare neurologic disease
- ML:
-
Machine learning
- EHR:
-
Electronic health records
- SPS:
-
Stiff person syndrome
- GAD-65:
-
Autoantibodies against glutamic acid decarboxylase 65
- CSA:
-
Contribution selection algorithm
- SHAP:
-
SHapley Additive exPlanation
- SVM:
-
Support vector machine
- AUC:
-
Area under the curve
- GERD:
-
Gastroesophageal reflux disease
References
EURORDIS. Rare disease impact report: insights from patients and the medical community 2013. Available from: https://www.eurordis.org/about-rare-diseases.
Nguengang Wakap S, Lambert DM, Olry A, Rodwell C, Gueydan C, Lanneau V, et al. Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database. Eur J Hum Genet. 2020;28(2):165–73.
Sintila SA, Boziki M, Bakirtzis C, Stardeli T, Smyrni N, Nikolaidis I, et al. The experience of a tertiary reference hospital in the study of rare neurological diseases. Medicina (Kaunas). 2023;59(2):266.
Gupta S, Faughnan ME, Tomlinson GA, Bayoumi AM. A framework for applying unfamiliar trial designs in studies of rare diseases. J Clin Epidemiol. 2011;64(10):1085–94.
Garcelon N, Burgun A, Salomon R, Neuraz A. Electronic health records for the diagnosis of rare diseases. Kidney Int. 2020;97(4):676–86.
Cohen AM, Chamberlin S, Deloughery T, Nguyen M, Bedrick S, Meninger S, et al. Detecting rare diseases in electronic health records using machine learning and knowledge engineering: case study of acute hepatic porphyria. PLoS One. 2020;15(7): e0235574.
Molnar MJ, Molnar V. AI-based tools for the diagnosis and treatment of rare neurological disorders. Nat Rev Neurol. 2023;19(8):455–6.
Lin S, Nateqi J, Weingartner-Ortner R, Gruarin S, Marling H, Pilgram V, et al. An artificial intelligence-based approach for identifying rare disease patients using retrospective electronic health records applied for Pompe disease. Front Neurol. 2023;14: 1108222.
Dalakas MC. Stiff person syndrome: advances in pathogenesis and therapeutic interventions. Curr Treat Options Neurol. 2009;11(2):102–10.
El-Abassi R, Soliman MY, Villemarette-Pittman N, England JD. SPS: understanding the complexity. J Neurol Sci. 2019;404:137–49.
Crispo JAG, Thibault DP, Fortin Y, Willis AW. Inpatient care for stiff person syndrome in the United States: a nationwide readmission study. J Clin Mov Disord. 2018;5:5.
Hadavi S, Noyce AJ, Leslie RD, Giovannoni G. Stiff person syndrome. Pract Neurol. 2011;11(5):272–82.
Rajput D, Wang WJ, Chen CC. Evaluation of a decided sample size in machine learning applications. BMC Bioinformatics. 2023;24(1):48.
Banerjee J, Taroni JN, Allaway RJ, Prasad DV, Guinney J, Greene C. Machine learning in rare disease. Nat Methods. 2023;20(6):803–14.
Baizabal-Carvallo JF, Jankovic J. Stiff-person syndrome: insights into a complex autoimmune disorder. J Neurol Neurosurg Psychiatry. 2015;86(8):840–8.
Caffrey D, Finn CT, Song SM, Burton F 3rd, Arsan C. Stiff-person syndrome and psychiatric comorbidities: a systematic review. J Acad Consult Liaison Psychiatry. 2021;62(1):3–13.
Cohen S, Dror G, Ruppin E. Feature selection via coalitional game theory. Neural Comput. 2007;19(7):1939–61.
Schölkopf B, Smola AJ. Learning with kernels: support vector machines, regularization, optimization, and beyond. Cambridge, MA and London, England: MIT press; 2002.
Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Adv Neural Inform Process Syst. 2017;30:4768–77.
Hegyi CA. Physical therapist management of stiff person syndrome in a 24-year-old woman. Phys Ther. 2011;91(9):1403–11.
Zamani M, Alizadeh-Tabari S, Chan WW, Talley NJ. Association between anxiety/depression and gastroesophageal reflux: a systematic review and meta-analysis. Am J Gastroenterol. 2023;24(4):593.
Okifuji A, Turk DC. Chronic pain and depression: vulnerability and resilience. In: Neuroscience of pain, stress, and emotion. Elsevier: Academic Press; 2016. 181–201.
Rosenblat JD, Cha DS, Mansur RB, McIntyre RS. Inflamed moods: a review of the interactions between inflammation and mood disorders. Prog Neuropsychopharmacol Biol Psychiatry. 2014;53:23–34.
Gold SM, Irwin MR. Depression and immunity: inflammation and depressive symptoms in multiple sclerosis. Immunol Allergy Clin North Am. 2009;29(2):309–20.
Louis-Jean S, Agrawal N, Chaudhry S, Mazer A. Biliary dyskinesia in stiff person syndrome: an association between reduced GABA production and gastroenteric dysmotility. J Community Hosp Intern Med Perspect. 2023;13(5):94.
Maier A, Mannartz V, Wasmuth H, Trautwein C, Neumann UP, Weis J, et al. GAD antibodies as key link between chronic intestinal pseudoobstruction, autonomic neuropathy, and limb stiffness in a nondiabetic patient: a CARE-compliant case report and review of the literature. Medicine (Baltimore). 2015;94(31): e1265.
Termsarasab P, Thammongkolchai T, Katirji B, Termsarasab P, Thammongkolchai T, Katirji B. Anti-glutamic acid decarboxylase 65 (GAD65)-associated syndromes. In: Stiff-person syndrome and related disorders: a comprehensive, practical guide. 2020. 55–71.
Schaefer J, Lehne M, Schepers J, Prasser F, Thun S. The use of machine learning in rare diseases: a scoping review. Orphanet J Rare Dis. 2020;15(1):145.
Nicholas AP, Chatterjee A, Arnold MM, Claussen GC, Zorn GL Jr, Oh SJ. Stiff-persons’ syndrome associated with thymoma and subsequent myasthenia gravis. Muscle Nerve. 1997;20(4):493–8.
Sarva H, Deik A, Ullah A, Severt WL. Clinical spectrum of stiff person syndrome: a review of recent reports. Tremor Other Hyperkinet Mov (N Y). 2016;6:340.
Martinez-Hernandez E, Arino H, McKeon A, Iizuka T, Titulaer MJ, Simabukuro MM, et al. Clinical and immunologic investigations in patients with stiff-person spectrum disorder. JAMA Neurol. 2016;73(6):714–20.
Funding
The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
Authors SHP, SHS, and FB contributed to the data collection, analysis, and interpretation of the results and to the composition of the manuscript. Authors CA contributed to the design of the analysis plan and supported the interpretation of the data. Authors BJ and MF contributed to the study design and edited the manuscript. All authors have critically reviewed the manuscript and approved the final version of the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
This study was approved by the Committee for Protection of Human Rights at Dartmouth College (STUDY02000166) and meets ethical standards in the Declaration of Helsinki. The need for patient consent was waived due to the retrospective nature of the study, the de-identified nature of the analyzed data, and the lack of patient interaction.
Consent for publication
Not applicable.
Competing interests
B.J receives research funding from NeuroPace, Inc. and Harvard Pilgrim, Inc. not relating to this research. The rest of the authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Park, S.H., Song, S.H., Burton, F. et al. Machine learning characterization of a rare neurologic disease via electronic health records: a proof-of-principle study on stiff person syndrome. BMC Neurol 24, 272 (2024). https://doi.org/10.1186/s12883-024-03760-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12883-024-03760-7