Affiliations: 1. Verantos Inc., Menlo Park, CA, USA; 2. Lundbeck, Deerfield, IL, USA; 3. Stanford University School of Medicine, Stanford, CA, USA; 4. StudyMetrix Research, LLC, St. Peters, MO, USA
Objective:
To develop and validate an algorithm for identifying concepts associated with migraine presence and progression in electronic health records (EHR) for use in real-world evidence studies
Design/Setting:
In a retrospective observational study, migraine concepts, migraine-related symptoms (dizziness, fatigue, light sensitivity, loss of appetite, nausea and vomiting), and migraine medication use (sumatriptan and rizatriptan) were identified in EHR using traditional (ICD codes, medication list, referred to as structured/EHR-S) and artificial intelligence (AI) techniques (eg, natural language processing [NLP], machine-learned inference) in unstructured data (EHR-U). These were compared to the reference standard of migraine diagnosis using chart abstraction by 2 clinicians.
Patients:
Patients with migraine (diagnostic agreement of independent chart review by 2 clinicians)
Main outcome measures:
Recall (proportion correctly identified), precision (proportion correctly identified/total identified), and F1-scores (the weighted harmonic mean of precision and recall)
Results:
For the term migraine (2642 occurrences), recall and precision were 96.8% and 98.0% for EHR-U and 66.6% and 99.5% for EHR-S; F1 scores were 97.4% for EHR-U and 79.3% for EHR-S (P<0.001). For migraine-related symptoms (243–4088 occurrences), recall and precision were 79.3%–96.6% and 81.0%–97.6% for EHR-U, and 0%–17.9% and 74.7%–100% for EHR-S; F1 scores were 80.7%–95.6% for EHR-U and 0%–28.9% for EHR-S (P<0.001). For migraine medications (102 and 510 occurrences), recall and precision were 98.0%–98.4% and 98.9%–100% for EHR-U, and 78.6%–81.6% and 100% for EHR-S; F1 scores were 98.7%–99.0% for EHR-U and 88.0%–89.9% for EHR-S (P<0.001).
Conclusions:
AI-based algorithms (eg, NLP, machine-learned inference) can identify migraine-related concepts and patients with migraine based on terminology and related symptoms. EHR-S is sufficient for evaluating pharmacologic interventions. Further study of application of these methods to identify progression and treatment response can support development of real-world evidence regarding effectiveness of migraine prevention and treatment.
Support:
Lundbeck
Presenting author information:
Name: Steven Kymes, PhD, MHA
Position: Director, Health Economics & Outcomes Research
Institution: Lundbeck
Address: 6 Parkway North, Deerfield, IL 60015
Phone: 224-727-7548
Email: SKYM@lundbeck.com
Author contact information
Roger Cady, MD
email: ROCD@lundbeck.com
Address: Lundbeck, 6 Parkway North, Deerfield, IL 60015, USA
Phone: 425-205-2900
Daniel Riskin, MD, FACS
email: science@verantos.com
Address: Verantos Inc., 325 Sharon Park Dr, #730, Menlo Park, CA 94025, USA
Phone: 650-777-7978
Anand Shroff, MS, MBA
email: anand.shroff@verantos.com
Address: Verantos Inc., 325 Sharon Park Dr, #730, Menlo Park, CA 94025, USA
Phone: 650-394-7130
Nada Hindiyeh, MD
email: nhindiye@stanford.edu
Address: Stanford University School of Medicine, 211 Quarry Road, Palo Alto, CA, 94304
Phone: 650-723-6469
Timothy Smith, MD, RPh, FACP, AQH
email: tsmith@studymetrix.com
Address: StudyMetrix Research, LLC, 3862 Mexico Rd, St. Peters, MO 63303, USA
Phone: 636-387-5100
Steven Kymes, PhD, MHA
email: SKYM@lundbeck.com
Address: Lundbeck, 6 Parkway North, Deerfield, IL 60015, USA
Phone: 224-727-7548
