Newsletter from The Neural Medwork | Issue #23

Bridging AI and Healthcare

Welcome Back to The Neural Medwork! Last issue we focused on understanding natural language processing models and created a practical example of using OpenEvidence when assessing a patient for anticoagulation. In this issue, we focus on three key topics at the intersection of AI and healthcare: the ELO scoring system and how it helps rank AI models, a study about how LLMs generate differential diagnoses, and a practical guide to using MedArena to select the right AI model for your clinical needs.

---

AI Concept: Understanding the ELO Scoring System in AI Performance

When looking at AI model leaderboards like MedArena, you'll often see models ranked by their "ELO score." But what exactly is this score, and how should you interpret it?

What is an ELO Score?

The ELO rating system was originally created for ranking chess players but is now widely used to evaluate AI models. Unlike simple accuracy metrics, ELO is a comparative ranking system that shows how models perform against each other.

How to Interpret ELO Scores:

·       All AI models on MedArena start with a base score of 1000 points

·       Current models on the leaderboard typically have scores ranging from approximately 900 to 1100

·       Higher scores indicate stronger performance in head-to-head comparisons

·       Even small differences of 20-30 points can represent meaningful performance differences

·       The score differences are more important than the absolute numbers

Understanding Relative ELO Performance:

Rather than focusing on specific thresholds, it's more helpful to consider:

1.     How far above 1000 a model scores (showing it wins more often than it loses)

2.     The gap between models (wider gaps indicate clearer performance differences)

3.     The confidence intervals (narrower intervals suggest more reliable ratings)

4.     Specialty-specific performance (some models excel in certain areas despite lower overall scores)

Chatbots are commonly made to compete against one another to determine which AI tools have the best usage and are ranked with ELO scores to create a leaderboard.

What Factors Influence a Model's ELO Score?

1.     Model performance in generating accurate clinical information

2.     Clinical relevance of responses to medical queries

3.     Consistency across different medical specialties and question types

4.     Human evaluator preferences when comparing outputs side-by-side

The key advantage of ELO is that it's dynamic - as models improve or new models emerge, the ratings adjust automatically. This gives healthcare professionals an up-to-date view of which AI tools are most reliable.

Research Spotlight: Evaluating LLMs in Differential Diagnosis

Paper: Large Language Models Can Generate Differential Diagnoses but May Not Improve Clinical Decision-Making (Nature, 2025)

This study assessed how LLMs perform in generating differential diagnoses and whether they actually enhance medical decision-making using real patient cases across multiple specialties.

Key Findings:

• LLMs generated comprehensive differentials but often struggled to prioritize the most relevant diagnoses

• Performance varied significantly by specialty, with higher accuracy in internal medicine and lower accuracy in emergency medicine and rare diseases

• Clinicians found LLM input useful as a "second opinion" but noted that incorrect AI suggestions could potentially introduce bias

Implications for AI in Medicine:

• LLMs are valuable as clinical reference tools but should not be relied upon for autonomous diagnosis

• The most effective implementation involves human oversight, with clinicians filtering AI-generated differentials based on their expertise

• Ongoing refinement is necessary to improve the prioritization and context-awareness of AI-generated diagnoses

This study highlights why tools like MedArena are essential - they help clinicians identify which models perform best for their specific use cases.

---

Practical Example: Comparing AI Models on MedArena for Clinical Decision-Making

Step 1: Accessing the MedArena Leaderboard

The MedArena leaderboard ranks LLMs based on their clinical performance, with an ELO-based scoring system.

Step 2: Comparing Top-Ranked Models

Below are real-world evaluations of different AI models currently tested on MedArena:

Model	ELO Score
Model A (openai/gpt-4o-2024-11-20)	1074
Model B (perplexity/llama-3.1-sonar-large-128k-online)	1019
Model C (openai/o3-mini)	993

Step 3: Evaluating a Model for Emergency Medicine

A hospital AI team wants to select an AI model for emergency decision support, prioritizing:

1. Accuracy in acute conditions (e.g., stroke, myocardial infarction).

2. Strong differential diagnosis capabilities.

3. Robustness against misinformation.

4. Next steps in management

Example Case Input

"A 72-year-old woman presents with sudden-onset unilateral weakness and slurred speech. BP 185/110 mmHg. Last known well 2 hours ago. What is the likely diagnosis, and what is the next step?"

Model Comparisons

• Model A (ELO 1074): Correctly identifies ischemic stroke, prioritizes tPA eligibility, and suggests imaging confirmation, followed by next steps in management; along with discussion of blood pressure control, with recommended dosing of medication.

• Model B (ELO 1019): Correctly identifies ischemic stroke, prioritizes tPA eligibility, and suggests imaging confirmation; along with discussion of blood pressure control but does not provide suggested medications

• Model C (ELO 993): Correctly identifies ischemic stroke, but does not discuss next steps in management.

Step 4: Selecting the Best Model

To ensure you are picking the best model Model for your environment, ensure to test it on a vast array of clinical scenarios. You may want to work through common, and rare scenarios to ensure the model can perform well in various situations. A model with a very high Elo rating in one scenario may not be the best model for your clinical environment but its always best to start testing with models that have the highest Elo score on the leaderboard.

Key Takeaways from MedArena

• ELO scoring helps compare AI models dynamically, ensuring the most up-to-date rankings.

• Continuous monitoring is essential, as AI models improve over time and rankings shift.

---

Closing Thoughts

This issue explored how the ELO scoring system provides a meaningful way to evaluate AI models in medicine, ensuring that clinical AI tools are selected based on their comparative performance rather than marketing claims.

Key Takeaways:

1. ELO scores offer a convenient ranking system that helps compare AI models based on their performance against each other

2. LLMs can generate comprehensive differentials but require human expertise to prioritize and contextualize results

3. MedArena provides practical tools for selecting AI models that align with your specialty and clinical needs

As AI continues to evolve, understanding how these models are evaluated becomes increasingly important for healthcare professionals looking to integrate these tools into their practice.

Stay tuned for our next issue, where we'll cover the latest advancements in AI for medical imaging and diagnostics.

Best,

Mohammad, Sameer and Michael