Case Study: Patient Risk Stratification¶

Case Study

Key Result: 18% reduction in 30-day readmissions, $12M annual cost savings, proactive care coordination for high-risk patients.


Agency Type	Health Department
Domain	Healthcare
Challenge	Identifying high-risk patients for proactive intervention
AI Approach	Risk stratification model with gradient boosting

Executive Summary¶

A state health department implemented an AI-based patient risk stratification model to identify individuals at high risk of hospital readmission. The system enabled proactive care coordination, reducing 30-day readmissions by 18% and generating $12M in annual cost savings.

The Challenge¶

Situation¶

2.5 million patient records in state health system
15% 30-day readmission rate (above national average)
Reactive care model waiting for patients to present
Limited resources for care coordination
No systematic way to identify high-risk patients

Problems¶

Readmissions costly to health system and patients
Care coordinators couldn't prioritize effectively
Risk factors identified too late for intervention
Manual risk assessment was inconsistent
Social determinants of health not considered

Business Impact¶

$85M annual cost of preventable readmissions
Patient outcomes below national benchmarks
Strained hospital capacity
Staff burnout from crisis-mode care

The Solution¶

AI Approach¶

Model Type: Binary classification (readmission risk) Architecture: Gradient Boosted Trees (XGBoost) Integration: Electronic Health Record system

System Design¶

flowchart LR
    subgraph IN["<strong>Patient Discharge</strong>"]
        I1[Hospital Data]
    end

    subgraph FE["<strong>Feature Extraction</strong>"]
        F1[Clinical Factors]
        F2[Social Factors]
    end

    subgraph MODEL["<strong>Risk Model</strong>"]
        M1[Risk Score 0-100]
        M2[Explain Factors]
    end

    subgraph OUT["<strong>Care Team Dashboard</strong>"]
        O1[Prioritized Worklist]
    end

    IN --> FE --> MODEL --> OUT

    MODEL --> HIGH[🔴 High Risk<br/>Immediate outreach]
    MODEL --> MED[🟡 Medium Risk<br/>Scheduled follow-up]
    MODEL --> LOW[🟢 Low Risk<br/>Standard care]

    style IN fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style FE fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style MODEL fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style OUT fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style HIGH fill:#ef9a9a,stroke:#c62828,stroke-width:2px
    style MED fill:#fff9c4,stroke:#f9a825,stroke-width:2px
    style LOW fill:#c8e6c9,stroke:#388e3c,stroke-width:2px

Risk Factors Considered¶

Clinical Factors: - Diagnosis codes (ICD-10) - Length of stay - Prior hospitalizations (12 months) - Comorbidity index - Medication count - Lab values

Social Determinants: - Housing stability - Social support - Transportation access - Health literacy - Food security

Engagement Factors: - Missed appointments - ER utilization - Primary care engagement - Medication adherence

Key Design Decisions¶

Decision	Choice	Rationale
Model type	XGBoost	Interpretable, handles mixed data
Prediction window	30 days post-discharge	Clinically meaningful
Risk tiers	High/Medium/Low	Actionable for care teams
Explainability	SHAP values	Clinicians need to understand why
Update frequency	Daily	Near real-time risk

Implementation¶

Timeline¶

Phase	Duration	Activities
Discovery	8 weeks	Clinical input, data assessment, ethics review
Data preparation	12 weeks	Data extraction, feature engineering, validation
Model development	14 weeks	Training, validation, clinical testing
Integration	10 weeks	EHR integration, workflow design
Clinical pilot	12 weeks	Two hospitals, process refinement
State rollout	16 weeks	Phased deployment to all hospitals
Total	72 weeks

Team¶

Role	FTE	Responsibility
Clinical Lead (physician)	0.5	Clinical validation, workflow
Product Owner	1.0	Requirements, stakeholder management
Data Scientist	2.0	Model development
Data Engineer	1.5	Data pipelines, infrastructure
EHR Analyst	1.0	EHR integration
Clinical Informaticist	1.0	Clinical data standards
Ethics Lead	0.3	Ethics review, bias assessment
Change Manager	0.5	Clinical adoption

Data Preparation¶

Data Sources: - Electronic Health Records (3 years) - Claims data - Social determinants (linked survey data) - Pharmacy records - External mortality data (labels)

Feature Engineering: - 287 raw features extracted - 156 features after selection - Temporal features (trends over time) - Interaction features (comorbidity combinations)

Challenges: - Missing social determinants data (30%) - EHR data quality varied by hospital - Label leakage risk with certain features

Solutions: - Imputation strategy with clinical input - Data quality scoring and filtering - Careful temporal splitting to prevent leakage

Results¶

Performance Metrics¶

Metric	Value
AUC-ROC	0.78
Precision (top 10%)	0.45
Recall (top 10%)	0.38
Calibration (Brier)	0.12
Positive Predictive Value	42%

Clinical Impact¶

Metric	Before	After	Improvement
30-day readmission rate	15.2%	12.5%	-18%
Care coordination calls	2,400/mo	8,500/mo	+254%
High-risk patients engaged	15%	68%	+353%
Average time to follow-up	14 days	3 days	-79%

Financial Impact¶

Item	Annual Value
Readmissions prevented	3,200
Cost per readmission	$15,000
Gross savings	$48,000,000
Program costs	$2,400,000
Care coordination costs	$8,600,000
Net savings	$37,000,000
ROI	336%

Fairness Analysis¶

Demographic	AUC	PPV	Disparity
Overall	0.78	0.42	-
Age <65	0.76	0.38	Pass
Age 65+	0.79	0.45	Pass
Urban	0.78	0.43	Pass
Rural	0.77	0.40	Pass
Indigenous	0.75	0.39	Monitor

Challenges and Lessons Learned¶

Challenge 1: Clinician Adoption¶

Issue: Initial skepticism from clinicians about AI predictions Solution: - Involved clinicians in model development - Showed SHAP explanations for each prediction - Positioned as decision support, not replacement Lesson: Clinician trust requires transparency and involvement

Challenge 2: Workflow Integration¶

Issue: Care coordinators already overwhelmed, couldn't add tasks Solution: Redesigned workflow to replace existing processes Lesson: AI must fit into workflow, not add to it

Challenge 3: Data Quality¶

Issue: Social determinants data missing for 30% of patients Solution: - Imputation model for missing values - Flagged predictions with missing data - Launched data collection improvement initiative Lesson: Accept imperfect data, but plan for improvement

Challenge 4: Ethical Concerns¶

Issue: Concerns about resource allocation based on AI Solution: - Ethics committee oversight - Clear policy that AI informs but doesn't decide - Regular fairness audits Lesson: Proactively address ethics, don't wait for problems

Challenge 5: Model Degradation¶

Issue: Performance dropped during COVID-19 Solution: - Rapid retraining on recent data - Added COVID-specific features - Enhanced monitoring Lesson: Build for adaptability, not just initial performance

Governance and Compliance¶

Governance Structure¶

Executive sponsor: Chief Medical Officer
Clinical governance committee oversight
Ethics committee review and ongoing oversight
Risk tier: Tier 3 (High)

Compliance Measures¶

Health Information Privacy compliance
Ethics approval from institutional review board
Model card published to clinical community
Regular fairness audits (quarterly)
Annual external review

Human Oversight¶

Risk scores advisory only
Clinicians make all care decisions
Patients can opt-out of program
Appeal process for risk classification

Transparency¶

Patients informed about risk stratification program
Explanation of factors provided to care teams
Published methodology to clinical journals
Open to external research review

Technical Details¶

Model Specifications¶

Algorithm: XGBoost (Gradient Boosted Trees)
Features: 156 clinical and social factors
Training data: 450,000 discharge events
Validation: Temporal split (2019-2020 train, 2021 test)
Regularization: L1 and L2, max depth limits

Infrastructure¶

Training: On-premise HPC cluster (data sovereignty)
Serving: Health department private cloud
Integration: HL7 FHIR API to EHR
Batch scoring: Nightly for new discharges
Monitoring: Custom dashboard + alerts

Key Features (Top 10)¶

Rank	Feature	Importance	Direction
1	Prior admissions (12mo)	15.2%	↑ risk
2	Comorbidity index	11.8%	↑ risk
3	Length of stay	8.4%	↑ risk
4	Medication count	6.7%	↑ risk
5	Days since last admission	6.1%	↓ risk
6	Social support score	5.8%	↓ risk
7	Primary care visits (12mo)	5.2%	↓ risk
8	Age	4.9%	↑ risk
9	Discharge disposition	4.3%	Varies
10	Housing stability	3.8%	↓ risk

Recommendations for Similar Projects¶

Do¶

Involve clinicians from day one
Prioritize explainability over marginal accuracy gains
Test for fairness across demographic groups
Design for workflow integration
Build robust monitoring for drift
Plan for external validation

Don't¶

Deploy without clinical validation
Ignore social determinants
Use AI for decisions (only decision support)
Skip ethics review
Assume stable model performance
Forget about patient consent/communication

Cost-Benefit Summary¶

Costs (First Year)¶

Item	Cost
Discovery & planning	$150,000
Data preparation	$280,000
Model development	$350,000
Integration	$220,000
Clinical pilot	$180,000
Change management	$120,000
Infrastructure	$100,000
Total Year 1	$1,400,000

Ongoing Costs (Annual)¶

Item	Cost
Infrastructure	$150,000
Model maintenance	$200,000
Clinical support	$150,000
Monitoring & audit	$100,000
Total Annual	$600,000

Benefits (Annual)¶

Item	Value
Readmission reduction	$48,000,000
Less: Care coordination	($8,600,000)
Less: Program costs	($600,000)
Annual Net Benefit	$38,800,000

ROI: 2,671% | Payback: 2 months (after full rollout)¶

Contact¶

For more information about this case study, contact the AI Toolkit team.

Related documents: How to Bias Testing | Model Card Template