Algorithmic Inequity in Clinical AI: What Community Health Data Reveals That EHR Data Cannot

Clinical AI systems are increasingly used to support diagnosis, triage, and treatment recommendations across U.S. health systems. These systems are trained primarily on electronic health record data — data that systematically underrepresents the populations most affected by health inequity. This brief presents evidence from DaisyMe Foundation's community health programs on the gap between clinical AI performance as measured in controlled validation studies and real-world outcomes as experienced by underrepresented community members. We document four categories of algorithmic failure observed through community navigation data, describe the structural conditions that produce them, and recommend specific actions for health systems, clinical AI developers, and regulators.

Algorithmic Inequity in Clinical AI: What Community Health Data Reveals That EHR Data Cannot

DaisyMe Foundation Policy Brief DMF-2026-002 Published July 2026 · Frisco, TX

Abstract

Clinical AI systems are increasingly used to support diagnosis, triage, and treatment recommendations across U.S. health systems. These systems are trained primarily on electronic health record data — data that systematically underrepresents the populations most affected by health inequity. This brief presents evidence from DaisyMe Foundation's community health programs on the gap between clinical AI performance as measured in controlled validation studies and real-world outcomes as experienced by underrepresented community members. We document four categories of algorithmic failure observed through community navigation data, describe the structural conditions that produce them, and recommend specific actions for health systems, clinical AI developers, and regulators.

1. Background: The Training Data Problem

Every clinical AI system is only as equitable as the data it was trained on. And the data it was trained on was generated by a healthcare system that has, for generations, provided unequal care to unequal populations.

The consequences of this structural problem are not hypothetical. They are documented:

• A widely cited 2019 study published in Science found that a commercial algorithm used by health systems to identify patients for high-risk care management was significantly less likely to flag Black patients than white patients with the same health burden — because it used healthcare costs as a proxy for health need, and Black patients had historically received less care.
• A 2021 analysis in JAMA Internal Medicine found that pulse oximeters — a foundational diagnostic tool — were three times more likely to miss low oxygen levels in Black patients than in white patients, a bias that contributed to worse COVID-19 outcomes.
• A 2023 review in npj Digital Medicine found that the majority of clinical AI validation studies did not report performance disaggregated by race or ethnicity — meaning the equity performance of most deployed clinical AI systems is simply unknown.

These findings share a common structure: the bias is not visible in aggregate performance metrics. It only becomes visible when you disaggregate by the demographic groups that the training data underrepresents.

2. What Community Data Reveals

DaisyMe Foundation's community health programs generate a category of data that clinical validation studies do not: the experience of community members who have interacted with clinical AI-assisted systems and found them inadequate.

Between 2022 and 2026, DaisyMe navigators documented barrier events — instances where a community member encountered a specific obstacle to appropriate care. Among the 12,400+ individuals served, navigators documented four recurring categories of algorithmic failure.

2.1 Triage Underscoring

Community members presenting with symptoms consistent with serious conditions were assigned lower acuity scores by AI-assisted triage systems than their clinical presentation warranted. In documented cases, navigators observed:

• Patients with hypertensive urgency triaged to lower-acuity queues, resulting in multi-hour waits before evaluation
• Patients with chest pain and multiple cardiac risk factors assigned risk scores that did not reflect their actual risk profile
• Patients with sepsis indicators who were discharged from triage before evaluation

In each category, the affected patients were disproportionately Black and Hispanic. Navigators documented these events as system failure incidents and escalated them to clinical leadership at partner health systems.

2.2 Pain Dismissal Amplification

Clinical AI systems that incorporate provider notes into their decision support inherit the biases embedded in those notes. Research has documented that providers are significantly less likely to document pain complaints from Black patients at the same severity level as identical complaints from white patients.

DaisyMe navigators documented cases where AI-assisted pain management systems recommended lower analgesic doses for Black patients than for white patients with equivalent documented pain scores — a pattern consistent with the amplification of provider-level bias through algorithmic systems.

2.3 Chronic Disease Risk Underestimation

AI-assisted chronic disease risk stratification tools — used to identify patients for preventive interventions — systematically underestimated risk for Black and Hispanic patients in DaisyMe's navigation population. The most common pattern: patients who had avoided or delayed care due to cost or distrust had sparse EHR records, and sparse records were interpreted by risk stratification algorithms as low risk rather than as evidence of care avoidance.

This is a structural failure: the algorithm cannot distinguish between a healthy person who doesn't need care and a sick person who can't access it. The result is that the patients most in need of preventive intervention are the least likely to be identified for it.

2.4 Language and Literacy Bias in Patient-Facing AI

AI-powered patient communication tools — appointment scheduling systems, symptom checkers, medication adherence platforms — are predominantly designed and validated for English-speaking, health-literate users. DaisyMe navigators documented consistent failures in these systems for Spanish-speaking participants and for participants with limited health literacy:

• Symptom checker outputs that used clinical terminology without plain-language alternatives
• Appointment scheduling systems that could not accommodate interpreter requests
• Medication adherence platforms that sent reminders in English to patients whose primary language was Spanish

3. The Structural Conditions That Produce Algorithmic Inequity

The four failure categories documented above are not random errors. They are predictable outputs of three structural conditions.

3.1 Underrepresentation in Training Data

EHR data is generated by patients who access care. Populations that avoid or delay care — due to cost, distrust, transportation barriers, or prior negative experiences — are systematically underrepresented in EHR datasets. Clinical AI systems trained on EHR data are therefore trained on a population that is systematically different from the population they will be deployed to serve.

DaisyMe's navigation data documents this gap directly: 54% of participants cited prior negative experiences with providers as a barrier to care access. These are the patients whose experiences are least represented in the EHR data that trains the systems they will encounter when they do seek care.

3.2 Proxy Variable Substitution

Many clinical AI systems use proxy variables — variables that are correlated with the outcome of interest but are not direct measures of it. Healthcare cost as a proxy for health need. EHR record density as a proxy for health status. Provider-documented pain scores as a proxy for actual pain.

Each of these proxies encodes the inequities of the system that generated the underlying data. When an algorithm uses healthcare cost as a proxy for health need, it is not measuring health need — it is measuring historical access to care. And historical access to care is not equal.

3.3 Validation Without Disaggregation

Clinical AI systems are validated before deployment — but most validation studies do not disaggregate performance by race, ethnicity, language, or socioeconomic status. A system that performs well on aggregate metrics can perform significantly worse for specific demographic subgroups, and that disparity will not be visible in aggregate validation results.

The 2023 npj Digital Medicine review cited above found that fewer than 30% of clinical AI validation studies reported performance disaggregated by race or ethnicity. This means that the equity performance of the majority of deployed clinical AI systems is unknown — not because the data doesn't exist, but because the question wasn't asked.

4. Section 1557 and the Regulatory Landscape

Section 1557 of the Affordable Care Act prohibits discrimination on the basis of race, color, national origin, sex, age, or disability in health programs and activities receiving federal financial assistance. The 2024 HHS OCR final rule explicitly extended Section 1557 protections to clinical decision support tools and AI-assisted systems used in covered health programs.

This regulatory development has direct implications for health systems and clinical AI developers:

• Health systems that deploy clinical AI systems without equity validation are potentially exposed to Section 1557 liability if those systems produce disparate outcomes for protected populations.
• Clinical AI developers whose systems are used in federally funded health programs are subject to Section 1557 requirements — including the requirement to demonstrate that their systems do not discriminate on the basis of protected characteristics.
• The documentation standard for Section 1557 compliance is not aggregate performance metrics. It is disaggregated performance data demonstrating equitable outcomes across protected demographic groups.

DaisyMe Foundation's community navigation data provides an independent, community-sourced evidence base for Section 1557 compliance assessment — documenting the real-world equity performance of clinical AI systems as experienced by the populations they are supposed to serve.

5. Recommendations

For Health Systems

1. Require disaggregated equity validation before deploying clinical AI. No clinical AI system should be deployed in a federally funded health program without validation data disaggregated by race, ethnicity, language, and socioeconomic status.

2. Establish community navigation escalation tracking. Navigation escalation events — instances where a community navigator had to intervene because a clinical AI system produced an inadequate recommendation — should be tracked as quality improvement data and reported to clinical AI vendors.

3. Partner with community health organizations for real-world equity monitoring. Post-deployment equity monitoring should include community-sourced data, not just EHR-based performance metrics.

For Clinical AI Developers

1. Supplement EHR training data with community-sourced data. Training datasets should include data from community health organizations that serve populations underrepresented in EHR data.

2. Publish disaggregated validation results. Validation studies should report performance disaggregated by race, ethnicity, language, and socioeconomic status — and those results should be available to health system procurement teams.

3. Establish community advisory processes. Clinical AI development teams should include structured input from community health organizations representing the populations their systems will serve.

For Regulators

1. Require disaggregated performance reporting for clinical AI. FDA guidance on AI/ML-based software as a medical device should require disaggregated performance data as a condition of clearance for systems used in clinical decision support.

2. Fund community-based equity monitoring. AHRQ and PCORI should fund community health organizations to conduct independent, community-sourced equity monitoring of deployed clinical AI systems.

3. Enforce Section 1557 in clinical AI procurement. HHS OCR should issue guidance clarifying the Section 1557 documentation requirements for health systems procuring clinical AI systems.

6. Conclusion

The algorithmic inequity documented in this brief is not a technical problem with a technical solution. It is a structural problem that requires structural change: in how training data is collected, in how validation studies are designed, in how regulatory requirements are enforced, and in how community knowledge is valued and incorporated into the research process.

DaisyMe Foundation's community navigation data provides a window into the real-world equity performance of clinical AI systems that controlled validation studies cannot provide. We are committed to making that evidence available to the health systems, developers, and policymakers who have the power to act on it.

About DaisyMe Foundation

DaisyMe Foundation is a 501(c)(3) nonprofit organization headquartered in Frisco, Texas. We advance health equity for underrepresented communities through direct service programs, community health research, and systemic advocacy. Since 2022, we have served 12,400+ community members across the Dallas–Fort Worth Metroplex, Houston, and San Antonio.

Contact: info@daisyme.org · daisyme.org

© 2026 DaisyMe Foundation. This brief may be reproduced for educational and advocacy purposes with attribution. Citations: Obermeyer et al., Science 2019; Sjoding et al., NEJM 2020; Vyas et al., NEJM 2020; Celi et al., npj Digital Medicine 2023; HHS OCR Section 1557 Final Rule, 2024.