A Healthcare AI Benchmark Suite for Uncertainty Recognition, Justification Boundaries, Confidence Calibration, and Safe Action
Version 1.0 | Final Portfolio Release | Meredith Reese, RN | July 2026
The Uncertainty & Confidence Management (UCM) benchmark suite evaluates whether AI systems remain within the limits of what the available evidence can reasonably justify.
Unlike benchmarks that reward confident or complete answers, UCM rewards appropriately bounded reasoning. It tests whether a model can identify what remains unknown, separate confidence across different objects, choose safe actions without forcing diagnostic certainty, and revise its reasoning when new evidence arrives.
Has the AI exceeded what it can justify?
Evidence → Uncertainty → Justification → Confidence → Action → Reassessment
Every benchmark asks:
- What evidence is available, and how reliable is it?
- What remains uncertain?
- Which uncertainty is decision-blocking, non-blocking, or conditionally blocking?
- What is the strongest conclusion the evidence justifies?
- How confident should the model be about each specific object?
- What action is proportionate now?
- What scheduled or triggered reassessment could change the answer?
The safest conclusion is the strongest conclusion the evidence can justify.
UCM penalizes both:
- overreach, including unsupported diagnosis, causal attribution, confidence, escalation, treatment, prognosis, or disposition; and
- false uncertainty, including unnecessary delay, blanket deferral, under-escalation, or refusal to act when the evidence already supports a safe action.
- 12 original healthcare AI benchmarks
- 360 fictional source documents
- 48 submission-ready batch archives
- 12 system/task prompts
- 12 strict grading rubrics
- 12 benchmark design notes
- Shared UCM glossary and suite revision standard
- Domains spanning triage, early sepsis, medication reconciliation, specialist disagreement, transfer records, diagnostic reliability, patient identity, pathology, late-result revision, escalation, prognosis, goals of care, and disposition
Each benchmark is distributed as a self-contained ZIP archive for direct download.
| # | Benchmark | Primary domain | Core question |
|---|---|---|---|
| 001 | Incomplete Clinical Picture | Ambulatory triage and incomplete clinical evidence | What conclusions and actions are justified before the diagnosis is complete? |
| 002 | Early Sepsis with Pending Data | Early infection, sepsis physiology, and pending diagnostic data | What stabilization, monitoring, escalation, or de-escalation is justified while source and organism remain uncertain? |
| 003 | Medication Reconciliation with Unknown History | Medication history, dosing, administration, and discharge reconciliation | Which medication objects are sufficiently known to support active status, dose, schedule, administration, or discharge instructions? |
| 004 | Conflicting Specialist Recommendations | Multidisciplinary specialist disagreement and plan reconciliation | Is the conflict real, already reconcilable, or compatible with a safe interim plan? |
| 005 | Transfer Patient with Partial Documentation | Interfacility transfer and incomplete external records | Which missing information truly blocks action, and which gaps should not delay stabilization or capability-based transfer? |
| 006 | Diagnostic Evidence Reliability | Diagnostic test provenance, validity, and evidentiary quality | Is the available diagnostic evidence reliable enough to support the diagnosis, causal attribution, or treatment? |
| 007 | Duplicate Patient Identity Verification | Patient identity, record matching, and action-specific verification | Is identity certainty sufficient for the specific action being considered? |
| 008 | Pending Pathology Before Treatment | Pathology interpretation and treatment readiness | What diagnosis, subtype, staging, and treatment-readiness claims are justified before pathology is fully final? |
| 009 | Confidence Revision After Late Results | Late-arriving laboratory, imaging, culture, and specialist evidence | How much should the conclusion, confidence, and action change after new evidence arrives? |
| 010 | Escalation Threshold Under Diagnostic Uncertainty | Escalation, destination selection, and level-of-care decisions | Does risk justify escalation despite diagnostic uncertainty, and what level or destination is proportionate? |
| 011 | Prognostic Uncertainty & Goals of Care | Serious illness prognosis, treatment benefit, function, survival, and goals of care | What can actually be justified about prognosis, and what should remain uncertain? |
| 012 | Disposition Under Residual Uncertainty | Discharge, observation, admission, transfer, and disposition deferral | When is residual uncertainty compatible with a safe disposition? |
- Portfolio overview — PDF
- Portfolio overview — editable DOCX
- One-page portfolio summary — PDF
- One-page portfolio summary — editable DOCX
Benchmark_001_...zipthroughBenchmark_012_...zip— twelve complete downloadable benchmark packagesUSING_THE_BENCHMARKS.md— quick guide for reviewers and evaluatorsUCM_GLOSSARY.md— canonical suite vocabularyUCM_SUITE_REVISION_STANDARD_v1.md— leakage controls, timeline-diversity requirements, and grading standardizationBENCHMARK_INDEX.md— portfolio-facing summaries of all twelve benchmarksCOVERAGE_MATRIX.md— cross-benchmark reasoning coverageEVALUATION_FRAMEWORK.md— UCM pillars and reasoning methodDESIGN_METHODOLOGY.md— benchmark construction principlesPACKAGE_MANIFEST.md— verified suite counts and public distribution structureRELEASE_NOTES.md— Version 1.0 release summaryFILE_MANIFEST.txt— public repository file listRIGHTS_AND_USE.md— copyright and reuse termsDISCLAIMER.md— fictional-data and validation limitationsAUTHOR.md— author background and related benchmark suites
- HORB asks: Can the AI reason through realistic healthcare operations?
- EIED asks: Has the AI earned the right to reach this conclusion?
- UCM asks: Has the AI exceeded what it can justify?
HORB emphasizes operational execution. EIED emphasizes evidentiary support for decisions. UCM emphasizes uncertainty boundaries, object-specific confidence, proportionate action, and reassessment.
Meredith Reese, RN
Registered Nurse | Healthcare AI Evaluation & Benchmark Design
GitHub: https://github.com/reesemeres-jpg
All scenarios and records are fictional. UCM is a portfolio and evaluation-design project, not clinical guidance or a psychometrically validated public leaderboard benchmark. No protected health information is included.