AI adoption in healthcare is rapidly increasing, yet challenges in trust and usability remain. Explainable AI (XAI) seeks to improve transparency, trust, and decision-making by providing users with clear, interpretable insights into AI-generated predictions. A recent stakeholder-driven scoping review, led by Hari Subramanian, explores key design principles for XAI in kidney transplantation, combining stakeholder engagement and literature analysis to develop AI systems that align with real-world medical needs.
Stakeholder Engagement Approach:
This study incorporated insights from 39 transplant professionals and patients over a 9-month period through workshops and interviews. Participants included transplant center professionals, Organ Procurement Organizations (OPOs), and kidney recipients, who assessed AI decision-support mock-ups and XAI interfaces to provide feedback.
Key Findings on XAI Design:
- Contextual Use of AI Predictions
- AI can serve as a screening tool, alert system, or second opinion depending on user needs.
- OPOs prefer AI to be used as a screening tool to optimize kidney placement, while transplant centers prefer to use AI as a decision-support system for a second opinion.
- Information Needs in AI Predictions
- AI should provide global (system-level) and local (prediction-specific) explanations.
- Surgeons prefer system-level binary predictions with confidence scores, while OPOs and recipients favor local explanations that outline the influencing factors.
- Personalization for Different User Groups
- Experience level matters: Novices prefer more detailed AI explanations, while experts prefer optional AI explanations.
- Risk tolerance varies: Some surgeons want more AI-generated insights before making decisions, while others prefer minimal AI input.
- Customization for Case Complexity
- AI should offer detailed explanations for complex cases while simplifying insights for routine decisions.
- Users favored expandable options to access explanations as needed rather than being overloaded with details by default.
Conclusion:
This research highlights the importance of stakeholder-driven XAI design in kidney transplantation. AI systems should be flexible, user-controlled, and context-aware, ensuring they enhance—rather than disrupt—clinical workflows. Moving forward, balancing transparency, usability, and trust will be essential in optimizing human-AI collaboration in medical decision-making. Check out the full paper here!
Citation: H Subramanian*, C Canfield, D. B. Shank. 2024. Designing Explainable AI to Improve Human-AI Team Performance: A Medical Stakeholder-Driven Scoping Review. Artificial Intelligence In Medicine. 149: 102780. https://www-sciencedirect-com.mst.idm.oclc.org/science/article/pii/S0933365724000228.