Category: Uncategorized

AI systems are widely used in healthcare for applications ranging from medical image scan analysis to cancer detection. Human-subject studies are essential for understanding the effects of AI in specific tasks or domains. For us, these experiments are crucial for assessing the impact of Explainable AI (XAI) on factors like task performance, user trust, and user understanding. Conducting these experiments enhances our understanding of human-AI interaction, ultimately shaping AI systems to be more effective and suitable for specific domains, users, and tasks. However, to develop theory, we need to conduct a large number of experiments to iterate on designs.

Unfortunately, in the healthcare domain, it is difficult to conduct a large number of human-subject experiments. Experts, such as doctors and nurses, are busy and don’t have time to participate in multiple studies. Therefore, we find it useful to conduct more general experiments in a non-healthcare context (i.e., an analogy domain) where we can use an online participant pool. Once we find designs that work well in general, we will be able to test them in the kidney transplant context via a SimUNet field trial.

Identifying an Appropriate Analogy Domain

In the kidney transplant process, the proposed AI system will be utilized by surgeons and coordinators with expertise in matching donors and recipients. Therefore, the analogy domain should enable people to leverage their knowledge to complement the AI, rather than blindly accepting its predictions. Expertise can be assessed through self-reported measures, career-related information, and objective knowledge tests.

The analogy domain must also have tabulated information, like the dataset used in the kidney transplant process, which contains elements such as donor age, gender, and serum creatinine levels. The outcome measure should be predictable, not random, allowing users to make informed decisions. Domains like fantasy football or sports betting are unsuitable due to high randomness.

Finally, the domain should involve subjective truth – a decision made on personal expertise and risk preferences, where two experts could genuinely disagree about the appropriate path forward. In the kidney transplant process, surgeons’ decisions to accept or not accept a donor kidney are used as the ground truth in AI’s training. This is different from an “objective truth,” such as the health outcomes that a patient realizes in the future. In many cases, such as when a transplant is not conducted, we don’t know what that objective truth would have been.

Based on these characteristics, we have developed an experimental task in the real estate domain. The decision-making process in real estate involves the assessment of multiple attributes and expert knowledge. This similarity makes it a useful domain for studying how AI explanations affect user decision-making. Many people have familiarity with the process of buying a house, so they understand the decisions that go into it. For real estate, there is data on the house as well as the buyer, which have to go through a matching process. This allows us to test different XAI formats that can eventually be translated to the kidney transplant context.

Contributors: Harishankar V. Subramanian (S&T), Casey Canfield (S&T)

Between December 2023 and May 2024, we have interviewed 10 OPO leaders from across the country. One observation has been the existence of conflicting visions for AI adoption.

Some leaders believe that there needs to be a coordinated, top-down approach to AI adoption to ensure fairness and equity. This would likely be led by government regulators. However, this is likely to be a slower process.

Others see a need for a more bottom-up approach to facilitate experimentation and innovation. This can support the identification of best practices, but not all OPOs are equally able to participate. OPOs that are bigger or have more resources will be better able to benefit from AI.

The bottom line is that the exact impact of AI on OPO practices is unknown. We are hoping to generate evidence on how much AI can help as part of our planned field experiments using UNOS’s SimUNet. There are pros and cons of both a top-down and bottom-up approach in the adoption of AI. We are currently working on drafting a paper to summarize our findings and encourage more dialogue on this subject. In the meantime, we want to know what you think!

So far, we have created 2 deep learning models that are focused on the OPO perspective:

1. Deceased Donor Kidney Assessment – which evaluates the likelihood that a donated kidney will be transplanted based on up to 18 characteristics (including biopsy information, if available). This can help OPOs determine whether a kidney is hard-to-place based on historical behavior. You can play with this model here: https://ddoa.mst.hekademeia.org/#/
2. Final Acceptance – which improves the estimate of the likelihood of transplant by incorporating recipient characteristics. This can be used to make an estimate for each recipient on the Match Run. For hard-to-place kidneys, this can help determine where to go for expedited placement. OPOs could save time by not sending offers that are very unlikely to be accepted.

Both models are trained from the OPTN Deceased Donor Dataset. We are planning to test the impact of these models using UNOS’s SimUNet, which is a research platform that is currently being expanded to include the OPO perspective as part of this project. To date, SimUNet studies have only focused on transplant surgeon decision-making.

To develop models that support transplant surgeons, we believe a more tailored approach is needed. Professor Cihan Dagli and Rachel Dzieran are working on a new model called Transplant Surgeon Fuzzy Associate Memory (TSFAM). The intent is for the model to use deep learning network model interfaces to capture individualized transplant surgeon practices and assessments through fuzzy associative memory. Fuzzy logic accounts for imperfect data and ambiguity, which is more consistent with how humans make decisions. We are identifying decision rules used by an individual transplant surgeon and then tailoring the AI-based decision-making model to support individual decision-making. Case studies are currently being reviewed for building the structure to collect the individualized transplant surgeon policies. The primary goal of this work is to support transplant surgeons by using their own policies when assessing deceased donor organs. Dr. Dagli has two new PhD students joining in Fall 2024 to continue to develop this new model.

Contributor: Rachel Dzieran (S&T), Cihan Dagli (S&T)

How does fairness work in organ transplantation? 

At present, the current organ allocation process is based on:

• First, relevant medical criteria – such as the severity of the potential recipient’s condition and blood type – as well as the geographic proximity of the donor and recipient.
• Secondly, the potential recipients are placed onto a list in which, after the previous criteria are taken into account, fairness in organ allocation is considered on a first come-first served model where those who have been on the waitlist the longest are prioritized to receive a matched organ. 

The issue at hand is that potential recipients matched with higher-risk kidneys often decline to accept such kidneys since they are likely to receive a more desirable kidney in the near future. This requires moving further down the wait-list, which is a time-consuming process and “time is tissue.” This project’s goal is to introduce an AI-assisted decision-making tool to operationalize fairness in the algorithm and provide a “first and best”-served model that reflects the same level of fairness as in standard organ allocation, but offers a more precise way of matching less desirable kidneys with recipients who are likely to accept them. In short, who is most likely to accept and to succeed with this kidney

Why AI? 

Algorithms can help simplify complex systems. However, algorithms use historical data to make estimates, which can introduce bias from previous decisions and changing policies. This is where the fairness of algorithms comes into play. We want to evaluate the models, and not the current process, to ensure the AI generalizations are actually helpful. Our AI tool aims to be a fair decision assistant. Surgeons and their patients have the final say and the power to override or ignore the algorithm’s prediction. Algorithms can’t see external factors, such as a support system for the recipient, or if the recipient is comfortable with the risk factors for a particular offer and other important factors for a successful transplant. Surgeons and transplant professionals have access to this. With the development of this AI, surgeons can see the recommendation and add additional factors to make the final decision of kidney transplantation in collaboration with their patient.

Project Overview:

This project aims to bring significant improvements to how kidneys are offered for transplant while examining and addressing biases in AI systems used to support decision-making. It involves developing fairer AI systems that balance the needs of transplant candidates, transplant centers, and OPOs. The main goals are to:

1. Understand different stakeholder preferences to identify biases in the AI system.
2. Combine these preferences to determine a fair approach.
3. Enhance the AI system’s fairness based on the combined preferences.

In simpler terms, the project looks at how biases in AI can affect decisions about kidney transplants and works to create a fairer system that considers the needs and views of everyone involved.

Contributors: Jason Eberl (SLU), Michael Miller (SLU), Venkata Sriram Siddhardh Nadendla (S&T), Mukund Telukunta (S&T)

The Artificial Intelligence and You Symposium hosted by the Center for Science, Technology, and Society was held on April 26^th at the Innovation Forum at Missouri S&T. Our team organized a panel discussion in which we discussed measuring and aggregating preferences in AI for kidney transplant healthcare. This included 3 short presentations followed by a panel discussion led by Casey Canfield and joined by Cihan Dagli, Daniel Shank, and Sid Nadendla.

Harishankar Subramanian spoke on integrating stakeholder input into the design of explainable interfaces. Explainable AI (XAI) helps users to understand system’s processes and logic, appropriately trust the system, effectively manage performance as well as fully understand the system and why the system is operating the way in which it is operating. One of the main findings was that the appropriate interface of an AI will vary based on user expertise and decision-making process. Future work will be done to investigate user understanding of AI and the users ability to judge when to rely on AI recommendations. 

Mukund Telukunta spoke on measuring social fairness preferences from non-expert opinions in kidney transplantation. There is a history of racial bias in the organ transplant process, which needs to be considered in the development of AI tools. Survey participants were presented information, which included the AI’s recommendation as well as the surgeons decision for 10 potential recipients, to evaluate the fairness of the outcomes. The AI decision support system was deemed fair. Future work will consist of gathering expert opinions, such as OPO’s, transplant surgeons, and patients. 

Amaneh Babaee spoke on her research on identifying organizational and individual factors in AI adoption for kidney transplant, specifically in organ procurement organizations (OPO’s). Interviews with OPOs suggest that AI will be useful if it can measurably speed up the allocation process. In addition, the implications of OPO size varies. A large OPO has more higher financial resources to adopt the AI but has a slower decision-making process. A small OPO has less financial resources, but can quickly decide whether or not AI is suitable for them. She is also deploying a survey to measure perceptions of AI. While both studies are still ongoing, it seems that AI has the potential to improve transplant outcomes, but regulatory hurdles may hinder its integration into existing operations.

In the panel discussion, one concern was that the AI could make incorrect recommendations, leading to negative outcomes. It is unlikely that this process would be automated and there will always need to be a human-in-the-loop. Transplant centers and OPOs will still need to rely on their expertise to fill in the gaps, since the AI does not have as much information as they do about a particular case.

We want to know what transplant stakeholders think about adding AI to the kidney placement process. Please let us know if you would like to participate in a survey or interview! Email Elham (abt8f@umsystem.edu) for more information. 

We are proposing that OfferAI, a hypothetical tool based on the one we are developing, could help people at transplant centers and organ procurement organizations (OPOs) (see diagram below). For a transplant center, AI could be used to accept or deny kidney offers faster. For OPOs, AI could be used to decide if the kidney is hard-to-place and help decide when to use a rescue pathway (i.e., accelerated or expedited placement).  

As part of our research, we are currently interviewing OPOs to get a better understanding of the potential benefits and drawbacks of implementing AI into their organizations. Interview data will remain confidential and will be solely used for research purposes.  

We are also seeking participants for a survey about AI adoption. We are comparing how attitudes, perceived risks, assurance and trust in AI, interpersonal influence, and government influence affect interest in AI adoption across transplant centers, OPOs, patients, and the public. This will help us identify potential barriers and opportunities for AI in the kidney transplant placement process.  

Right now, we are recruiting people who work at:  

• Transplant centers: take the survey here
• OPOs: take the survey here 

We recently hosted a webinar to provide an update on our research progress:

• We have launched a prototype of OfferAI for deceased donor kidney assessment. Please try it out and let us know what you think.
• We have drafted a review paper about how to design explainable AI to improve human-AI team performance.
• We are preparing to launch 2 surveys related to adoption and fairness to understand how different stakeholders perceive OfferAI and believe it should be implemented.

Review the recording and check out the slides!

Cihan Dagli and Richard Threlkeld presented a poster titled “AI-Enabled Digital Support to Increase Placement of Hard-to-Place Deceased Donor Kidneys” at the 2023 American Transplant Congress in San Diego, CA.

Our team of researchers at Missouri S&T, Saint Louis University, and United Network for Organ Sharing was awarded a $1.8M research grant from the National Science Foundation.

Check out the press release!

Collaborative Research: FW-HTF-R: Embedding Preferences in Adaptable Artificial Intelligence Decision Support for Transplant Healthcare to Reduce Kidney Discard

Overview. Transplantation provides patients suffering from end-stage kidney disease a better quality of life and long-term survival. However, approximately 20% of deceased donor kidneys are discarded and never transplanted. While some discards may be medically appropriate, others reflect missed opportunities. Even kidneys deemed less desirable may provide survival benefits to some recipients. Organ Procurement Organizations (OPOs) have great difficulty finding transplant centers to accept less medically desirable (or higher risk) kidneys. At their discretion, OPOs can use accelerated placement to bypass the priority list for “hard-to-place” kidneys. However, due to a lack of data-driven guidance, this mechanism is not systematically applied and likely underutilized.

To enable transformative change, we propose to integrate Artificial Intelligence (AI) decision support into the kidney offer process to support both demand at the transplant center and supply at the OPO. Key workers include OPO staff (organ procurement coordinators, operations directors, medical directors), transplant center staff (coordinators, physicians, surgeons), and transplant patients. Based on preliminary work from our planning grant, we are developing an AI decision support system for (a) transplant centers to accept/deny high-risk kidney offers and (b) OPOs to identify hard-to-place kidneys sooner. In this research, we will (1) measure human preferences to customize the support system’s operation and interface, (2) aggregate fairness preferences as defined by diverse stakeholders to improve fairness in the model output, (3) evaluate the effect of embedding uncertainty and explainability into the interface, (4) develop deep learning ensemble models that can adapt over time while being explainable, and (5) conduct a randomized control trial using United Network for Organ Sharing (UNOS) Lab’s SimUNet, an ecologically valid kidney offer simulation platform for behavioral experiments, to estimate the impact on kidney discard. This research is driven by an existing partnership between transplant and ethics experts at Saint Louis University Hospital (SLU), behavioral scientists at UNOS, and experts in AI and human factors from Missouri University of Science & Technology (S&T).

Keywords: kidney transplant offers, trust, AI decision support system

Intellectual Merit. AI systems suffer from technical and human-integration challenges. Existing deep learning models are generic black boxes and cannot adapt in real-time to specific users. We propose to implicitly and explicitly solicit preferences as part of the training process to allow users to integrate the model predictions into their existing decision-making process. As part of this process, we will aggregate fairness notions across stakeholders to determine appropriate metrics for improving model performance. Within the deep learning model, we will impose trade-offs to increase fairness without significantly reducing accuracy, enhance explainability by converting feature relevance scores and convolution layer weights into linguistic expressions, and use transfer learning to rapidly integrate new data (such as preferences for customizing to specific workers) into deep learning models. In addition, we will conduct human subjects experiments to evaluate how uncertainty and explainability presentation influence trust, confidence, ease-of-use, and performance. Ultimately, each of these pieces (training, preference elicitation, deep learning model, interface presentation) will be combined and evaluated behaviorally using the ecologically-valid UNOS Lab’s SimUNet to estimate the impact on kidney discard.

Broader Impacts. Ultimately, this research aims to reduce kidney discard for high-risk organs by at least 10%. This work will support critical advancements in ethics and training, issues that will be critical in overcoming system-level barriers to integrate AI into healthcare. In the transplant context, this will support the evidence-based application of more controversial management strategies, such as accelerated placement, which bypasses transplant centers that are unlikely to accept high-risk kidneys. In addition, this project will expand SimUNet to incorporate the OPO perspective to demonstrate the value of system-level behavioral science platforms for contexts with supply and demand roles. Ultimately, this project will train students from diverse backgrounds in transdisciplinary convergent research and serve as the basis for course projects and modules in systems engineering, psychology, and philosophy.

Our team of researchers at Missouri S&T and Saint Louis University was awarded a planning grant from the National Science Foundation.

Check out the press release!

FW-HTF-P: Collaborative Proposal: Teaming Transplant Professionals and Artificial Intelligence Tools to Reduce Kidney Discard

Overview. While over 94,000 people are on the kidney transplant waiting list, less than a third will receive one this year. Thousands of usable kidneys are discarded due to inefficient workflow processes and negative perceptions for using lower quality organs. Organ Procurement Organizations (OPOs) have great difficulty finding transplant centers to accept procured lower quality kidneys. A single kidney can run through thousands of offers before one, if any, transplant center accepts it. The current process of manually placing lower quality organs via phone calls and emails is not working.

We will revolutionize this process with the aid of an artificial intelligence (AI) system and usable trustworthy interfaces that are fully integrated into the transplant workflow between demand-side (transplant center) and supply-side (OPO) organizations. Over the course of the planning grant, we will (1) document a transplant work system architecture and identify challenges for re-designing this work process, (2) develop a proof-of-concept AI system to predict which candidates are most likely to accept a lower quality kidney that is at risk of discard, and (3) perform human subjects experiments to scope the interface design and predict technology adoption factors. These efforts will build capacity for integrating AI into transplant healthcare and scope future research activities in a larger FW-HTF proposal. We frame this challenge as a system engineering problem to integrate technical, human, and process elements into our proposed solution.

We will perform system-based participatory research centered around a design-a-thon event to (a) build consensus on design criteria for the AI and interface in a realistic workflow, (b) evaluate mock-ups in focus groups, and (c) anticipate challenges for system implementation. The key workers include organ procurement coordinators, transplant coordinators, and transplant physicians and surgeons. This research is driven by an existing partnership between transplant experts at Saint Louis University Hospital (SLU) and experts in AI and human factors from Missouri University of Science & Technology.

Intellectual Merit. Despite their popularity, AI systems suffer from technical, human, and integration challenges. It is time-consuming and costly to manually design neural architectures, so we propose an approach that uses evolutionary algorithms to find the optimal architecture for a particular data set. This will facilitate real-time adaptation as the data inputs evolve over time. In addition, it is critical for AI systems to be explainable and transparent, particularly in high stakes contexts. We will perform human subject experiments with lay populations to evaluate how uncertainty visualizations and metrics influence performance, confidence, trust, technology acceptance, and willingness to choose riskier options. In a larger FW-HTF proposal, we aim to integrate this solution into the actual workflow between Mid-America Transplant and regional transplant centers, including SLU. Once validated, this research can also be applied to other data-intensive high-stakes scenarios (e.g. military operations, automated critical infrastructure).

Broader Impacts. Increasing automation via AI will reduce organ discard and increase the number of available kidneys for transplant by increasing utilization of lower quality kidneys. These efforts to increase the efficiency and efficacy of workers and systems involved in allocating organs for transplant is particularly critical because a recent Executive Order aims to make OPO performance metrics enforceable via regulation in the near future. This is a particularly controversial approach because evaluations of OPO performance are really an evaluation of the whole system. Reducing kidney discard is a two-way street because the demand-side (i.e., transplant centers) must also be more willing to accept imperfect kidneys.  

Findings on technology acceptance will influence the design of future training programs. In addition to pursuing a future FW-HTF proposal, we will build on this planning grant by pursuing a National Research Traineeship award to develop an evidence-based interdisciplinary graduate-level program on human systems integration.