The UK government recently consulted various UK institutions on the role of AI in Healthcare and who it should be regulated. Led by the MHRA, numerous institutions voiced their opinions, including NICE, CQC and the ICO as well as NHS Trusts, universities and professional bodies. The government has since accepted all 44 recommendations made by the commission. Here I’ll focus on it’s implications for explainable AI in the UK healthcare sector.
The report doesn’t explicitly mention XAI, but is does mention several targets which could benefit from it, discussed below.
Recommendation 9. Transparent information for users and the public
The MHRA should consider issuing guidance to support developers in providing clear, transparent information to users and the public, such as through the use of model cards, careful user-centred interface design and dynamic labelling to support the safe use of software and AI-enabled devices.
This focus on transparency is similar to the EU’s AI Act. The key caveat to this recommendation is that is that users of AI devices must be able to question how these devices operate. There are many facets to this, and different stakeholders will have different questions, but some clinicians will want to know why an AI is making a certain prediction, and XAI can go some way some answering such questions. This doesn’t necessarily need to involve clinicians at the point of care delivery, but could involve ‘algorithmic consultants or computational nurses’ – healthcare professionals explicitly trained in AI – who would be responsible for ensuring safe deployment and continued audit of locally deployed AI. These roles are not yet established, but would work well with other recommendations of this commission (explicitly recommendations 32–33).
Recommendation 12. Human factors and usability expectations
The MHRA should consider providing clarity through guidance on human factors, usability considerations and expectations to support safe use as part of the design, development and testing of software and AI-enabled medical devices.
The overwhelming emphasis in the report is on safe deployment of AI. At least in the earlier years of deployment this is likely to mean that AI is basing it’s predictions/recommendations on features which accord with the current medical evidence base. It may well be possible that AI can go beyond what we currently know and use features hitherto unknown to be important. That may well drive innovative research, but for clinical deployment clinicians will want it to accord with known features. Either way, we first need to know which features are being used and XAI can go some way to illuminating this (concept bottle neck models would also be desirable for this).
Recommendation 39. User-centred design and transparency
The MHRA should work with relevant partners to develop guidance for developers on user-centred design considerations that support transparency for device users, including patients, professionals and the public, and enable simple and intuitive reporting of user feedback on the device experience.
The implications for this are similar to the above recommendation, but the emphasis is on interpretation:, in particular how to interpret its outputs. This has many facets depending on who needs to make the interpretation – patients may want to understand quite different things to clinicians. Either way, the medical imperative of informed consent and shared decision making make the ability to interrogate and interpret AI outputs in some manner simply essential to how modern medicine is practiced.
XAI isn’t a one size fits all solution to any of these recommendations, but it should be considered one potential tool amongst an array of tools. Which, if any, it is suitable for will require researcher and clinical collaboration and extensive testing in any deployment context.
Leave a Reply