Artificial Intelligence promises a range of practical applications to improve efficiencies within the medical sector; in particular, the continued development of machine learning-enabled medical devices (MLMDs) can help medical professionals deliver higher quality care to their patients at an increased speed.

However, these devices are often inscrutable due to their use of deep neural networks which rely on so much training data that it becomes impossible to fully understand how they reach decisions (often referred to as “black box” AI).

The Medicines and Healthcare Products Regulatory Agency (MHRA) recently published a collection of six “guiding principles” to ensure that MLMDs are developed to be transparent, helpfully categorised as “who”, “why”, “what”, “where”, “when”, and “how”. These principles are developed in collaboration with the US and Canada and build on 10 guiding principles for good machine learning practice identified in 2021.

Who – Developers should consider those that use the device (such as medical professionals), those that receive care from it (such as patients), and those that decide how it is used (such as medical support staff).

Why – The MLMD should clearly convey its intended purpose; for example, should it replace a professional’s judgement, or merely inform it?

What – The device should provide a clear, comprehensive and accurate description, including its intended purpose, users and patients, and how it was developed (such as the training data used). Furthermore, MLMDs should clearly demonstrate the “logic” that they use, provided that this is available and easily understood. 

Where – The user interface should be designed to make the device responsive and enable the user to easily locate information.

When – The MLMD should provide the right information at the right time. For example, it may need to show specific warnings at certain stages of a procedure, such as when a professional is about to perform a high-risk activity.

How – The developer should employ human-centred design principles, involving third parties in the design process and seeking regular feedback to ensure that the device caters to the end user’s needs. The device should also use appropriate language; whilst a medical professional will want comprehensive information on how the device works, information aimed at patients should be clear and simple.

Based on these guiding principles, it will be key for companies involved in the development of MLMDs to consider how they can improve transparency in their systems to adapt to the evolving needs of patients and professionals alike. Key to this will be ensuring that their development processes incorporate human-centred design principles as neural networks increase in complexity.

The MHRA’s six guiding principles can be found here.