By Mark Watts
Human-centered artificial intelligence deployment takes into consideration the system design to bring ambient intelligence to a project. I would like to use the computer assisted diagnostic (CAD) report as an example of how frictionless value can be added to the radiologist’s workflow. Traditional text-based radiology reports may be significantly improved by incorporating multimedia elements and interactive functionality, including hyperlinks between report text and specific findings on images. Such advanced reporting was developed to improve clinical care, but the hyperlinks can also be leveraged to speed up the development and clinical deployment of regulatory-compliant AI.
An example of a scheme of interactive reporting is when a radiologist annotates the location of an abnormality on an image with a measurement, an arrow, a bounding box, or a freehand region of interest, and dictates, “There is a focal liver lesion near the posterior branch of the left portal vein.” Giving a verbal command to the reporting software (such as the word hyperlink), triggering automatic insertion after the word vein of a hyperlink “2.1 cm (series 9, image 67)” linking to the abnormality location. A multisite implementation of feedback and monitoring for artificial intelligence development and clinical deployment using a centralized, cloud-based infrastructure may look like the following example.
Main processes include:
1) generation of hyperlinks to findings in radiology reports during routine interpretation
2) storing the images, reports and labels in the cloud
3) developing algorithms based on the stored data
4) silently testing algorithm prototypes against production data and receiving feedback
5) iteratively retraining until clinical deployment; and monitoring clinically deployed algorithms
For accuracy degradation, alerting and notifying regulatory bodies should occur as needed.
A report consumer, such as a referring physician, clicking on an embedded hyperlink, invokes a viewer to display the relevant images, with annotations marking the specific findings mentioned in the text. Referrers and other radiologists appreciate this functionality, and the benefits to patient care can be substantial. Hyperlink creation does not significantly disrupt the process of dictation; at sites where it is available, hyperlinking quickly gains traction.
Hyperlinks embedded in interactive reports connect report text with lesion locations on images. We propose that NLP applied to report text containing hyperlinks can produce stronger labels than NLP alone. Analysis of the text associated with hyperlinks can establish their significance and meaning. Because hyperlinks inserted to benefit clinical care are repurposed, there is no loss in reporting efficiency, and large numbers of strong labels can automatically be collected during routine interpretation. This provides the necessary foundation to collect large, reliably labeled cohorts of training data and an automatic feedback loop for algorithm development and monitoring by:
- automatically classifying examinations by their findings
- automatically preparing cohorts of examinations for training AI algorithms that detect, locate, measure and segment certain findings
- deploying such algorithms into radiologists’ routine reporting workflow for testing
- silently (without disturbing the radiologists), automatically obtaining feedback on algorithm accuracy, by comparing their output with the hyperlinked findings of the radiologists
- retraining the algorithms by using the original training data combined with the new test data to improve accuracy
- repeating steps 3 to 5 with the retrained algorithms, until necessary accuracy is achieved
- monitoring approved algorithms using the same process; and
- for surveilling accuracy and producing alerts, resulting in automatic feedback to users, administrators and manufacturers.
This process can be deployed across multiple sites with a cloud-based implementation, with appropriate privacy protection. Documentation produced by this process would support the FDA proposed total product life cycle approach to AI and ML development, feedback, monitoring, failure detection, and ongoing retraining. Implementation at diverse sites may provide the data to produce generalized, less brittle algorithms. Analysis of feedback may suggest factors that adversely affect performance.
Multiple commercial vendors of PACS, reporting and image sharing tools have already developed platforms that provide many of the capabilities necessary to implement interactive reports, and standards exist to encode and transfer the structured content. The Healthcare Information and Management Systems Society and the Society for Imaging Informatics in Medicine have also established a working group supporting interactive multimedia reporting. Integrating the Healthcare Enterprise profiles are being considered for supporting interactive multimedia reporting. The ACR has developed and deployed a powerful infrastructure including registries, Certify-AI, Assess-AI, and the AI-Lab, which facilitate algorithm evaluation.
The application of AI and ML in clinical practice requires a more rigorous approach than is currently used. Scalable, efficient collection and the use of curated, labeled medical images is urgently needed to support the development and improvement of algorithms, and ongoing monitoring of performance. Our proposed approach will accelerate the development and clinical deployment of regulatory compliant AI algorithms by leveraging hyperlinks produced in routine clinical reporting to provide reliable labels, a source of feedback, and data for monitoring of algorithms. Continuous feedback from radiologists allows detection and correction of degradation, protecting patient safety, satisfying upcoming regulatory requirements, promoting trust, and increasing the value of radiology reports in a manner respectful of radiologists’ time.
We are on a journey and the challenges of human-centered artificial intelligence can seem difficult. The focus is on the radiologist and adding ambient intelligence to their workflow. To do this we must design a frictionless information technology platform that provides all shareholders with value. One step at a time we can move forward.
– Mark Watts is the enterprise imaging director at Fountain Hills Medical Center.
Editor’s Note: This is Part II of a two-part series.
