NEW

GE HealthCare Preps for SIR 2024

GE HealthCare will showcase its latest technologies in image guiding solutions, surgery, ultrasound and CT-navigation at the upcoming 2024 Society of Interventional Radiology (SIR) Annual Scientific Meeting taking place March 23-28 in Salt Lake City, Utah. The...

Samsung Accepts Healthcare Innovator Award

Boston Imaging, the United States headquarters of Samsung’s digital radiography and ultrasound business, was awarded HHM Health’s Healthcare Innovator Award

AHRA Co-Founder Passes Away

On March 2, 2024, Louise Broadley passed away at 101 years old. AHRA shared the news via an email and website post that reads, “She was a noble woman, who not only achieved many milestones within the imaging profession, but likewise paved the way for aspiring leaders...

Artificial Intelligence Paper Outlines FDA’s Approach to Protect Public Health and Promote Ethical Innovation

Today, the U.S. Food and Drug Administration (FDA) released its “Artificial Intelligence and Medical Products: How CBER, CDER, CDRH, and OCP are Working Together,” which outlines how FDA’s medical product centers are working together to protect public health while...

Assessment of AI Research report

Mark Watts

By Mark Watts

Henry Ford said, “If I had asked people what they wanted, they would have said faster horses.” This sums up the current understanding of artificial intelligence in imaging, a customer can easily describe a problem they’re having — in this case, wanting to get somewhere faster — but not the best solution.

In your research of solutions, I recommend that you compare the algorithm research reports – Summary, Mechanism, Validation and Performance, Warnings and Error messages.

With over 100 imaging AI products on the market, I thought I would share an algorithm report.

The American College of Radiology (ACR) recently held a webinar on the subject of “Build vs. Buy” AI for imaging that will be the subject for my next article. One of the key points of the discussion was regarding the need to validate the bought algorithm against a patient population and cost, speed of deployment, scale ability, integration concerns and relevance.

I chose this Navicular Fracture Algorithm report for this column because of its thoroughness. To assess the value proposition of these products we must understand its limitations and expected contribution vs. the effort to purchase, install and support.

This example of a summary gives parameters for what, who and how the algorithm was developed.

Summary

This model is based on convolutional neural networks and uses conventional radiographs of the hand, wrist and scaphoid to detect scaphoid fractures. It applies two networks consecutively: a segmentation network localizes the scaphoid and passes the relevant region to a detection network for fracture detection. It was developed in 2019-2020 at the Radboud University Medical Center and Jeroen Bosch Hospital in the Netherlands. Only radiographs with an anterior-posterior or a posterior-anterior projection of the hand in a neutral position are supported.

The Mechanism section allows you to evaluate the repeatability in a health care system.

Mechanism

Input: Data type: anterior-posterior (AP) or posterior-anterior (PA) radiograph (X-ray) of the hand, wrist or scaphoid.

File format: DICOM or MHA file containing the Pixel Spacing Attribute (tag: 0028,0030) or Imager Pixel Spacing Attribute (tag: 0018,1164). The Photometric Interpretation Attribute (tag: 0028,0004) should be “MONOCHROME2” (minimum value is intended to be displayed as black).

Target population: All patients with a sufficiently developed scaphoid bone (at least seven years old, preferably older than 18 years).

Output: Score: scaphoid fracture score between 0 (fracture absent) and 1 (fracture present).

Overlay: Class activation map with a heat map color coding that indicates which regions in the radiograph were influential to the prediction of the model: colder (blue-green) and warmer (yellow-red) colors respectively indicate low and high importance regions. The red canvas represents the selected region provided to the fracture detection network.

Model type: Convolutional neural network.

Training data location and time-period: Picture archiving and communication systems of the Radboud University Medical Center and Jeroen Bosch Hospital in the Netherlands, radiographs acquired between 2003-2020.

The validation section allows one to evaluate the scope and depth of the research.

Validation and performance

Scaphoid segmentation network:

Evaluation metrics: Dice similarity coefficient (DSC) and (symmetric) Hausdorff distance (HD).

Ground truth: manually drawn segmentation mask of the scaphoid.

Test set: 208 hand, wrist and scaphoid radiographs from the Radboud University Medical Center and Jeroen Bosch Hospital.

Results: The network achieved an average DSC of 97.4% ± 1.4 with an HD of 1.31 mm ± 1.03.

Scaphoid fracture detection network:

Evaluation metrics: sensitivity, specificity, positive predictive value (PPV), and area under the receiver operating characteristic curve (AUC). Automated image crops were used for evaluation.

Ground truth: binary scaphoid fracture label (fracture present/absent) obtained using a CT scan following conventional radiographs of the hand, wrist or scaphoid within four weeks.

Test set: 190 hand, wrist and scaphoid radiographs (95 fracture, 95 no fracture) from the Jeroen Bosch Hospital.

Results: the network achieved a sensitivity of 78% (95% CI: 70, 86), specificity of 84% (95% CI: 77, 92), PPV of 83% (95% CI: 77, 90), and AUC of 0.87 (95% CI: 0.81, 0.91).

Uses and directions

Benefits: Automated scaphoid fracture diagnosis may diminish the risk of missing a fracture, reduce the costs of additional imaging studies and unnecessary therapy, speed up diagnosis and allow earlier treatment.

Target population and use case: Patients clinically suspected of having a scaphoid fracture typically undergo conventional radiography. The model preprocesses incoming wrist, hand and scaphoid radiographs in the picture archiving and communication system, and may be able to assist residents, radiologists or other physicians by acting either as a first or second reader, or as a triage tool that helps prioritizing worklists.

General use: This model is intended to be used by radiologists for identifying scaphoid fractures on conventional radiographs of the hand, wrist or scaphoid. It is not a diagnostic for scaphoid fractures and is not meant to guide or drive clinical care. It should only be used to complement other pieces of patient information related to scaphoid fractures as well as a physical evaluation to determine the need for scaphoid fracture treatment.

Appropriate decision support: The model predicts whether any scaphoid fractures are present. The radiologist examines the predicted scaphoid fracture score and provided visual evidence along with other clinical information to determine if a scaphoid fracture is present.

Before using this model: Test the model retrospectively on a diagnostic cohort that reflects the target population that the model will be used upon to confirm validity of the model within a local setting.

Safety and efficacy evaluation: An observer study with retrospectively collected data was conducted and illustrated that the model was able to detect scaphoid fractures just as well as 11 radiologists, achieving a comparable AUC. Future research should investigate to what extent this model could improve the diagnostic performance of radiologists.

The warning section points out known issues and potential liabilities. I would be concerned if this level of transparency was not offered.

Warnings

Risks: Even if used appropriately, radiologists using this model can misdiagnose scaphoid fractures. Delays in diagnosing a scaphoid fracture put patients at a greater risk of developing a non-union fracture, which may lead to complications such as avascular necrosis, carpal instability, osteoarthritis, and ultimately functional loss. False positive diagnoses lead to unnecessary wrist immobilization, which increases health expenditure and decreases patients’ productivity.

Inappropriate settings: This model was not trained or evaluated on radiographs in which the scaphoid is incompletely depicted, obstructed by casts or implants, excessively damaged or malformed, (partially) resected, underdeveloped (children), or not depicted with an AP or a PA projection. Do not use the model when any of these circumstances apply. Only use radiographs that have been made specifically for fracture diagnosis and depict a hand or wrist in a neutral position (i.e. fingers pointing upwards).

Clinical rationale: The model provides visual evidence for the predicted scaphoid fracture probability. This evidence is only interpretable if the input radiograph is similar to the radiographs used for training the model. Clinical end users are expected to place the model output in context with other clinical information to make final determination of diagnosis.

Inappropriate decision support: This model may not be accurate outside of the target population (primarily adult patients). This model is not a diagnostic and is not designed to guide clinical diagnosis and treatment for scaphoid fractures.

Generalizability: This model was evaluated within the local setting of the Radboud University Medical Center and Jeroen Bosch Hospital. Do not use this model in an external setting without further evaluation.

Discontinue use if: Clinical staff raise concerns about the utility of the model for the indicated use case or large, systematic changes occur at the data level that necessitates re-training of the model.

The Error section is helpful for the technical validation of the products

Error messages

“Uneven pixel spacing encountered. Image scaling might not be accurate”: The uploaded image has a different pixel spacing in the X and Y direction. The model uses the pixel spacing information to rescale the image in order to normalize the scaphoid size before segmentation. Provided that the pixel spacing information is inaccurate, the scaling operation may result into an unnatural looking image.

“Pixel spacing of (1,1) encountered. Pixel spacing information could be wrong or missing”: The uploaded image most likely does not contain valid pixel spacing information. Accurate pixel spacing information is required to normalize the scaphoid size before segmentation.

“Bounding box is relatively small or large. Possible scaphoid segmentation failure”: The height or width of the bounding box derived from the generated scaphoid segmentation mask is outside three standard deviations from the mean height or width of the bounding boxes in the training data. This may indicate a failure of the segmentation network to localize the scaphoid.

“Cannot classify the image due to segmentation failure”: The segmentation network failed to segment any pixels of the scaphoid and no bounding box could be generated. Therefore, there is no region of interest to process by the fracture detection network

I hope this tour of an artificial intelligence algorithm report will prepare you to ask for and receive a better and more rewarding solution if you build or buy. We are still at the beginning of this AI in imaging journey. We are in the “early adopter phase” as Dr. Nina Kottler, MD, called it on the recent ACR call.

Henry Ford’s famous quote about the Model T was, “Any customer can have a car painted any color that he wants, so long as it is black.” The Model T only came in black because the production line required compromise so that efficiency and improved quality could be achieved.

My goal is to educate so that you can purchase and not be sold a solution that is limited.

– Mark Watts is the enterprise imaging director at Fountain Hills Medical Center.

 

Previous

Next

Submit a Comment

Your email address will not be published. Required fields are marked *

Open