BACKGROUND AND PURPOSE:

• Liao et al. (JCO Clinical Cancer Informatics, 2025) compared the accuracy and reproducibility of ChatGPT’s recommendations in response to breast cancer patient questions vs consensus expert opinion

METHODS:

• Comparative study
• Participants
  • Breast cancer patients participating in a weekly international breast cancer webinar series
• Comparisons
  • Responses given by a tumor board of renowned experts
  • Responses given to the same question by ChatGPT-4.0
• Study design
  • Questions were supplied to ChatGPT 3 separate times to assess reproducibility

RESULTS:

• 362 breast cancer patients
• ChatGPT-generated content that was entirely concordant with the recommendations of breast cancer experts: 46%
  • ChatGPT’s responses were not necessarily incorrect but often omitted specific details about clinical management
• ChatGPT’s responses demonstrated higher concordance with experts on topics related to earlier stages of breast cancer (0, I, II, III), compared to questions asked by advanced (IV) patients
  • P=0.019
• ChatGPT’s responses were less accurate when responding to patients about
  • Molecular markers and genetic testing: P=0.025
  • Antibody drug conjugates: P=0.006
• ChatGPT’s responses that were entirely consistent across different patients with the same question: 32%

CONCLUSION:

• Questions directed at ChatGPT by breast cancer patients entirely matched the responses written by experts less than half of the time
• Additionally, ChatGPT’s responses were inconsistent between users, even when patients asked the exact same question
• The authors state

As currently constructed, ChatGPT is not engineered to generate identical outputs to the same input and was less likely to correctly interpret and recommend treatments for complex breast cancer patients

Learn More – Primary Sources:

Accuracy and Reproducibility of ChatGPT Responses to Breast Cancer Tumor Board Patients