BACKGROUND AND PURPOSE:

• Ayers et al. (JAMA Netw Open, 2023) evaluated ChatGPT responses to public health questions from the non-professional community

METHODS:

• Cross-sectional study
• Questions
  • 23 questions in 4 categories
    • Addiction | Interpersonal violence | Mental health | Physical Health
  • Questions used a common help-seeking structure
• Response evaluation criteria (performed by two authors, independently)
  • Was the question responded to?
  • Was the response evidence-based?
  • Did the response refer the user to an appropriate resource?
• Study design
  • Interrater reliability was measured
  • The percentage corresponding to each outcome (overall and among categories) was calculated
  • The number of words in ChatGPT responses and its reading level were assessed using the Automated Readability Index

RESULTS:

• Median ChatGPT response length: 225 (IQR, 183 to 274) words
• Mode reading level: 9^th grade to 16^th grade
• ChatGPT recognized and responded to all 23 questions in 4 public health domains
  • Responses determined to be evidence based: 91% (95% CI, 71 to 98)
  • Responses that made referrals to specific resources: 22% (95% CI, 8 to 44)
    • Addiction: 2 of 14 queries
    • Interpersonal violence: 2 of 3 queries
    • Mental health: 1 of 3 queries
    • Physical health: 0 of 3 queries

CONCLUSION:

• The advice ChatGPT provided to public health queries was generally evidence-based but infrequently provided referrals to specific resources
• The authors state

Limitations of this study include relying on an abridged sample of questions whose standardized language may not reflect how the public would seek help (ie, asking follow-up questions)

Additionally, ChatGPT responses are probabilistic and in constant stages of refinement; hence, they may vary across users and over time

Learn More – Primary Sources:

Evaluating Artificial Intelligence Responses to Public Health Questions