TY - JOUR
T1 - Physician Assessment of ChatGPT and Bing Answers to American Cancer Society's Questions to Ask about Your Cancer
AU - Janopaul-Naylor, James R.
AU - Koo, Andee
AU - Qian, David C.
AU - McCall, Neal S.
AU - Liu, Yuan
AU - Patel, Sagar A.
N1 - Publisher Copyright:
© 2024 Lippincott Williams and Wilkins. All rights reserved.
PY - 2024/1/1
Y1 - 2024/1/1
N2 - Objectives: Artificial intelligence (AI) chatbots are a new, publicly available tool for patients to access health care-related information with unknown reliability related to cancer-related questions. This study assesses the quality of responses to common questions for patients with cancer. Methods: From February to March 2023, we queried chat generative pretrained transformer (ChatGPT) from OpenAI and Bing AI from Microsoft questions from the American Cancer Society's recommended "Questions to Ask About Your Cancer" customized for all stages of breast, colon, lung, and prostate cancer. Questions were, in addition, grouped by type (prognosis, treatment, or miscellaneous). The quality of AI chatbot responses was assessed by an expert panel using the validated DISCERN criteria. Results: Of the 117 questions presented to ChatGPT and Bing, the average score for all questions were 3.9 and 3.2, respectively (P < 0.001) and the overall DISCERN scores were 4.1 and 4.4, respectively. By disease site, the average score for ChatGPT and Bing, respectively, were 3.9 and 3.6 for prostate cancer (P = 0.02), 3.7 and 3.3 for lung cancer (P < 0.001), 4.1 and 2.9 for breast cancer (P < 0.001), and 3.8 and 3.0 for colorectal cancer (P < 0.001). By type of question, the average score for ChatGPT and Bing, respectively, were 3.6 and 3.4 for prognostic questions (P = 0.12), 3.9 and 3.1 for treatment questions (P < 0.001), and 4.2 and 3.3 for miscellaneous questions (P = 0.001). For 3 responses (3%) by ChatGPT and 18 responses (15%) by Bing, at least one panelist rated them as having serious or extensive shortcomings. Conclusions: AI chatbots provide multiple opportunities for innovating health care. This analysis suggests a critical need, particularly around cancer prognostication, for continual refinement to limit misleading counseling, confusion, and emotional distress to patients and families.
AB - Objectives: Artificial intelligence (AI) chatbots are a new, publicly available tool for patients to access health care-related information with unknown reliability related to cancer-related questions. This study assesses the quality of responses to common questions for patients with cancer. Methods: From February to March 2023, we queried chat generative pretrained transformer (ChatGPT) from OpenAI and Bing AI from Microsoft questions from the American Cancer Society's recommended "Questions to Ask About Your Cancer" customized for all stages of breast, colon, lung, and prostate cancer. Questions were, in addition, grouped by type (prognosis, treatment, or miscellaneous). The quality of AI chatbot responses was assessed by an expert panel using the validated DISCERN criteria. Results: Of the 117 questions presented to ChatGPT and Bing, the average score for all questions were 3.9 and 3.2, respectively (P < 0.001) and the overall DISCERN scores were 4.1 and 4.4, respectively. By disease site, the average score for ChatGPT and Bing, respectively, were 3.9 and 3.6 for prostate cancer (P = 0.02), 3.7 and 3.3 for lung cancer (P < 0.001), 4.1 and 2.9 for breast cancer (P < 0.001), and 3.8 and 3.0 for colorectal cancer (P < 0.001). By type of question, the average score for ChatGPT and Bing, respectively, were 3.6 and 3.4 for prognostic questions (P = 0.12), 3.9 and 3.1 for treatment questions (P < 0.001), and 4.2 and 3.3 for miscellaneous questions (P = 0.001). For 3 responses (3%) by ChatGPT and 18 responses (15%) by Bing, at least one panelist rated them as having serious or extensive shortcomings. Conclusions: AI chatbots provide multiple opportunities for innovating health care. This analysis suggests a critical need, particularly around cancer prognostication, for continual refinement to limit misleading counseling, confusion, and emotional distress to patients and families.
KW - artificial intelligence
KW - ChatGPT
KW - health literacy
KW - patient information
UR - http://www.scopus.com/inward/record.url?scp=85181088199&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85181088199&partnerID=8YFLogxK
U2 - 10.1097/COC.0000000000001050
DO - 10.1097/COC.0000000000001050
M3 - Article
C2 - 37823708
AN - SCOPUS:85181088199
SN - 0277-3732
VL - 47
SP - 17
EP - 21
JO - American Journal of Clinical Oncology: Cancer Clinical Trials
JF - American Journal of Clinical Oncology: Cancer Clinical Trials
IS - 1
ER -