ChatGPT Passes US Medical Licensing Exams (kinda)

[mercurialworld]

Quote

 

ChatGPT, the viral chatbot that has raised concerns from teachers and academics over its ability to cheat on essays and exams, has now passed a Wharton MBA final exam, the United States Medical Licensing Exam, and components of the bar exam. 

 

A Wharton Professor conducted a study in which he used OpenAI’s GPT-3, the language model on which ChatGPT is built, to take a final exam of a core MBA course. He concluded that GPT-3 would have received a B to B- on the exam.

 

The professor, Christian Terwiesch, found that GPT-3 performs the best “at basic operations management and process[ing] analysis questions.” For these, the chatbot provided both correct answers and excellent explanations as to why an answer was selected. In the paper’s summary, Tewriesch acknowledged that GPT-3 is by no means perfect. At times the bot made mistakes in simple mathematical calculations and wasn’t able to handle more advanced process analysis questions."

Source: https://www.vice.com/en/article/akebwe/chatgpt-is-passing-the-tests-required-for-medical-licenses-and-business-degrees

 

What do you guys think? Do you think that ChatGPT is a threat to academic integrity, or is it simply a tool that can be used for good or bad?

Edited January 29, 2023 by mercurialworld

[Kummercell]

yet it couldn't write the review I was supposed to write but was too lazy to actually read the book 

[mercurialworld]

4 minutes ago, Kummercell said:

yet it couldn't write the review I was supposed to write but was too lazy to actually read the book 

[MingYouToo]

Med student here (M1), this is actually true but kind of misleading.

 

It did pass the bare minimum USMLE requirements (barely >60%), but in practice it performed TERRIBLY, just enough to pass but no sane PD (program director) would look at the ChatGPTs results and think 'this guy would make a great resident' (which is the purpose of the USMLEs).

 

The questions its fed is also answered limited and terribly so, it can only answer direct/first order questions but when you start giving it questions that include irrelevant/distracting information + second/third order questions (questions that are lengthy but only really require like 2 lines of text to answer) plus reverse engineered (from 'what is the diagnosis' to 'what would we expect the labs to look like if we assume patient has sickness/pathology X) ones it fails badly.

[mercurialworld]

44 minutes ago, MingYouToo said:

Med student here (M1), this is actually true but kind of misleading.

 

It did pass the bare minimum USMLE requirements (barely >60%), but in practice it performed TERRIBLY, just enough to pass but no sane PD (program director) would look at the ChatGPTs results and think 'this guy would make a great resident' (which is the purpose of the USMLEs).

 

The questions its fed is also answered limited and terribly so, it can only answer direct/first order questions but when you start giving it questions that include irrelevant/distracting information + second/third order questions (questions that are lengthy but only really require like 2 lines of text to answer) plus reverse engineered (from 'what is the diagnosis' to 'what would we expect the labs to look like if we assume patient has sickness/pathology X) ones it fails badly.

that's super interesting, thank you for the info!

I'll update the title to reflect this info

[elevate]

46 minutes ago, MingYouToo said:

Med student here (M1), this is actually true but kind of misleading.

 

It did pass the bare minimum USMLE requirements (barely >60%), but in practice it performed TERRIBLY, just enough to pass but no sane PD (program director) would look at the ChatGPTs results and think 'this guy would make a great resident' (which is the purpose of the USMLEs).

 

The questions its fed is also answered limited and terribly so, it can only answer direct/first order questions but when you start giving it questions that include irrelevant/distracting information + second/third order questions (questions that are lengthy but only really require like 2 lines of text to answer) plus reverse engineered (from 'what is the diagnosis' to 'what would we expect the labs to look like if we assume patient has sickness/pathology X) ones it fails badly.

Yes, from what I've used of it, this holds true. It feels very much to be still in beta.

 

But with that said, it is kind of incredible what it can do right now, and it's clear that it will learn at an accelerated rate from what we've come to expect from these initiatives. 

 

Of all the random ground-breaking startups and whatever else from the last decade or so that undeservedly got attention, this is the first one that feels like it's planted the seeds of something truly revolutionary. 

[MingYouToo]

38 minutes ago, elevate said:

Yes, from what I've used of it, this holds true. It feels very much to be still in beta.

 

But with that said, it is kind of incredible what it can do right now, and it's clear that it will learn at an accelerated rate from what we've come to expect from these initiatives. 

 

Of all the random ground-breaking startups and whatever else from the last decade or so that undeservedly got attention, this is the first one that feels like it's planted the seeds of something truly revolutionary. 

That is very much true. This could be the eureka moment for service based technology, ChatGPT, as dark and macabre as it is, could signal advancement in labor for the service industry, waiters, cashier services could be aided by this tech.

 

However this is nothing to fret for physicians as AI is still probably a thousand years away from making clinical decisions and improving patient care. At best this'll probably the foundation for automated history taking, pathology lab interpretations (CBC, CMP, COAG), but the decision making and the actual actions will still be physician based.

 

This tech could prove VERY useful however in BPOs such as call centers and whatnot.

Edited January 29, 2023 by MingYouToo

[Pheromosa]

i'm using it to write a speech on academic integrity now

[Mr.Link]

The fact that it can literally be used to do prehistoric format homework like book reports and summarization with minimal margins of error and being almost unnoticeable. Low capacity high school teachers of filler subjects found obsolete. 

[Henry Torres]

Yep, it's not always right but capable of passing an exam