AI is now outperforming human doctors in the emergency room diagnoses, new Harvard study reveals

A new Harvard study has found that advanced AI models can outperform doctors in diagnosing patients, including in high-pressure emergency room (ER) scenarios. The researchers pitted OpenAI's advanced o1 language model directly against hundreds of physicians across multiple diagnostic touchpoints, finding that the AI consistently outperformed doctors in both diagnosing conditions and planning clinical management.

The researchers found that o1 performed on par with or significantly better than the human experts on a variety of tasks:

“Although applying AI to assist with clinical decision support is sometimes viewed as a high-risk endeavor, greater use of these tools might serve to mitigate the human and financial costs of diagnostic error, delay, and lack of access” the researchers said

“Our findings suggest that LLMs have now eclipsed most benchmarks of clinical reasoning, motivating the urgent need for human-computer interaction studies and prospective clinical trials to rigorously assess the potential of AI systems to improve clinical practice and patient outcomes.” they added

However, the researchers also warn that clinical medicine is filled with non-text inputs like patient's level of physical distress or the interpretation of medical imaging. This, they suggest, means there is a need for future research to asses how humans and machine and collaborare effectively.

In a statement to The Guardian, Arjun Manrai, one of trhe lead authors of the study noted, “I don’t think our findings mean that AI replaces doctors,”

“I think it does mean that we’re witnessing a really profound change in technology that will reshape medicine.” he added