Big Data in Medicine: Promises and Pragmatism
Updated: Apr 14
Greene, Jeremy A., and Andrew S. Lea. "Digital futures past the long arc of big data in medicine." The New England journal of medicine 381, no. 5 (2019): 480.
Big data has been an ever louder buzzword in medicine. Data technology is often seen as promising in making "the tsunami of data" generated in medical practice useful to patients. Among other purposes, big data processing and machine learning systems are being developed to support the so-called precision medicine, where large numbers of data are distilled into treatments that fit individual patients.
Despite the exponential rise of technological innovation, Jeremy Greene's work, including his recent book The Doctor Who Wasn't There, shows that technologies don't only belong to the future, but also have their past. Perhaps surprisingly, in their short piece on big data Greene and Lea (2019, p. 480) argue that throughout the history of technological involvement in medicine
any fundamental problems remain unchanged - and not all challenges of digital medicine can be resolved by new technologies alone.
Concerns over how to manage the increasing amount of data in medicine are far from new - they have appeared already several decades ago. Since then, the solutions have been sought in technology. But digitalising medicine has been challenging from the start. While advocates of medical computing argued that technology could address the "growing physician shortage and allow doctors to focus on more human interaction" (Greene and Lea 2019, p. 481), its critics feared that the personal bonds between doctors and patients would be disrupted.
But there were - as they still are - also more practical concerns: besides being helpful, could computers be an additional source of errors? After all, technology is made by people, and people make mistakes and have biases, so wouldn't this be reflected in their technological products? Furthermore, since medical experts often disagree about correct diagnoses and the "ground truth" is elusive even in medicine, how could developers define standards against which algorithms could learn and against which algorithmic decisions could be evaluated? Such concerns persist to this day, despite significant technological advancements.
Already in the late 1950s, the first attempt was made in the United States to create a "total hospital information system" based on electronic health records (EHR). However, the project, designed as a collaboration between a hospital and a technological company, failed. Not only were there pragmatic issues with integrating computers into the hospital's day-to-day functioning, but there was also lack of mutual understandings among multiple stakeholders that was crucial to support institutional change at a hospital. Despite this early fiasco, in recent years EHR have replaced much of paper documentation in the US heath care system. Still, computers have proven most useful with billing, admissions, and clinical laboratories rather than with written patient records.
According to Green and Lea (2019, p. 482),
The promise of producing new, lifesaving forms of health data has yet to be fully realized, yet the EHR has altered doctor-patient relationships, increased the amount of time clinicians spend documenting their efforts, and been identified as a leading source of physician burnout.
The most successful attempt to digitalize medicine has been with managing the increasing output of published medical research. Digital systems for this purpose have been developed since the late 1950s in the US. The contemporary result of these efforts is PubMed, a universally accessible index of medical literature. However, Greene and Lea (2019, p. 483) warn that
today, researchers searching the clinical literature often conflate PubMed with the sum total of medical knowledge, without realizing that its content originally dated back to only 1963.
This misconception can have life-and-death consequences: in 2001, a healthy volunteer died in a drug trial, an event which could have been prevented had the researchers located the information of the drug's adverse affects which had been widely circulated in the 1950s.
Beyond managing medical data, the Covid-19 pandemic has given a significant push to the use of technology to deliver health care at a distance. Telemedicine, telehealth and telemonitoring systems suddenly became seen as the best, perhaps even the only option, to reach patients remotely, in a safe way. In the US alone, the use of telehealth has increased 38-fold in the first year of the pandemic. However, these technological solutions often depend on digital platforms that are not accessible to everyone. To overcome this issue, Greene, a medical doctor himself, turned to one technology that has been the most widely adopted for decades - telephone. When it comes to health care, pragmatic solutions still win over high tech promises.