Machine Learning for MDs Weekly Digest
What’s New in ML for MDs
Welcome to the ML for MDs Newsletter. The mission of ML for MDs is to connect physicians interested in machine learning. This newsletter provides the most relevant news, journal articles, and jobs at the intersection of medicine and machine learning.
Fun Facts
- The earliest case of data visualization being used to influence public policy was around getting better sanitary conditions for British soldiers.
- An AI-powered software was created that could predict the results of the Oscars with 90% accuracy.
This Week’s Top Stories
- Federated Learning for Instruction Tuning (FedIT) is a “collaborative learning technique…[that] enables many clients to learn a shared model jointly without sharing their sensitive data. In particular, in our proposed federated instruction-tuning, clients initially download a global LLM from a central server and subsequently compute local model updates using their respective local instructions. These local updates are then transmitted back to the server, where they are aggregated and integrated to update the global LLM.” So client data remains on user devices for improved privacy.
- Because I am cheap and love the name: Authors of a paper on FrugalGPT found that “models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude”. They “propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.”
Weekly summary
This is a simple overview of a very complex subject, but I thought it was organized in an intuitive format, and added a few of my thoughts also:
Data Collection and Algorithm Development
Data Collection
Problems with medical data collection
- Institutions hesitant to share their data due to privacy concerns
- The NHS allowed DeepMind to access 1.6m patient records without patient consent in 2018 to widespread outcry
- General Data Protection Regulation (GDPR) in Europe can make it harder to share data and stifle innovation
- Hard to analyze the quality of the data used
- Gaps in patient care when they change institutions
- My addition: many healthtech companies are positioning themselves with patient data as their revenue plan making sharing more expensive and less likely
Potential solutions
- Employing client-side encryption
- My addition: New AI privacy models may help with solutions
- Use federated learning to train models without data dispersion
Algorithm Development Concerns
Potential Bias Due to:
- Lack of adequate data related to ethnic and racial differences
- Reflection of biases inherent in medicine
- General concerns in algorithm development like overfitting, data leakage, explainability
- “it is worth noting that the mechanism of action of many commonly prescribed medications…is poorly understood, and that the majority doctors have only a basic understanding of diagnostic imaging tools like magnetic resonance imaging and computed tomography”
Possible solutions
- Thoughtfully building multi-racial/ethnic data sets
- AI methods to reduce bias in data sets
- Transparency in model development
Ethical concerns
Possible problems
- Accountability/Liability for errors
- My addition: In any workflow involving physicians, physicians will likely be ultimately liable, as we currently are for any decisions made by any humans that make suggestions to us
- Not clear if current malpractice covers the use of AI or when it will enter the standard of care
- Fairness
- Transparency
- Informed consent
Possible solutions
- Governmental intervention
- NHS developing an Ethics AI Initiative
- Governance frameworks
Social concerns
Possible Problems
- New/worsening distrust of medical system
- Concerns about job loss
- Automation bias and de-skilling of physicians
Possible Solutions
- Precautionary approach (EU)
- Assume bad things will happen and legislate proactively
- Permissive approach (US)
- Wait until things go wrong and then make rules about it
Clinical implementation concerns
Possible Problems
- Efficacy data currently lacking but seems likely to demonstrate efficacy
- Same information may occasionally give different answers
- Large capital investment by health systems
- Workflow disruption and impact on patients during implementation/system learning phase
- My additions:
- Workforce training regarding limitations of AI
- Possible harms of failures and hallucinations
Possible Solutions
- Involvement of clinicians
- Validate and use on appropriate populations
Community News
- If you haven’t introduced yourself, please do so under the #intros channel.
Thanks for being a part of this community! As always, please let me know if you have questions/ideas/feedback.
Sarah
Sarah Gebauer, MD