At the event “The Dark Side of Health AI and the Light at the End of the Tunnel” at the University of Oxford, Henry Jakobsson, Head of the BU MedTech at Horváth, and Jakob Groll, specialist at Horváth for the topic of AI in healthcare, interviewed Dr. Leo Celi, a physician scientist at the intersection of medicine, data science, and public health at the Massachusetts Institute of Technology. The discussion explores the opportunities and challenges of AI in healthcare, as well as the systemic changes needed to make AI truly impactful. They also talk about the role of startups and consultancies.
Leo, during the event, we discussed both the opportunities and challenges of AI. In your keynote, you asked how we can create an AI environment that is truly worthwhile. What is your answer to that?
Dr. Leo Celi / We need to prepare society to have a clear understanding of what we want to optimize and what we want to achieve with AI. Without answering these questions, we will face significant problems.
For me, the best preparation for AI is to reflect on what enables humans to thrive and what helps us become better versions of ourselves. Of course, that’s a very difficult problem to tackle. But if we don’t address these fundamental questions, we risk creating systems that perpetuate existing issues rather than solving them.
Often, AI models are evaluated primarily on accuracy. Achieving 95% accuracy is frequently celebrated, but that alone is not sufficient. What other factors should be considered beyond accuracy?
Dr. Leo Celi / We need to come up with metrics that better align with what we value – such as improving patient outcomes. What we’ve learned is that accurate models don’t necessarily lead to better health outcomes; in fact, they can even reinforce existing problems if based on flawed or biased data. The metrics we use should therefore be actual health metrics – those that reflect improvements in population health. For example, if an AI model is developed to predict maternal complications its success should not be measured solely by predictive accuracy, but by whether its implementation leads to a measurable reduction in maternal mortality rates. Only by aligning evaluation with real-world impact can we ensure that AI serves the needs of patients and communities.
Accuracy is important, but it’s not enough. Using historical data to define accuracy can lead to issues, especially if the data itself is flawed. We need to measure accuracy against what we want the data to represent, not just what it currently is.
For example, if an AI model is designed to predict maternal complications during delivery, the metric shouldn’t just be how well it predicts those complications. The real metric should be whether maternal mortality rates decrease as a result of using the model. Predicting something for the sake of prediction doesn’t help unless it leads to actionable steps that improve outcomes.
What we are doing now is incomplete. Simply predicting an event without providing actionable suggestions on how to respond to these predictions is insufficient. This means that the AI modeling community needs to work more closely with the people on the frontlines of care. You cannot develop these models in isolation, in a lab or classroom, without understanding how they will fit into real-world workflows and relationships.
And if you don't understand how the different actors play together, then what you're doing is you're making one cog of the wheel very shiny, and you don't see that the entire wheel is still stuck. What this requires is that AI modelers become better systems thinkers because the problem that we're trying to solve has existed for decades, if not centuries, and to be able to solve this is not going to be straightforward.
It's not about developing one algorithm. Clearly, you have to be able to incorporate this algorithm into a complex workflow. That might entail rearranging the different components and other algorithms, too. So that becomes very problematic for startups because startups have a very limited view of what they're trying to achieve. And for that reason, 99.99% of startups fail. To me, that's opportunity because I hate the saying fail fast, because in healthcare, fail fast means killing people. And we need to be to do better than that.
What is the best next step in AI development for MedTech and healthcare? Should the focus be on systematically changing the healthcare system, or should it lie in advancing the development of AI models themselves?
Dr. Leo Celi / It has to be both. You’re not just developing AI models; you need to be fixing the world at the same time. That’s a daunting challenge, and many people find it paralyzing.
But I see opportunities here. Startups, for example, often see challenges as opportunities. I think the solution will involve multiple approaches: building better models, redesigning systems, and zooming out to reflect on what we truly want to achieve with AI.
How can MedTech companies identify the right problems to solve with AI?
Dr. Leo Celi / This requires a paradigm shift. It’s not just about consulting the market or patients; it’s about co-designing solutions with them. We need to work closely with patients and communities to identify the problems that truly need to be addressed.
We also need to learn from history. One of the biggest issues with startups is that they don’t do their due diligence. They don’t ask why similar efforts in the past have failed. Everyone thinks they’re the first to come up with a solution, but that’s rarely the case.
There’s no repository of failed startups, no database where we can analyze what went wrong. Creating such a resource could be incredibly valuable. One of my favorite quotes now is: “What we have learned from history is that we never learn from history.”
So what I find almost miraculous is that in the last ten years that we have been doing this, we never have to come up with a clear roadmap, it just magically appears. So yes, you bring people who think differently across generations. The roadmap conjures itself before your eyes and to a certain extent, it makes our job easier.
But the thing is, it requires a leap of faith. You cannot tell investors trust us, it's going to work out, because these are very risk averse individuals. And they are not reassured that the magic will happen as soon as we completely revamp the way we think through things. But I could tell them that from experience. Magic happens if you completely, completely transform how you identify a problem.
Public-private partnerships are often seen as a way to drive innovation in healthcare. In your view, what are the key challenges these partnerships face, and how can we make them more effective in delivering meaningful health outcomes?
Dr. Leo Celi / We need to ask ourselves why public-private partnerships haven’t delivered better health outcomes. We’ve seen collaboration between academia, industry, and government – but often forget the most important partner: patients. That’s where innovation is needed. How can we rethink the way we engage, work, and learn from one another?
There may also be a business opportunity in designing infrastructure, tools, and AI that truly integrate diverse perspectives and lived experiences. But to do better, we need to understand why past partnerships have failed. Unfortunately, there’s a lack of data capturing how these collaborations unfolded.
To move forward, we must engage people who don’t think like us – those historically excluded from these conversations. That’s why we involve high school students and Gen Z in our work. They bring fresh perspectives and may offer truly out-of-the-box solutions.
What role can consulting firms like Horváth play in this process?
Dr. Leo Celi / The real opportunity lies in designing systems – systems for education, regulation, knowledge creation, and learning. AI has exposed how broken our current systems are. Incremental changes won’t fix this. We need to rethink, re-engineer, and redesign these systems entirely.
Consulting firms can play a critical role by bringing together diverse actors to redesign systems and ensure they are fundamentally different. Simply sprinkling AI into broken systems won’t work.
Take electronic health records, for example. These systems were designed for administrative tasks, not for learning or improving care. Incremental changes have been promised for years, but the fundamental issues remain. Worse, these flawed systems have become the foundation for health AI, even though they were never designed for that purpose.
AI isn’t going to fix these problems on its own. We need fresh voices, fresh players, and fresh ideas to reimagine the entire system.
This is an abridged version of the original interview.
About Dr. Leo Celi:
Dr. Leo Celi is a faculty member at the Massachusetts Institute of Technology leading the MIT Laboratory of Computational Physiology, an associate professor at Harvard Medical School, and an intensivist at Beth Israel Deaconess Medical Center in Boston (MA).
Your Contact
Henry Jakobsson