Four Questions with Munjal Shah

Munjal Shah co-founded Hippocratic AI, which builds voice agents that take on clinical work health systems can't staff — post-discharge follow-ups, chronic care check-ins, medication questions. He spoke with Christina Farr about scope safety, who is liable when a clinical AI gets something wrong, and why he thinks AI ends up growing the healthcare workforce rather than shrinking it.

This conversation has been edited for length and clarity.

Christina Farr: Given the work that you do with health systems, do you think this is a moment for the healthcare industry to say, “We need to be thinking far more about security and safety?” Because if I'm an AI technology and I wanna inflict maximum damage, this is where I'm gonna look first.

Munjal Shah: I think healthcare's always been a target. I don't think it's a new target. I don't think there's a new need to defend itself. I think we just always have to defend against different threats, and we have to build systems that are robust, because healthcare is critical.

We've benefited a little bit from seeing what things can happen, so we know how to protect them. And unfortunately, it's hard to say— if I came to you in 2023 and said, “All right, we're not gonna launch anything beyond GPT-3.5.” Would you have known what to protect against in slowing it down? I don't think anybody would. So unfortunately, a little bit of this is we do have to see progress to see what is the defense needed to stop that progress from doing damage.

There's always a complexity here that I don't think is simple. Some of the things that we focused on at Hippocratic on the safety side was really the realization that, one, you don't wanna diagnose, you don't wanna prescribe. So we call that scope safety.

I think the second element that we realized was you don't just use one model. We use thirty-one models to supervise a single main model—thirty supervisors. Because we're doing it in voice, voice requires you to use a model that has a lower latency.

So it can't take forever. And if you look at all the smart models we're worried about right now, they're taking five minutes, ten minutes to give you an answer 'cause they're really, really, really big models. And so the analogy I draw here is, we're using thirty-one cat brains or mice brains working together.

And they're saying, “Hey, the chimpanzee we put in the cage just escaped.” I'm like, “Yeah.” But it turns out thirty-one cats do not equal a chimpanzee in ability to think through complex things. And that's really where we realized actually the way to make this safe is to use a whole host of models working together.

Christina Farr: That's a really good point and you see it across other industries as well — that everybody wants to use agents because it's the hot new thing and they wanna use it in ways that can create an amazing sounding press release. But oftentimes those are not the right ways to use the AI agents. I have a friend, for instance, who works on a trading desk and they wanna develop an AI automated trader, but then she's like, “Well, what about the applications just within the CRM to do way more basic stuff that I think AI is much more suited for?” And meanwhile, the point of actually making the trade is actually the ideal human moment because that's when you can connect with the client, right? And I think we're doing the same thing sometimes in healthcare, where we jump straight to the sexiest, most press-release-style applications and we don't think about just the basics. What's your view on that?

Munjal Shah: You know, if you think about what's sexy to a lot of people, “Well, I'm gonna build an AI doctor.” I'm like, “You're crazy. You're gonna kill somebody building an AI doctor.” So many of us that have had health issues, myself included, know that health is so complicated, and every person can be an N of one and has a specific set of circumstances that leads to kind of a corner case.

And I think it's those corner cases that make medicine complex—that's why you don't wanna do that. But if you look at the disease burden in the country, it's primarily driven from chronic disease. Now let's look at our three big chronic diseases. Take diabetes. Is it hard to diagnose diabetes? Do we need more diagnosis capacity? It's pretty easy to diagnose diabetes. What's your A1C? Okay, you have diabetes. What about high blood pressure? Actually, it's pretty easy to diagnose high blood pressure. But managing high blood pressure is super hard, and managing diabetes is super hard.

So what we need is a lot more management capacity for these chronic conditions. My mother has high blood pressure, and she can never seem to get comfortable with her high blood pressure meds. They make her dizzy. They made her almost fall over the other day. She just doesn't like 'em, and so she doesn't take 'em.

And titrating her medication just-right is something that the system doesn't have enough capacity to [do]. To have her go back-in every single week and tweak it a little and tweak it a little. The same thing's true for migraine meds. The same thing's true for Parkinson meds with Levodopa.

I mean, there [is a] constant need for even titration. Why don't we have more titration capacity? In fact, why don't we start there? 'Cause that might lead to more adherence. That might lead to more compliance. That might lead to better health outcomes. Now that would still require a more clinical agent than the ones we sell today.

That would require an FDA approval. If we are gonna go beyond the scope of an RN, which is where we are today, I think you go into a little bit of the scope of an NP or a PA, where you're just doing some of that, and you slowly move your way up instead of try to move your way down from a doctor. We've taken this bottoms-up strategy 'cause we just think it's a lot safer.

Christina Farr: We just had John Whyte from the AMA and Zeke Emanuel — who wrote that big piece with Neal Khosla, son of Vinod Khosla, in JAMA — reflecting on this exact question. Can you create an AI doctor? We had the two of them [on Lifers] on this debate. Many physicians that watched it texted me, and they said, “Look, we'd be fine with the lowest risk stuff,” like you mentioned titration. An AI can do that. We'd be okay with considering prescription refills as another example, which we've already experimented with in Utah with the company Doctronic. But I think the big question for them is liability. So if a mistake is made in this model or in this imagined future where we're ceding more and more to AI, is it the AI company, is it the Hippocratics of the world that take responsibility for a mistake, or is it still the institution and the provider that are using these tools with their own patients?

Munjal Shah: I believe you cannot build a clinical AI without taking liability. And so first, if something were to go wrong, the health system we work with or the payer we work with would probably be the first point where a lawsuit would occur.

Our contracts allow them to come after us. There are many software vendors out there whose contracts do not allow — they have a very limited set of things that they'll take responsibility for, and ours do. Not only that, [but] we built kind of a fortress balance sheet so that even if that insurance is exhausted, we are actually good for it.

Because there's no point in saying you're liable if you're not good for that. But all of that being said, we're not here to debate financial liability. We really just don't wanna hurt patients, right? And I think that's where we've just taken so much care to make sure we built a system and a redundant system and a redundancy to the redundancy and an escalation protocol that transfers the call.

And we've seen this in action now. When I look at all of the calls we've done, we've [made] some very high clinical calls—still within the scope of an RN—but we've [made calls] on patients recently discharged for congestive heart failure from the hospital. [We've] followed up on them and made sure that they were doing okay. And as you can imagine, there's a lot more risk in that than just calling to check on your blood pressure.

One of the things that we've seen is you can ask your AI, “Hey, can I take some ibuprofen for this condition?” Or, “I'm having some pain after the surgery.” And a lot of times the answer is, “Sure, just don't take more than two thousand milligrams.” But if you have chronic kidney disease stage III, B, or IV, you can't take two thousand milligrams. In fact, it'll kill you, because your kidneys just can't process it.

We've actually put an engine that does exactly this. We call it the condition-specific disallowed OTC engine, and it's a model that just looks at every turn of the conversation and says, “Are we talking about over-the-counter medications? Okay, does this person have a disallowed condition?”

When I co-founded the company, we brought together three kinds of expertise: entrepreneurial expertise; my co-founder, Meenesh, who's a Johns Hopkins-trained doctor, who actually ran El Camino, which is a two billion dollar health system, as their COO, and was at a number of other health systems before that in senior executive roles and was an ER doc for many years.

And brought that together with Subu, who's our chief scientist and actually was lead author on the Orca instruction tuning paper, which was one of the seminal papers on how to fine-tune models. And you really need all of that expertise together as the founders of the company and then expanding on that as we built out the entire company.

I think today fifteen percent — we're about 253 people now, and fifteen percent of the company are full-time clinicians. If you count other clinicians we hire on a part-time basis, it's probably far more than that. And that's just unusual in a tech company. Usually, you have one. You have one as your CMO, and you're like: “Okay, I have a CMO. Great.” And I have a few advisors. We have a very different approach here.

Christina Farr: I'm gonna say the quiet part out loud. Imagine you're Travis Kalanick back in the day building Uber. You knew that at some point you were gonna piss off the taxi lobby. You knew that at some point the government was going to have questions about what you were doing, and you knew that you were changing culture. When you were building this business, on the one hand, you've gotta know — and even though nothing you've said so far would, if I was running a nursing union, nothing you've said so far would flag for me necessarily. But if I'm a health system and I'm gonna spend money on a company like Hippocratic, and I'm gonna spend off my own balance sheet, at some point I'm gonna wanna start seeing ROI. And I'm thinking that the ROI, and you tell me if I'm wrong, but that probably looks like I'm not hiring quite as many human clinicians and members of staff as I did in the past, and we both know that healthcare is a huge employer in this country, and that is where so much of the spend resides. So did you go into this company knowing at some point you were gonna have to face that question of, are we building something where healthcare is gonna have a smaller workforce, and that's a fight that we're gonna have at some point?

Munjal Shah: I think it's the opposite. I think healthcare's gonna have a bigger workforce.

In fact, I think healthcare has to have a bigger workforce. I look at the aging population in this country. Do you think a seventy-year-old uses the same healthcare as a thirty-five-year-old? Do you think they use two times the healthcare of a thirty-five-year-old? No, they use like ten times, right? Count the number of appointments they go to.

My mom is eighty-three. I know how many doctor's appointments she goes to. It's a lot. My dad's eighty-four, same thing. And so we need way more capacity, not less capacity. But yes, we'll go bankrupt providing that additional capacity. We don't have that many people who want to be — there's a nursing shortage in the country already.

So then if you take my aging population thesis, that nurse shortage just gets worse. And so what are we gonna do? Well, actually, the interesting insight that we found that even was new, if you had talked to me a year and a half ago, I wouldn't have told you this. What I'm seeing now is that the use cases that work, that actually help patients, that actually bring ROI for health systems are not the ones you think they are.

So there's Copilot, Autopilot, and Infinite Pilot. This is this framework we use. And so Copilot are things you do with the human in the loop.

Most people knew us for our main Autopilot product, which is where we started in the middle. We said, “There's no leverage in just leaving somebody in the loop. You're just not gonna get enough done. You'll get ten percent more efficiency gain, but you won't change the world, and you won't deal with this coming crisis of capacity.”

And so second, we thought, “All right. We're gonna go after things you do today that you just can't staff that we can help you do.” And yes, it is cheaper, fine. But those did not turn out to be the use cases that worked the best. The ones that worked the best are the Infinite Pilot — things you never thought to do until you had an infinite supply of clinicians to throw at the problem.

So recently, there was a heat wave in New York, and this was for an at-risk Medicare Advantage payer. We called fifty thousand people at the hottest two hours of the day and did a full heat-wave, heat-stroke education and told them where the nearest cooling center was.

Nobody does that today. Why don't they do that? 'Cause it costs about ten million dollars a day if you do it with humans. You'll have to find two thousand to three thousand clinicians to make those calls. The ROI doesn't work.

Watch the full conversation on Lifers: