It’s been a while since I talked with Owen Tripp, CEO of Included Health. They’ve now introduced Dot their AI companion which had a big upgrade last week. We talked a little about that and I snuck in their video comparing the Dot Experience with a standard LLM. But the conversation really got into how do we make AI safe and trustworthy–which is definitely the hot topic these days. Owen is putting together a coalition of the willing to work on that exact topic. I’ll be watching closely–Matthew Holt
This was such a great discussion I wanted to publish the transcript. The way I do that is to copy the YouTube-generated transcript and drop it into Claude to smooth it over. I then read it, and if I think it’s made an error, I dip back into the video and listen to what actually happened and make a correction. This is all to say: I think this transcript is pretty accurate, but it might have a bunch of AI- and human-generated mistakes.
Matthew Holt
Matthew Holt with The Health Care Blog, back with Owen Tripp, the CEO of Included Health. Owen, we haven’t chatted in a while, and the good news is absolutely nothing has happened with Included Health, or about this topic you may have heard of called AI, in the last six months — but apparently we’re all stopping now, or something. Anyway, just to bring everyone up to speed: Included Health, for those who haven’t seen it before, is a company that now encompasses a lot of different healthcare services, going all the way from second opinions — which I’ve used — to primary care, all the way to navigation, working with many big employers, including the biggest public-sector employer in the world, in the union, I guess — CalPERS — and many others. You’ve been talking a lot about a number of things, like health plans, but probably the most significant thing Included Health has done in the last year is come out with its own AI platform, called Dot. So let’s start here — we’re going to go bigger into AI, but let’s start with: what is Included Health doing for its customers with AI right now?
Owen Tripp
Yeah, good to see you. One of the things I think we can celebrate at the start of our conversation, especially as two old dogs in healthcare — I’m not sure, actually — I think on this podcast we’ve talked about the story of us first meeting, in which you told me this would never work.
Matthew Holt
That would never work — I said that second opinions alone would never work.
Owen Tripp
Anyway, one of the things I think we can celebrate is that this topic, which hopefully you and I are going to spend some time on today, is really one where I don’t think we’re starting from behind. Those of us who are technologists working in healthcare really feel like we’ve always been trying to drag healthcare into modern frameworks — data compatibility, member-facing and member-usable experiences, provider tooling — all that stuff has always felt like we were operating years, if not decades, behind. This is one where I feel like the best applications of AI today are already in healthcare, or at least some of them are, and Included Health is very proud to lead the way. It’s been the most fun I’ve had on product, product design, and technology since probably when we launched into navigation almost a decade ago.
So I’m excited to talk to you about it today. Really, what’s on my mind — I can set up what Dot’s been doing, but I have to tell you that I come into this conversation today, on Tuesday, September 15th, with my mind and heart fully on: how do we make sure that all of this innovation, all of this incredible power we’re putting directly into people’s hands to access better healthcare, is safe, is private, is free of bias? These are topics that are just screaming loud in my head, and I want to tell you about what we’re doing with Dot, but specifically what we’re leading with on those key domains of safety and privacy. Matthew, if I may, I’m just going to set up what Dot does for your viewers, because they may not be super familiar with it. If you’re not one of the many, many million Included Health members today, you might not have had a chance to play with this. So Dot is a member-facing assistant that can straddle mind, body, and wallet across the healthcare domain, and indeed tries to make those differences actually disappear, because what we know about member and patient need is that when you’re going through something, you’re likely going to have medical questions, you’re going to have financial questions, you’re going to have administrative questions about what’s covered and how you get access to it. You’re going to want to merge physical and mental health in a way that feels natural, and doesn’t require you to take some off-ramp into another agent, another experience.
So Dot’s starting strategy was really to be your front-end, superpowered medical member of the family who could take you across all of those domains. And to give you a sense of history here — while we’re talking about it today with a freshness, as if we’re just launching — this is really about two years of experience we’re going to talk about today, in playing around with, and then ultimately productizing, what that member-facing AI can do. We’ve launched this to millions of members. This isn’t some speculative set of information I’m going to share with you today — this is born specifically out of our direct experience, watching people have very long conversations with Dot. In some cases these go 20 or 30 minutes of engagement, and people are having conversations where they disclose, on a comparative basis, more information than we sometimes see in our primary care practice, where a human is acquiring the same level of history, concern, need, and chief complaint, and so on. We see that Dot is able to handle seamlessly questions about what’s covered by the plan design, who’s in network, what they’re specialized for, and then — today, and this is the big update we’re sharing with the world — in this most current version of Dot, we can actually convert that advice, recommendation, and understanding into action. We think that’s going to be a big part of the future of healthcare AI — that it all has to convert to action. So you can schedule appointments, you can follow up on medical records, you can explore and pre-select different members of your care team to make sure they’re part of your ongoing care. So that’s a little bit of foundation and background on Dot, and I’m happy to start our conversation there.
Matthew Holt
Well, let’s dig into a bit of what’s going on here, because you’re seeing the big LLMs — and some of the smaller ones, I’ve been playing with one called Inciteful Med, which is very interesting — as well as Anthropic, ChatGPT, starting to advertise how they’re linking back to find your medical record at whatever place and bring it in — sometimes using partners, sometimes doing it directly, who knows — but they’re clearly starting to go down the path of realizing that just answering your question about the generic issue you have isn’t enough. So you have the advantage of not only knowing a lot about your members and customers — what plan they’re in, probably a lot about their health, and so on — you also have on your team, primary care, mental health, specialty referrals, and a bunch of other stuff. So now, moving from questions to action — yes, I get the admin, my deductible is whatever, if I want this drug it’s going to cost me that much, which is very valuable — but the bit people are getting most interested in, and we’ll lead into our later conversation about privacy, security, safety, and where this all goes with AI, is: okay, how much can the AI do, and at what point does it need to bring in a human? And how much time, effort, energy does the human — the clinical provider, or whatever flavor — save because the patient has had that conversation with the AI, with Dot, first? So give me a flavor of where that line is today — what people are saying, when does it go to a human, how does it organize that interaction — and then where do you think it’s going?
Owen Tripp
Yeah, let me give you some good use cases of what it can do today, and then I’m going to back up and go to the part of what we bring to the equation that makes all this possible, because you’ve covered a couple of pieces of that, but I want to highlight a few others. So — what can a super-intelligent automation do when it has agentic capabilities to reach across the rest of the system and conduct activities on your behalf? I highlighted the ability to schedule an appointment — that in and of itself is actually quite a complex experience. You have to understand what health plan and insurance coverage that member has, understand their medical history and specific needs, be able to select a provider or providers well-suited and high-quality for the underlying member need, then find a way to manage the schedule on behalf of both the member and the provider, and then confirm that appointment and make sure all the necessary information is transmitted to both the provider and the member. So that’s actually, inside of it, quite a complex set of activities. All of those are covered, and can cover outpatient activities of really any variety today.
One of the cool things we’ve worked on, which might seem small, but I can assure you anybody who’s recently tried to book an appointment has experienced this: you go in thinking you want to book a specific provider, only to discover that provider isn’t available, but the office, in a friendly manner, suggests five other people who could take you faster. Well, to handle that situation, we have to have switching logic built in that helps us understand, in advance, whether any of those other providers would actually be suitable. I think we’ve talked in the past about our approach to algorithmic assessment of quality, so we need to know immediately that, of those five other potential providers, one or two are actually okay, and three others are on the no-fly list, and be able to seamlessly switch that logic in the moment. So just unpacking that one transaction in healthcare helps you understand the complexity there. But that’s not Dot’s only trick. Dot can fetch and review your plan documents, your explanation of benefits — we can and will alert you to charges that are out of line, and then automatically suggest that we should go fight on your behalf. That’s why we’ve returned millions of dollars of patient-responsible payments to the healthcare system, because they were overbilled by the provider systems themselves. I’ll give you a third example — we all know that many of us who enjoy commercially covered health plans, the kind our employers provide, with all their benefits — of course, Included Health offers one to all of our employees and their families — well, those benefits aren’t always easily recognized or memorized. So another trick Dot has — one we’ve had from the beginning, we’re just sharpening the capability — is to say, ‘Hey, we know you’re coming out of this knee episode, we’ve talked about your bill, we’ve talked about the best place to get your pharmacy, but now we really need to talk about physical therapy — and you do have Hinge as a benefit, which is going to be less expensive for your specific need than other providers. We think that’s a great place to start — can we go ahead and enroll you directly?’ That’s the way we want to build across the rest of the ecosystem — instead of paper partnerships with other benefits providers, the intent is to build freeways that let members connect directly to those benefits. Those are just examples of how we move across mind, body, and wallet — of course all of our virtual care services too, whether it’s behavioral health, from coaching to psychotherapy to acute psychiatric care, primary care, urgent care, specialty care, all the second opinions — all of these are integrated through this singular AI platform, and the idea is that the member shouldn’t have to figure out where they need to go, which is, of course, what we’ve all had to do up until this point.
Matthew Holt
Yeah, I’ve actually recently been enrolled in a new plan, and it’s quite something to see they have a list of, I don’t know, 20-odd point solutions, including maybe seven or eight in mental health and three or four in MSK, and you think, how on earth would anybody figure out the difference between these? So I think if you have an AI tool — whether it’s Dot or anything else — especially one that’s linked into those and knows, okay, I’m an employee at Walmart and I have these six things available to me, there’s probably one, or one or two, that are actually better for me — and if you’ve got the data behind that, that’s a very powerful thing.
Owen Tripp
Well, and this is — you and I have been not only students, but I think boosters, of this ecosystem for a long time, and one of the things I’m excited about is that point solutions, which have always had clinical validity but just haven’t had much engagement, might see that they’re used more, and used more appropriately. I’m really excited about that. You can go through the list — fertility benefits, neurodiversity benefits and support systems, weight loss, other cardiometabolic care — on down the list, you can find the most appropriate solution. Those should have a deflationary effect on healthcare, because in almost all cases the digital equivalent of that service or solution is less expensive and easier to use. The problem has just been that it’s never been easy to find on the benefits menu. Now, there will probably be some pricing effects, as those same vendors figure out that the PEPM contracts they’ve been on don’t make sense anymore, but I think all of that would be a welcome change.
Matthew Holt
Yeah, and how their stuff gets paid for, and who recommends it, and all the rest, is a separate bucket we can get into. The Peterson folks are running around now saying the kidney disease programs don’t work — I saw that from them yesterday.
Owen Tripp
Yeah, we see that.
Matthew Holt
All right, let me push you a bit more on this. So I’ve been telling Dot something about my health, and all the rest of it — how much can Dot tell me? And at what point do you decide, okay, I’ve got to stop and get you an appointment? And does it have to be an appointment, or is there a live-human option? What kind of humans do you have available right then and there, versus who I have to wait 20 minutes for, versus who requires an appointment next week?
Owen Tripp
This is a great question, and I’m going to answer it on two levels. First, let me start with, actually, the provocative wrapper of this whole thing, just to make sure we’re in agreement: AI today — even consumer-grade AI, which isn’t what we’re talking about here, we’re talking about enterprise-grade, medical-grade AI — but even the consumer-grade AI available to most people already can do more than its owners are letting it do. In other words, we could push these things all the way through to diagnostic care. It just turns out that most owners of those AI systems don’t want to inherit the liability and the ambiguous law associated with that. But I think that’s coming. So, with that as a baseline for where we’re starting, the problem actually becomes: how do you safely deploy and manage those agents, or assistants, to help people get care that’s actually safe and contextually relevant? I’ll get to the human-in-the-loop part in a second, but a lot of our early effort was really about making sure we established and trained for guardrails that helped Dot specifically alert and pull people into conversations as needed, either to advance care because it needed human review and approval, or to schedule a necessary follow-up. And critically — although thankfully these incidents are fewer — to make sure we’re escalating immediately on signs of suicidality, signs of an acute event requiring intervention. We’ve had this built into even our original virtual care practices, by the way — a clinician could push a button on the back end and dispatch EMS, because we know where people are calling from, and unfortunately we have to do that hundreds of times a year. So now Dot can do the same flagging from a safety-incident perspective, but also has a lot of guardrails so it doesn’t wander off beyond the scope of prescribing, recommending, and so on. I think we’ve shown you some side-by-sides of consumer-grade AI versus Dot on these topics, where Dot is operating more in the context of what is safe, efficient, evidence- and law-based care in each of the states in which we operate.
At this point there’s a 2 minute demo of how Dot works
Matthew Holt
I get that, and — my personal experience, which hasn’t included using Dot, because I’m still certainly not as good a health member — one day I’ll get a job with the State of California and be able to use it.
Owen Tripp
You should be a fireman in our home county (note: Both Owen and Matthew live in Marin County, CA)
Matthew Holt
I should become a volunteer fireman or something useful, once I get my shoulder and knee fixed, that’ll all happen. Anyway — but feeding my now rather extensive imaging and diagnostic history into both Inciteful Med and Claude, they are coming up with quite a lot. You’ll say, ‘what about this,’ and they’ll say, ‘yeah, in many cases here’s what we’d recommend for this to happen — here’s a course of action, you should probably get this procedure, you might want to get that procedure. If you’ve got this injury, you may be able to wait and do some PT; with this other injury, you may not be able to wait, because it’s going to cause muscle loss, bone loss, whatever.’ So I’ve got all this running around — I’m getting a lot of advice to go talk to the medical system with, from these already. So is that diagnostics?
Owen Tripp
It’s getting there, right — it’s certainly part of what you’d normally get when you’re talking to a friendly cardiologist or orthopedic surgeon across the fence, or at a cocktail party — that’s the kind of thing you’d get towards.
Matthew Holt
So I guess, how far, in Dot’s case, are you going down the path toward ‘here are the types of things that could happen’ before you actually introduce someone to a specialist or primary care, versus, “it sounds like you’ve got enough going on here, based on what we know from your medical record, that we want to get you in front of those people’?
Owen Tripp
Yeah, let me answer that on two levels. Right up front — today, in its current release, it will hand you off, when that clinical interaction needs to happen, either to a member of our own practice, or a recommended member who’s high-quality, taking new patients, accepts your insurance, and is in the local community — and I think we do that really quite well, and continue to work as hard as we can to reduce every piece of friction. So I’d say the commercial consumer LLMs are running ahead of that.
Matthew Holt
They’re not running ahead of — I mean, I’ve had ChatGPT and Claude tell me specifically, ‘okay, you’ve had this condition, you need to get this fixed using this kind of—’
Owen Tripp
Oh, no, no, no, I agree with you — I haven’t tested all of those things within the last 24 hours, and they’re moving so fast. No, this is why I said that provocative foundational statement — I think what’s actually possible today, if we removed the governors and guardrails from even our own AI, built here in San Francisco, we’d find that we could let it loose on any number of problems, and it could — if not specifically diagnosing and ordering — take you 97% of the way there. That is not, at least today, the safe, appropriate, and most effective way to think about healthcare delivery, and we’re going to get to that in our conversation. But the other level I wanted to answer on your prior question, which I think is useful for you to understand, and a way to better know our company — we really think of our corporate citizenry on two levels. There’s what we’ve put into Dot, which I’m telling you about today and we’ll continuously update, and then we think of ourselves as a massive stakeholder and influencer on what future possibilities should and could look like. There, we’ve been public about this — we’ve partnered, over the last year, with one of the leading frontier labs on how to do safe AI treatment of care, and there’s a study we’re working on with them, actually several studies now, some already published, some still to be published, that measure the efficacy of these tools in doing exactly what you’re describing.
However, getting to a place where an AI can appropriately diagnose or differentially diagnose your sinusitis is going to be a win, but not sufficient for where healthcare delivery needs to go. It doesn’t actually lead to activation against that problem — meaning, okay, now let’s go through these therapies or recommended lifestyle changes. It also isn’t particularly accountable or responsible to your overall healthcare picture — thinking about how much it’s going to cost, what’s going to be covered, what the next steps are. And it doesn’t incorporate the whole-person-care element of this.
If I can, I want to pause, because I skipped over this, but I think your audience tends to be pretty sophisticated on this stuff, so I want them to build the whole picture in their mind. We have the ability to train and deliver and be intelligent on all the topics you’d expect of a modern AI company on the delivery of care, and that’s great — but to actually solve the healthcare conundrum in the United States, you’re going to need a lot more than that. So the other things we bring to the party: a connected EMR that connects to over 70% of practicing physicians in the country, where we can do push-pull on records — it’s not just about what you supply to us, or your Apple Health record, which at best is a very thin slice of your overall health experience. We also merge in the entire financial experience — what you’ve spent in claims, what your claims history tells us about where you’ve been, what your pharmacy benefit manager data tells us about where you’ve been and what you’ve experienced — and then, critically, the actual overall wrapper of what your plan design allows you to do and not do. Unfortunately, on the cost side, this is where people find themselves hitting the rocks — they think they’ve found the perfect treatment or medication, only to discover it’s not covered by their plan, or that it’s covered but at enormous personal expense. So to really address the whole problem, you have to merge all of that information together, and that’s something we’re already doing, and we think it will be possible for other AI companies to work on, but not necessarily out of the box the way it is for Included.
Matthew Holt
Okay, I get that, and I think that in itself is a couple of tremendous leaps. The first is — setting Included aside for a moment — a random person can take their diagnostics, scans, and labs, get them online, and feed them either via the LLM doing it themselves, or just by copying and pasting, and get back a lot of information that really could only have come from a clinical professional or doctor, you know, two or three years ago.
Owen Tripp
Totally.
Matthew Holt
And you get much smarter — as I said, I’ve been going through a bunch of health stuff, and I’ve gotten incredibly smart about some of it very quickly. And then you do worry that sometimes you’re the roadrunner No, I mean the coyote running off the cliff chasing the roadrunner — you think you’re doing really well, until you realize you don’t have years of medical practice to fall back on.
But no, there’s no question that many patients are getting very smart about this, and really diving in — you hear these stories about people whose kids have some weird condition, couldn’t find out what it was, they put it into ChatGPT, and it spits out the answer that 27 doctors didn’t give them.
Owen Tripp
Yep, you have that whole movement, and that’s a huge win.
Matthew Holt
It’s a huge win. The second thing is what you’re doing, that most people don’t have yet — and I think we can both continue to poke the rest of the healthcare system, plans, and others to get people there — which is, yes, it’s all this stuff you know about people from the admin side, the financial side, the clinical side, and you haven’t even started chucking in things like continuous monitoring, what’s in your Apple Health record, or your MyChart, or whatever it is — where there’s a ton of things going on that probably haven’t been picked up, or if they have been picked up, haven’t been explained clearly and properly, or for whatever reason, people are left in kind of a mixed state. There should be much more clarity coming out of these AI tools — and a tool like Dot, I’m sure, does that: helping get somebody to the right place, but also helping prep them to get there, and helping the person on the other side — the provider they’re seeing — know what’s coming. Can you talk a bit about how you see that: the more informed AI, the more informed member, and the more informed clinician working together in the future?
Owen Tripp
Yeah, it’s a good push, and, to make sure we’re all on the same wavelength, these things, for as magical and powerful as they already feel, are still only training on largely the open internet, plus a few proprietary data sources they’ve purchased. A lot of what needs to happen when you go through the rotating door at the front of a hospital or clinic, enter the exam room, and then have a bunch of decisions to make and steps to take afterward — that workflow is actually hidden from, or opaque to, the AI. And why is that important? If you’re trying to advise on treatment decision support and build consensus among a provider, a broad care team, the member, and perhaps the member’s payer, you actually need the ability to understand what’s going to happen next — that domain knowledge about where the patient is likely to need to go. If people don’t understand what I’m talking about, go test the AI on exactly the experience of a patient post-operatively, for your favorite condition — what’s the recovery room going to look like, what’s going to be billed, what’s the anesthesia contract at this hospital, what’s going to be on the formulary at your pharmacy. It’ll quickly tell you those things are important, but it’ll have no ability to actually parse them and make them available to you. That’s just going to take a domain-specific company to really push through those and build those workflows, and make sure we’re all operating on the same page, with a shared understanding of what needs to happen. I’m very hopeful about that — not because I see it working already, there’s a lot of room to grow and do more work.
But ultimately, and this gets into the big part of the conversation I want to make sure we cover, we have to do that in a way that addresses the fact that all this information members are interacting with through these AI systems has to be built around a privacy notion that wasn’t considered in HIPAA. It has to be built around a set of safety standards that no malpractice laws really consider, that no compliance laws today really consider. You have to think about these AI models like a new version of a health system, one with really different standards around personalization, privacy, and safety — because, while I think we’ll all be delighted by the magic of the access, and frankly the low marginal cost of that access — and these are good things, like the examples you’ve given, the ones I’m giving, these are wins for society and humanity — they come with massive risks. They come with risks to safety, risks to privacy, risks of bias, risks of malicious intent and manipulation. I’d love to tick through a few of those examples today, if we have time, because I think this is where the industry — and here I mean the people building these things, me and my colleagues certainly, but also every other company out there — are real, big stakeholders in where this goes from here.
Matthew Holt
So before we get there — let’s say I’m a clinician working with Ami Parekh (Included’s Chief Clinical Officer), or one of your team, and a member comes to me who’s been using the AI. How much of that conversation gets shuffled through to me, so I can see what’s happened to them and where they’ve been, before they arrive in my telehealth visit?
Owen Tripp
Oh, you get all of that. Thanks for asking. Matthew, the fundamental architecture of our company is that — in the past I’ve talked about how there are these two places in healthcare, and we need a third place. You’ve got the health insurer, who pays for the bills, exists in the suburbs of town in a windowless building, and is in the business of paying claims. And you’ve got the health system, which is often in the middle of town, and has no idea how much things are going to cost. They have their big system — they call it the EMR — the payers have their big system, they call it the TPA, administration, claims processing, whatever you want to call it. We need to merge those two things together, because, to your setup, the provider needs to understand, when they see Matthew, that all the things Matthew has already said that are relevant to his history get incorporated into the discussion you’re going to have. I don’t know if you remember, but at the outset I mentioned people are having these extraordinarily long and detailed conversations with Dot. Part of what’s powerful about that is that we’re able to collect history that, to the member, may have seemed inconsequential — they don’t connect it to their symptoms, they don’t connect it to anything that matters in their medical life, but in fact it’s hugely consequential. This is a shared conversation — if you think of Dot as another member of the care team, alongside your human provider, with all the support services we already offer today, then you’re actually developing that whole picture, and everybody else is contextually aware. It’s not sitting in these two isolated systems, with the poor member trying to merge it together themselves.
Matthew Holt
So you’ve got, I assume, a summary of that chat, or that conversation, in front of the clinician? We can start here talking about safety and liability and privacy, right — you’ve got a self-contained universe where it goes from the member’s chat with Dot through to some summary that helps advance, and inform, the clinician of what’s going on when Matthew, or whomever, shows up in their virtual exam room?
Owen Tripp
They’re using that.
Matthew Holt
I assume — I don’t know, but I assume, given that you’re also using some kind of scribing and summarization to develop the record — you’re building the record as you go, but as you said, you’re also surfacing things that presumably are important to the clinician. So that all sounds great and wonderful. Are you concerned about how Included Health is managing that process, or are you concerned about how everybody else is managing that process? Or are you concerned about both?
Owen Tripp
I’m concerned about both. And I’m ready to update you today on things I think we’ve started to figure out and taken appropriate steps on, as well as some areas we know are important that we still need to figure out ourselves.
Matthew Holt
So let’s hold for one second — this week, or the last six or seven days, I can’t even keep count anymore — we’ve had a lot of back-and-forth. The fellow who left OpenAI because he thought it would be safer, and then decided it wasn’t. Dario Amodei has written a 4,000-word piece about how we need to get external people into Anthropic and others, how everyone should work together — not only nationally, among the frontier labs, but also, let’s get the Chinese on board, because this thing could kill us all — that back-and-forth. And Trump says it’s all fine, which almost certainly means it’s not.
[laughter]
So, in the context of that — and I know people are worried about AI systems giving bad actors the ability to do both, say, a Hugging Face-type attack, but also to create pathogens, and who knows what — in the context of all that, let’s stop and ask: what are the concerns that you have? And then we’ll talk about the steps you think you and others can take. But what are the main concerns?
Owen Tripp
Yeah, so, I want to say this sentence first, before the next 20 sentences follow: I am an AI optimist, in general. I see these technologies as having transformatively positive properties for humanity, and specifically within the healthcare domain — I think the chances and likelihood of tremendous good outweigh the opposite. And yet the risks are asymmetric in their potential damage, and that’s why I think all of us who are building these things have to be incredibly responsible about trying to examine, as best we can, and safeguard against current and future questions as we design these things. I’d say I do worry, when I’m in conversations with peer companies today, and even more so with the buyers of these solutions, about how few questions people are actually asking — if they understood the risks, and I’m going to get to a few of them, they’d probably think differently.
So, you ask what I think the problems are that need to be solved — in no particular order — I’d say the most pervasive one, based on how these things have broadly been designed — and this one is not true at Included Health — is that people, usually for reasons of speed or cost or both, have strapped themselves onto one of the foundation models without negotiating who owns the data and how it will actually be managed. Because the least expensive way to obtain a foundation model to power whatever you’re working on is to allow that foundation model to train on everything you’re telling it. That’s a huge problem for healthcare — it probably violates federal law, or it will.
Matthew Holt
So let’s just be clear — you’re saying, I’m a healthcare company of some flavor, I want to use AI, and whatever I’m doing, I’ll happily feed my data and my patients’ data and my processes into the foundation model, and then, basically, the foundation model has it.
Owen Tripp
You got it — it trains on it. And, by the way, it’s training on everybody else’s data as well.
Matthew Holt
That’s right.
Owen Tripp
This is bad, because when you put personally identifiable human information into these training models, those models are absolutely open, and will use that to train on any number of other things and characteristics it would like to know about and associate with those human beings. And even if you’re doing that on an anonymized basis, you’re running the risk of training that data set in a way that can be undone — there are already public examples of this happening. People have deanonymized data — or, when you’re uploading unnamed, but one-to-one, human-based data, it’s easy to lock that into that human’s profile forever.
Because once it’s inside the training data set, it’s very hard, if not perfectly impossible, to pull it back out. I’ll give you the most obvious example — I tell people I’m a lot of fun at cocktail parties, but this is the one thing I tell people never to do: people are uploading their personal genome sequencing data directly into these consumer-grade AIs. You can’t change your DNA, at least not today — so what you’re doing, even if there’s no name on the report, is telling that model everything about yourself that is uniquely you. That’s a problem for how that AI could, in the future, price you for life insurance, or your likelihood of being a good mate or parent, or whatever. So I worry about that, because there’s been this one-way flow, because it’s been the economically easiest path to put data directly into these frontier models. Now, at Included — and this is the model I’m about to publish an open letter on, because we want to make our approach publicly visible, hopefully as a template for others to use, but also to comment on and help us improve too, since this needs to be a multi-company approach — we’ve built it so that not only do we anonymize data, none of it can be retained where we use external models. All of Dot’s prompt engineering stays on our side of the house. All of our client data exists in a private cloud, controllable exclusively by those clients and the members who use them. These are really important controls for how data can flow. So the first one, broadly, is the topic of privacy — and where you’ll see us go on the whole privacy-by-design journey is that we believe the member — this isn’t yet live in the product — but we believe the member should be able to control all of that data down to the individual level, meaning, ultimately, we’d like the member to be able to control what persists in their personal record. I think that’s going to be a really big topic.
Matthew Holt
Before you leave that one, I think, from an optics standpoint, and probably from a user-control standpoint, what you’ve just laid out is the most efficient and clean way of doing it — but there are a couple of things going on. One is, if the answer is, ‘well, I’m getting this now’ — when I ask, say, how long Claude can look at my UCSF record, and is it a day, a week, a year — the most it gives me is a year — I’m not sure I’m that worried about what Claude is reading from my UCSF record. It’s going to know a lot about me, for sure, but that’s pretty helpful — it’s giving me back very useful stuff, as we discussed earlier — and I’m not absolutely certain I’d want to switch that off. So if you—
Owen Tripp
Well, I don’t think you want to — I get that example, and, if we could, I’m going to abstract away from Anthropic and Claude specifically, because I don’t actually know everything they do —
Matthew Holt
I did pick the model you’re not in a relationship with, by the way.
Owen Tripp
Well, no, no, we work with them too, on certain stuff — but I want to say the thing I’m pointing to is probably not the use case you have in mind. Now, if, in coupling your electronic medical record with Claude, that had express uses of that data — not only could it not retain data private to you, it couldn’t even train and abstract certain concepts from your medical record — which I doubt they gave you. My guess is they are doing that, because that’s a big part of—
Matthew Holt
I’m sure they are, and I’m actually not sure that’s a bad thing
Owen Tripp
Well, listen, I think you can have your cake and eat it too, is my point. You can have all of that insight, surveillance, and the ability to take next steps on your health — which is presumably what you cared about when you coupled with it — combined with the ability to retain sovereignty over your own data. I started a reputation and privacy company before this, and we spent a lot of time on this: all of us, as humans, are willing to give up privacy in exchange for certain things — we’re willing to have data used to advertise to us in order to get free services, that was the starting concept behind Google as a search engine. But this is really different, because your health data is immutable — it’s unchangeable, and it can be used to do a lot of things in the future. Frankly, it could be used to do a lot of things today that you might not like, or be fully aware of. And it’s not clear that law and technology security have caught up with that use case. So my cake-and-eat-it-too moment is: why couldn’t you glean all of that information, have the benefit of that self-understanding and discovery and access, and have the ability to say, ‘okay, great, when I leave this session, or terminate my membership with this company, I want that data out of here.’ I think we should build that — I think that’s quite exciting.
Matthew Holt
I’m just wondering if everybody did that, would the model not — would it become stupider, because it lost that data and didn’t know things anymore? Because, remember, what we were saying 10 minutes ago, is that right now, a lot of stuff goes on in the world of healthcare that’s germane to the patient experience and the clinician experience, but doesn’t get captured anywhere, and someone’s got to put that into the flow to make the medical AI better — like Dot, better than the generic ones. So I don’t know enough to understand whether it needs to know what my imaging studies were from 2014 to be more intelligent about this.
Owen Tripp
I suspect it needs to. I think there’s a bunch of data it’s already getting that has certain permissions, and it can train off of what the provider system itself is using to ensure privacy, compliance, and safety. And I guess what I’m saying is, all of us who are these next-generation health systems, if you will, have an obligation to think on those terms, and think about the power of when that data is beyond our reach. But listen, we’ve only gotten through one of these — and I want to make sure I call out a couple of others, because there has to be a unified framework. So, another one — we know, because these things train on the open internet, and because they’re highly biased by the data sets they’re training on, that there’s actually bias against populations that are served. I’ll give you a couple of examples that have already been discussed, but you can imagine these getting deeply amplified. We know, for example, when we’re looking at the prevalence of disease and using population-health algorithms to say those who are the highest spenders need the most intensive healthcare services, that there’s a problem — whole populations are underrepresented in those data sets because they’re not spending as much, because they don’t have as much money to spend. That’s a problem. A different example — this is all published, this isn’t me pulling a case study from within our company — is that when you look at automated and semi-automated dermatology scans, they’re largely trained on lighter-skin models, and miss things in darker-skin models. This is all changeable, but you have to be able to look at that and make sure that, when you’re building your agents and your AIs, you’re expressly trying to lean against any bias that comes into it.
Of course, the scariest version of this, because it trains on the open internet, is that it will pull in unproven and questionable medical approaches, but deliver them in a plausible narrative that makes you feel good as the patient reading it — like, ‘oh, I should inject myself with ivermectin, because it’s going to work,’ even though it doesn’t, and you could be led to believe, by the AI, that it’s true. So that’s a huge one. And then the last thing I’ll say, broadly, on the topic of safety — and I’d point people who have time to read it to the piece we published with the New England Journal of Medicine Catalyst, with some other researchers — is that our approach to safety is human-in-the-loop: over a certain level of severity, in any interaction with Dot, there’s a human reading it.
That’s more expensive for us, but we think it’s critically important — that’s also part of how we train Dot. We built a clever way to do switching on the back end, so people available on shift can actually take a look at these as they’re coming through, making sure we’re covering 100% of it. But that’s how we’ve trained our human-in-the-loop systems, because there’s a real risk of hallucination, a real risk of injury. Again, if people want to read it, there was a pretty major finding — I don’t know if you saw this — from the auditor general in Ontario, Canada, that a shockingly high percentage of the cases they reviewed, where AI was taking ambient scribing data from physicians and building care plans, had medications swapped, incorrect medication amounts, services recommended that weren’t actually recommended, hallucinated recommendations for blood tests when people didn’t need them. These are real concerns. The problem is that the AI’s conversational style is so seductive that we can easily believe these things are happening correctly, and so we, as companies and stakeholders in this early movement, really have to commit to addressing it.
Matthew Holt
Yeah, I think the question of how to deal with hallucinations — and hallucinations being generated not only from text, but also from audio, from voice, and in ambient scribing — continues to be something people are struggling with. I’d say this is another case where, look, we had tons of mistakes before — it’s not like the baseline was great.
Owen Tripp
No, for sure — tons. The baseline was not great.
Matthew Holt
But so I’m optimistic about that. Having said that, we’re also getting a bunch of case law starting up — like, can you record someone without their permission, who’s entitled to the document. There was a case today in Washington state where someone was trying to get an ambient scribe transcript out, but the law said it belonged to the provider — the patient couldn’t access it. Is a patient allowed to record their own visit? What happens if the patient’s recording and the provider’s recording don’t agree? You already have —I’ve experienced this with my own medical records — stuff in the record that’s wrong, and no easy way to correct it. This has been a problem; I think it gets amplified now, because we’re recording directly into the AI a lot of things that just didn’t get captured at all before.
Owen Tripp
Yep.
Matthew Holt
So I’m with you on that — clearly we know there are mistakes, and we know AIs don’t agree with each other. I had a ridiculous case today where I was trying to figure out why the PIN code on my computer had changed, and whether the AI had hacked it, or my daughter had deliberately hacked it, or — probably that one. Anyway, ignoring the answer — I did it on Gemini, I did it on Claude, and they completely disagreed with each other about what was going on. I don’t know the back end of either,
Owen Tripp
But they were both equally declarative and authoritative! So I think the issue I’m pointing to isn’t so much that issues, problems, or mistakes didn’t exist before — obviously you and I know very well that they did. What I worry about most is the complacency that will settle over people, just assuming that what they’re getting is superior to what a physician was giving them historically, because I think we all have — perhaps insufficient, but some amount of — awareness and self-advocacy when we go into the health system, to make sure things are working out the right way. When we’re told certain things by our friendly AI, we have to be careful. I had an experience trying to dose Tylenol appropriately for a kid who can’t yet take the pill form, and the AI massively hallucinated on the math. It would have been a stupid dose — he’d have been drinking two bottles of it — obviously I wasn’t going to do that, but had I, it probably would have killed him. That’s an issue we can laugh about, but there’ll also be a sense of security, peace, ease of use, and efficiency, when you don’t have that kind of medical-grade AI guardrail saying, ‘okay, this is dosage, we need a human to review this, and we can get somebody on the line within seconds.’
Matthew Holt
So the drug-overdose one is a super interesting one — don’t forget, this is the centerpiece of Bob Wachter’s earlier book — the drug overdose at UCSF, which came out of Epic and a combination of the pharmacy, and it was a teenager, and there was a calculation that didn’t get caught, very similar to what you just described. In that case they did give the equivalent of two bottles, and it killed the patient, and everybody was at fault — from the person who designed the system, to the pharmacist, the nurse, and arguably the patient himself should have known better — but in the end, it went all the way to that massive overdose.
My sense is that we’re going to get more and more comfortable with AI telling us the answer. The AI is probably going to hallucinate less and less, but it’s still going to depend on guardrails.
Owen Tripp
Depends on how it’s trained.
Matthew Holt
Well, I guess there’s two things — there’s training, and there’s final oversight, and then, as you mentioned, the guardrails.
Owen Tripp
Exactly.
Matthew Holt
How do you think that will play out for — including — how do you think it’s going to play out in an actual case, like the Tylenol dosing example, or whatever drug it is?
Owen Tripp
Well, on this point — like most topics in business, and let’s not forget these are businesses we’re talking about — they’re going to operate against their ultimate incentive set. We know, for example, thinking pre-transformer, about the original search and social media sites we gathered information from — those companies performed better on revenue the more they reinforced what we already wanted to believe and do. That was true because advertising worked, and we’d spend more time on the websites. I’m afraid the same will be true with these consumer-grade models, because people will be looking for confirmation of things they already believe, and that’s just human nature.
So, without picking on any of them individually, I just don’t think it will be in their natural incentive set to say, ‘no, actually, we’re going to inject expertise and evidence and truth’ — setting aside the whole political part of that conversation, which has been nasty the last few years around the role of science — I just don’t think it’s in their business incentive set to pause a conversation like that. However, companies like ours at Included Health, and I imagine many of our fellow travelers on this road, are going to say our incentive set is to deliver high-quality care — we’re measured against those outcomes, we’re also evaluated on whether members like and trust us. It’s the conjoining of those two things — we have to actually be able to deliver the outcomes too. So health systems, both current and future-state, are going to be evaluated more on safety and quality.
Matthew Holt
I’d say, to wrap this point, there seems to be a bit of back-and-forth about how you deal with this — your neighbor and fellow CEO, Jeff Tangney at Doximity, has put together an advisory board of the wise and so on — and I’m a little confused and concerned about how you put guardrails in when you’ve got — you were talking about human-in-the-loop, for you at Included — but there’s only so much people can read, and sure, obviously the AI can spit things out at scale, you can put guardrails in, and hope, but then every session is individual. I’ve had this conversation before too — back in the day, Mayo Clinic would write a learned tome, for both the clinical view and the consumer view, about diabetes, whatever it was, by their committee, and that was it — it got stamped, and now the AI is spitting a new one out, more and more individually, every time. So, can this work? Can we eliminate the hallucinations, the dangerous stuff, the telling-you-what-you-want-to-hear from AI? Can we put those guardrails in, the way you’re talking about?
Owen Tripp
I know we can absolutely put safety guardrails in, because we’ve already done it — we’ve already flagged and bounced a whole bunch of safety incidents. Now, our approach so far is quite human-intensive, and that’s true, and we’re okay with that — that’s our legacy, and we’re proud of the track record we have on safety. Some of the other topics I raised, I think, are going to take even more work. But what’s so critical — my call to action here — is that those of us who are building these things, already operating at the edge of the envelope, need to get together to work on the principles of what design looks like, because the existing law and regulation don’t really contemplate a lot of these use cases, and what we will have to do by law, and what we should do, are different — and law ultimately catches up with ethics, but not as fast as any of us would like. So this is my plea — that we’re going to start to organize a bunch of like-minded people to agree on principles, and hopefully show a technical blueprint, so people don’t have to invent these things on their own, on how you actually do it.
Matthew Holt
No, that sounds very sensible. Obviously the current administration has a very different view than where the previous administration was going, and who knows what it’ll be like in a few weeks, after the midterms, or a few years after, if there’s a change in administration — but that’s a long time to wait for that to catch up. Right now, we have this early intention to cooperate among the big frontier models — I don’t think we call it that — how would you say what you’re talking about your call to action, is reverberating among your fellow leading AI companies?
Owen Tripp
So, the people I talk to — and I’m lucky enough to be here in San Francisco, close to a bunch of the companies doing this work — I think they’re not only open to it, they also sense it, and some of them are at different levels of their journey on the ability to deliver against it, but there’s an openness to the conversation and a collaboration that’s really exciting. We want to put together a group with a clearer point of view that can help educate and articulate that point of view to other people, and we’re just starting the work to put that together. But this is the thing I wanted to bring to you today, because I think, in this week, where we’re having this massive national and global questioning around the safety of AI, we have to look directly at the lives of the people around us that we’re charged to take care of, and make sure we’re doing all we can, in our specific corner of the neighborhood, to think through the risks of these things. And again, the risks pale in comparison to the benefits — we’ve already seen it, it’s going to be huge — and I’m personally a user, I’d recommend everybody use it — but I think doing this the right way is going to be meaningful to the future of our country.
Matthew Holt
Fantastic. Well, I look forward to hearing, relatively soon I hope, what’s going to happen out of the groups you’re initiating. I think there’s going to be more national conversation about this — clearly we need to be paying very close attention, both in healthcare and outside of it. We’ve heard a lot, obviously, over the last few months about whether — I don’t know, whether we can slow down. I do worry — and I’m not the biggest fan of Scott Bessent and that crowd — but when you think about the Chinese — this is also not just a healthcare or consumer issue, it’s a national security issue.
Clearly we’ve all seen the movies about the bad things that can come out of this — but there’s a lot of great stuff that can come out of AI too, some really optimistic books about abundance, and clearly a lot that can be done in healthcare, not only in drug discovery but in fixing the way healthcare is delivered, which you’re working very hard on. So we have to get it right, and it’s obviously something we all have to focus on. I look forward to hearing back from you, Owen, soon, about how you’re doing that.
Owen Tripp
We’ll do it.
Matthew Holt
I’ve been talking with Owen Tripp, CEO of Included Health, and we’ve gone deep into the world of AI — we’ll talk more about this soon.
