---
title: "How Meta Preserves a Best-in-Class Engineering Culture in a Regulated Environment"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/ukv3wvjs4z"
content: auto-caption transcript, proper-noun corrected
---

# How Meta Preserves a Best-in-Class Engineering Culture in a Regulated Environment

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/ukv3wvjs4z)

---

it's so wonderful to have you here today. My name is Erez Kaminski. I'm the CEO and founder of Ketryx, and I'm very excited to be joined today by Lucas Fernandez from Meta Reality Labs, to talk about how Meta preserves its best in class engineering culture while building to medical device standards. We have been fortunate to partner with Meta from the start of the regulated product development journey. When Lucas found us through a webinar like this, learned a lot about our product and the vision, and was one of our early customers in understanding what is a a way to develop regulated software, regulated AI, regulated product that still preserves all the necessary needs of the highest safety and functional safety validation requirements and quality requirements while combining that with best in class engineering tools and practices.

So thank you so much, Lucas, for your partnership and your support of us. Before we get into how Lucas and his team, are making this possible, I wanna introduce Lucas. Hi, Lucas. Hello. Hey.

Good morning. I think it's the afternoon in some parts of the world. It depends on the geolocation. Exactly. So Lucas is the director of medical devices at Metareality Labs, where he leads a pretty broad cross functional team responsible for driving medical device development and compliance.

He's worked closely with engineering, product, clinical, quality, regulatory, to create an environment where engineers can move fast and do their best work while delivering regulated hardware and software and AI, built on biosensing, and kind of a a connected ecosystem. Previously, he's held leadership roles at NanoStim, Sumo Logic, Verily Life Science, a Google company. At Verily, he was the employee number two in their quality assurance department and joined when there are just a few hundreds under the org name of Google Life Science. He's also part of Boston Scientific and too many industry leaders to really mention on this call. Lucas has implemented zero to one quality management system that helped launch the first leadless cardiac pacemaker in NanoStim.

He was part of Alphabet's dive into regulated digital devices as part of Verily, and now he's doing it again, at Meta Wearables with a lot of, I think, lessons learned from that process. He's a named inventor on three medical device patents, and he holds a BS in mechanical engineering and an MS in management science and engineering from Stanford. Welcome, Lucas. Couldn't really share more. Just so excited to have you here.

And I think our idea is to have one of the conversations we always have together when we meet and sit and talk about life science and and AI and development. So welcome. Thanks. I would say, actually, one thing maybe I would correct the way that I turned, but you mentioned the other I I I stumbled onto you guys through one of your webinars and and was actually there. But I think even much more than even what the original product itself was, which was more on the philosophy that as you and I talked around how we think about quality management systems, how we really think about building, compliance that really builds on the quality of the product first.

I think that was, like Yeah. Two more seconds. Can't hear Lucas. Lucas, can you just see if on the bottom if there's a a mic that you need to raise up? Because I can hear you well.

I mean, I see everything on my screen says Okay. I think you're good. Okay. Some people are saying that I can hear you. So I guess it's a.

Okay. Okay. The, yeah. I was saying yeah. I think I say obviously, the Citrix platform, one you guys are trying to do with the the system is very exciting, but I think much more for me was really that the the philosophical alignment around how to really think about and approach, quality management and, regulated product development is which I think I got, I was very happy.

I stumbled upon you guys on LinkedIn, actually, I think, the first time. And then from there, obviously, the relationship we've built now for a couple of years. Yeah. It's, it's been very exciting. And and just seeing the work your team is doing, it's pretty mind blowing.

How do you how do you kind of, start this up and start it in a way that is safety first but also extremely innovative and not giving up of any of those parts. So let me let me start with the opening question I wanna ask, which is, your background comes from a variety of places, but also more traditional life science organization. Right? Massive scale kind of companies that do this, in the highest risk levels. How did it feel stepping into Meta's consumer tech culture?

It's you know, I've worked in biopharmaceuticals. It's very different to go into a consumer tech company. Yeah. Exactly. And I think I'd say for me, you know, you mentioned a little bit in the intro, my first kind of trial by fire to this was when I when I joined Google life sciences, which then became Barely.

And I'd say there, I suffered a lot, like, culture shock of going across it. And and, thankfully, sort of, as I survived to the other side. And so at Meta, I'd say, a little bit more I knew what I was walking into. But, really, I I think the the fundamental difference that I see and that I've you know, I have nothing else you could say. I I I'm trying to evangelize into more of the medical device world is really, like, you know, the product first focus.

Right? It's alright. What is the product? How good is the product? Is is it the right product for the customer, and how do we improve it and ship it as fast as possible?

And so then it really became a a challenge for me of trying to figure out how do I, you know, really lean into those attributes of consumer tech technology companies as we build more regulated products, but also recognize that, yeah. Hey. First of all, you know, safety and effectiveness mean something, and it means something very important because of the the types of products that we build in the medical device industry. And we have regulations. Right?

And so the idea is not to just, ignore or run over or run, around regulations, but say, no. Okay. How can we really also the way I look at it, and I just told this to my team this last week, is, yeah, we wanna be able to do great work internally, but we also need to be able to, like, demonstrate it to the this outside, you know, parties or stakeholders. They need to sit like, need to believe us. And the way they believe us is through, at the end of the day, the documents and the rules that we generate and, obviously, the product performance in the field.

And so we need to basically marry those two, in a way that, ideally, we're meeting people where they're at in terms of internally of how we do work, but then can, you know, translate that enough, into what is a regular expecting to see or be able to essentially, yeah, verify. Right? And that's where a lot of the documentation and the filings are around is they, oh, yeah. Hey. They say they do all this great work.

I can see with this evidence. I wasn't there in the room when they were doing it. I can see with this evidence that, yeah, they really know what they're doing. Yeah. And I think it's challenge for you because the you're not, like, trying to do it at a startup.

Right? You're not trying to do it at such massive scale that, like, this is you know, you are impacting a lot of people the moment anything turns on. I think, that was one of my big lessons doing it at at mega kind of size companies is, you know, whatever we're doing is gonna go to a million plus people. Let's make sure we're doing it in the right way. Exactly.

Yeah. Yeah. I was gonna say that. I think to me, that one's sorry. From a scale perspective, it's it's both scale, obviously, in terms of the impact.

And so, obviously, I you know, for for Meta, if you're not talking about millions of users, it doesn't really move the needle. It's not worth our time, so to speak. But then also internally, right, I mean, it's Meta is a very large company, and the way meta infrastructure is built internally and part of what allows it to be very successfully responsive is that it's very integrated. Right? And so, obviously, then anything that you wanna change or adapt for a regular medical device can't, like, down meta dot com right, facebook dot com or Instagram dot com or whatever.

And so how to how to really, work within that space provides also some really interesting limit well, not limitations, ways to figure out how to embrace and create more almost, I'll say, API like contracts across different parts of the organization such that we can really, you know, do do the things that we need to do but not all of a sudden disrupt, you know, what is, at the end of day, the moneymaker, which is all these other things over here at the moment. Yeah. And I think I love that because it's all these things are so interwoven. Right? Like, systems engineering, the way they think of systems of subsystems.

Right? It's just another name for microservice architecture. An API is another name for an interface specification. Right? Like and we're just kind of renaming things, and I think that causes a lot of confusion.

Right? Like Yeah. Exactly. No. I think that's one of things where, you know, actually, I I commented on on a post you did recently, like, kind of advice to new people who are entering this space, and I said learn, like, the medical device lingo.

But, also, yeah, conversely, there's many things that is the exact same thing, you know, fundamentally. We just call it a and they call it b. And so how to really, first of all, distill that down to what is the fundamental understanding or the fundamental requirements of that thing, whatever it be, and then how do we utilize that to show that, yeah, that we have the right levels of controls and and risk systems to to say, no. Hey. Yeah.

That's fine that we rely on this whole other thing over here that's not part of the QMS or whatever. We have this contract, and we have this kind of API agreement across for how we exchange critical information. Yeah. And and it's just things are for formal and well defined. Like, you know, someone tells you it's a critical thing or it's a critical risk, you're saying, like, you know, critical.

It means something very specific. Are you sure it's actually critical? Critical means that it can Right. Know, impact the life of a certain amount of people. Like, that's the definition of a critical safety event, you know, or critical high criticality.

It's interesting. So I know that most people know Meta as a as a consumer company. Right? I've been Facebook user most of my life probably by now. It has a very fast paced, best in class engineering team, very mature systems, you know, testing documentation, kind of everything needed to build at scale.

And I'm sure that whatever you guys can, can find, y'all built internally to make sure you have the best in class tools. And Reality Labs is creating something a little bit different. So for folks who don't know Reality Lab yet, can you share a little bit more about what your team is building and what that means, you need that's different from a tip typical consumer kind of b to c company? Yeah. So, yeah, so I I'm sure most of you guys know.

Right? So Instagram, WhatsApp, Facebook are are, you know, kind of the what we call the family of apps companies. That's what our famous are. Reality Labs was started basically to be a hardware company or a hardware division and really was about trying to sort of really started to think about what are the next generation computing platforms after cell phones and computers. Right?

We all have laptops. We all have cell phones. We've lived with them forever, and it feels like at this point. And at some point, something's gotta come next because, you know, we already went from desktop computers to laptops. We went from home phones to to cell phones, and so really Reality Labs was, first and foremost, you could say that is, like, the the core idea.

And then, it did a, I'd say, a second, I'll say, adjustment to that mission when AI really blew up in the sense of now all of a sudden, if you talk about having AI devices, and so how do you really take AI along with you throughout your day? And I realized I probably should've worn my Ray Ban Meta smart glasses rather than my regular glasses. Missed opportunity for marketing there. But, really, that is what we're trying to build that reality of. Reality of is in Macy are these hardware platforms, and, obviously, then the software and the experiences on top of that, they really now, bring AI with you throughout your day and throughout your life.

And, obviously, in health, it's it's that's one of the main reasons why people buy wearables today. Right? If you own an Apple Watch, if it wasn't your primary reason, it was your secondary reason why you bought that Apple Watch, it was either step tracking, sleep tracking, blood pressure, you know, you name it, those kind of things. And so anytime now you have a a wearable computer effectively, you can connect that with biosensors and collect all sorts of different data and to help people then, lead healthier lives. And so that's where, you know, we see the big opportunities and and how my team fits in there.

And even actually at the most basic level, a lot of people don't realize it, but just glasses are a class one medical device. And so even just the mere fact of wanting to put it on a glasses form factor, we are, you know, registered, and and we have to follow FDA QA QSR requirements and, you know, ISO international requirements for them for them as well. Yeah. Different, right, than how you would think of just, like, making a pure kind of b to c app. And, you know, in our last webinar, we're talking to Flowhelp for, like, exactly the same challenges.

Right? They're also in in, like, the hundreds of millions of of downloads that's gets a little bit different when you now need to make it rigorous. So leading into that, like, I know that in the Valley, a lot of my friends are developers in that kind of West Coast style development that I think Meta is kind of proliferated and and in many ways started. They're used to moving fast and breaking things. And that typical agile model, it doesn't work in the same way when, validation and robust life cycle management is required.

Right? We could do agile, but we do need to do it in a way that's, that allows us to have the rigor we need. And that rigor is there to keep patients and consumers safe. So I wanted to ask, what were your engineers most worried about when we started preparing for building medical grade software and medical grade hardware? Yeah.

The I'll say that right. The number one question every engineer is gonna ask you is, like, okay. What is what extra work are you gonna make me do? Right? Bottom line.

Right? Because and and and if you think about it from a a software developer as probably being one of the most expensive assets you have at your company, and and the way that you get the most value out of them is by just having them creating code for the product. Anytime you take away from them from that activity to, you know, do something else. Right? And documentation is, you know, nothing else you can say, probably the one of the most negative things you can ask them to do in the sense of they don't feel as nonvalue added.

Right? I mean, self documented code. Right? All these practices have really have come along in long ways. Or even just code readability is is really seen like, no.

Wait. I you want me to write a document? Like, why do you need me to write a document or, things along those lines? So I'd say that's usually where the the biggest concerns come along. And then second, yeah, right, I think this thing around, how do you really think about from a verification validation, code coverage, testability?

Do you actually you know, rather than say, oh, well, we'll we'll, you know, we'll we'll send, like, we'll do a canary release, and we'll see in the canary release what happens, or or we'll kinda do an alpha out there and catch bugs that way. It's like, no. No. That's not that. You know, we we wanna do that too in case we miss something, but we need to do much more ahead of time, before we release it.

And so, obviously, you know, branching strategies, how you control, you know, even be able to, I say, fix it forward, but fix it forward internally. And then at some point, say, okay. Now we reached a level of maturity where we can actually ship it out, and there, the fixes are gonna be, you know, either less critical or or more minor. Those are the kind of things where I think, having the right tools and infrastructures that can sort of help. The way I, I've, know, explained it, I've I've whereas I've mentioned to you before, almost like you need, like, almost like a a governor or a dashboard where you just see all these things happening and you create and insert some rules into the process for those critical things that you need without having to say, oh, no.

Yeah. Any engineer out there can push code to production. Right? I don't need to have, like, a release captain or whatever you wanna name you that has to now be the critical path bottleneck because that's the only way we can do these things. No.

No. We can build in an integrated tools. We can actually have, you know, rules either through just very general algorithm rules or through AI agents that can say, nope. This is good. Keep going.

Or no. He'll and then I'll pump the brakes on that one. This needs now some additional review or escalation. Yeah. And to me, that's this idea of a a a rule engine or a a symbolic engine, symbolic AI then enforces it or, you know, kind of we call it a lot enforcement, which is basically a way to take the verifiable rules of the quality management system, that are kind of, you know, true or false Boolean, much of them, right, in nature, put them in a computer, connect that into your audio tools, your CICD pipeline to make sure those questions are answered.

And then we want our people to be able to spend their time on the things that are not yes or no and, obviously, just kind of verifiable. We want we want them to spend time on on more complex questions like risk management and critical thinking and change impact analysis, not, you know, did I do all the tests for that change? That's, like, a very different question than what should I test for the change. Yeah. Although, yeah.

I mean, that and that's the part, though, I would say as I look at right now and I look forward, I would argue, like, we should already been doing all that already. Like, that what everything you just named off are, like, the easy stuff that, honestly, we should have done ten years ago. Like, that's not even, like wouldn't have been that hard to code up. Now going forward, it's like, okay. Now can you actually start using more, like, AI agents to even do that more of that that nuanced work?

And, really, then you you really just have humans where it's, like, either very highly highly complicated, very highly critical, I think along those lines. But, you know, I now use, you know, the the Ketryx AI agent in Ketryx. So, like, literally this week, it's like, hey. Just write me the justification as to why a type test for software, and I have to do a sample size of twenty nine, is valid and makes sense. Right?

And the things spit out in literally thirty seconds for me. Like, the here's the here's why and here's these references, and you can, like, pull some stuff off the web of different, I you know, publications from the FDA and stuff like that. I like, yep. There you go. And I, like, copy paste, dropped into my document, moving on.

Right? And so those are kinda things that are not yes, no true, false rules, but can even now be scaled and automated. Right? Like, I don't need not to have me have memorized that justification or I've known it, or, you have it right ready handy. I was like, okay.

I know it. Mean, I still need to know that, yeah, hey. That is a thing. Right? I need to know that type testing and and what's the difference between type testing and variable testing, all these different things is a thing.

But I can now, instead of me spending even an hour trying to write up the justification, think about the online signs, I can do it in five minutes. So now I've I've become way more efficient in how I use it how I use my time on the really critical strategic things that I need to make sure either internally or as I'm preparing, you know, submissions to to regulators are are aware, like, you can say the maybe you can say the the I don't if the gray areas, but, right, if if I were to look at a a risk map of a product, I'm saying, okay. What are the riskiest areas of that product? And that's where I wanna spend most of my time is on those really higher risk areas because it's a new technological question. It's a new it's efficacy question.

It's a new safety question. I wanna spend a hundred or ninety nine percent of my time on that stuff. One percent also which is table stakes. Yes. It's important.

It needs to be done. Some of it is obviously still gonna still be very complex, but that I can I know in a five minute conversation, can tell you what those things are, but it would take me five weeks to get it done in a more traditional way? And now I can actually have these more agents and helpers to, do that much faster for me. Yeah. And and it just comes back to this, like, traceability is something that's very deterministic.

Like, we should be able to generate those matrices ahead of time, use them, and not spend our time generating the matrices. Instead of that, we should generate our spend our time thinking, does it make sense? Are our risk correct? Like, hey, agent. Can you go, read mod for a second about products that are like my products?

Do I cover all the latest and greatest risks? You know, I think that's one of the biggest Yeah. Hey. Read me read me the cybersecurity guidance and tell me, have I hit every point on the cybersecurity guidance in this document right here? It's like, no.

You missed this one. Right? Me having to go read the whole thing, put them side by side on my computer. I'm like, oh, yeah. Oh, yeah.

No. Maybe. Right? So Yeah. And and the evidence documents need to come out of the work.

Like, I think that's, how it used to be, how people intend it to be. You know, I we me and you often talk about how traceability is actually an r and d thing. It used to be like you know? Yeah. That's how you built a product, and now it's kind of we've shifted it right on the v instead of left on the v.

And it's actually just because it's become so complex. It's just so hard to do that. And so moving into those kind of questions about the problem is, like, as you built your submission ready process, where did agility break down for the team, and what from your past experience did you lean on to fix it? What was the main focus of what you wanted to do better this time? And I know you have a lot of autonomy with the team.

And so how how are you thinking about those problems that you've seen before? Yeah. So I said, first of all, like, the kinda say the I don't if you call it the joke or the the tagline we have, right now at Meta on my team is, like, this is, my revenge tour when it comes to quality management systems. You know, getting it obviously, walking into a company that had nothing. So and really because not a medical device company, like, nobody above me knows or understands what I do, so, obviously, nobody can stop me, so to speak.

And so that's that's obviously been great and and very exciting to give me a lot of, kind of freedom to to operate or try new things. And, and, yeah, right, the I think the the combination of what you do see, I'd say, I where you see agility breakdown generally, in, you know, say, in very modern software development is actually yeah. Because systems are so big or so complex and things happen so fast, stuff just gets missed. Right? Just because, again, we we just can't keep up.

And so and, yeah, usually, happens is you fix it forward or you catch bugs later, you just oh, and you iterate and ship very quickly, which I would still fundamentally argue that it it actually do build a much higher quality product of the long term that way. Alright? And so to, you know, to be able to quickly catch things, fix things, deploy them, does get you to a much better product than the, oh, okay. We're gonna spend, you know, five weeks just writing requirements, and we're gonna make sure every requirement is rare, and then we're gonna move into design and putting all design. Right?

That does not build fundamentally does not build a better product. Obviously, then in the case of of medical devices, yeah, there's still logs. Mean, no. Some of these, we do have to get right up front. Right?

The the the critical pieces of it, we have to get right up front. And so that that's kinda balancing those two things. But then then, yeah, right, the I'd say even as we, you know, we engage kind of in in in traceability exercises, through, you know, either using using Ketryx as a tool or some of the other ways, just with our own internal systems, you when you just as soon as you have somebody paying attention to that, you see, oh, wait a second. What about this? And we talk about this, but then it didn't go anywhere or anything along those lines.

And so that's where I see really still even, I'll call it maybe, in more of just consumer Internet companies, there's a a higher error rate accept accept that's acceptable, and they just take a take it as a cost, so to speak. Right? But, honestly, I I I would argue that even in those in in those companies, you actually can bring that down in a meaningful way without, you know, massive impacts of productivity. Right? Because that's always they always go, oh, it's gonna slow us down.

It's gonna slow us down. And so so that's where I I see, the nice thing I kind of where I'm gonna get to work right now is kinda I feel like it's a it's a middle sweet spot. I get to sort of, like, you know, if else test out some of these claims of mine, of how to do things that really, at the day, rely on still, you know, incredibly smart and probably capable of people, but that also then I now have systems where myself, you know, can very quickly take a look and see, okay. Wait a second. What's missing?

What's here? What's not here? Right. I can Like, is it ready to ship or not? Yeah.

Yeah. Is it ready to ship or not? And and in in a sense where I'd say I see a big part of my role as as we get ready to to ship or launch a product. Like, I'm effectively auditing. Right?

I I'm never gonna look at the entire thing on the all those lines, but I'm gonna I'm gonna spot check some things or or or we all check things at a higher level of completeness. And and then she said, wait a second. What if I hear what I hear? Alright? And and so then there, you know, I still either sometimes I catch stuff and often but oftentimes, I say, I get that whole comprehensive view, and that is because, yeah, you have a system.

You know, you have a you have a way of integrating all this information. Otherwise, it's just you rely on people from all over the place. Right? And the and mean, like, all these different functions, and you kinda have to trust a lot more because you don't have a a way to really say, nope. Yep.

That can't happen unless this happened, and this can't happen if unless that happened, or even just have be able to do much more comprehensive reviews of something in a much more automated fashion nowadays where, you know, a fifty page report or a fifty page document, rather than me having to spend a day reading it or half a day reading it, I can have an AI agent read it for me, analyze it, and then highlight for me, you know, that he's, hey. Go look at these areas. And, yeah, is that gonna be perfect? Nope. Right?

Things obviously still make mistakes. But I would argue my old way made actually probably way more mistakes. I just maybe didn't know them half the time. Yeah. I think that that's part of the issue, right, that we think that if you had hundreds and thousands of of, you know, quality professionals, for example, checking things for such a complex systems, that I I don't know if that helps because it's so hard to understand the context.

Like, kind of the complexity of systems has way outpaced the ability of the human mind to comprehend them. Like, if you do want to try to understand BMW factory is, like, you just can't grasp it. Can't grok it. No. I mean, I read I read a paper, this last week from some researchers at Stanford did, basically, it's funny because, like, you'd say the the clickbait headline was, you know, AI agents make a significant number of errors in, you know, health care practice when looking at, like, you know, EHR notes.

But then as you read it, like, oh, actually, by the way, compared to the average doctor, actually, they still made less. Because the average doctors make a ton of significant errors all the time, but we've accepted that as, like because they're humans. Right? And and similar, like, if you look at, you know, Waymo has published stuff about self driving cars and you know? Yes.

Do they get an accent from that one on top? Yes. They also get an accent ten times less often than humans do, but we we, you know, we we hold almost, like I said, the computer to a higher bar and that which we should we should. Right? But that's always a thing where I I always I I'll say, in many ways, I am a perfectionist, but I I also know perfection is impossible.

Right? And so there's always gonna be some level of imperfection that has to happen for us to be able to ship product. Because, you know, that's the only thing I always tell folks is, like, yeah. We can be a hundred percent compliant by shipping nothing. Yeah.

Also not have jobs. Right? Because Yeah. We ship nothing, we will be out of business. So we have to ship something, and it better be pretty darn good, but it's never gonna be perfect.

Yeah. I've never met a perfectionist quality leader. I thought, you know, just casual. I think it's it's part of the problem, and what I I you know, we're trying to design very deeply into our system is this combination of the generative AI, which can make mistakes but always requires human review and approval, so we can't make any mistakes to a controlled artifact. And then the symbolic engine, which is deterministic in nature, and and in that sense, if you configure it correctly, it can't make mistakes because Right.

Like, either all the traceability is done and people have approved it and you can release or you can't. And I think that, to me, I've always had as a way to, like, reduce a lot of pressure, especially for an executive that gets this huge stack of paper to sign. And you now know, like, at least the things that kind of you could control, right, you were controlling. Right? You have this computerized quality assurance system.

And that kind of leads me to my next question, which is where did you see in your past the most friction between your engineering culture? What were you thinking of doing? The legacy ALMPLM systems you had to use, and people use today to meet medical grade expectations. And and how do you think about it in the lens of looking to add AI capabilities into a product, right, which needs to release really fast? Yeah.

Well, I'm not gonna name names. Yeah, but you're right. I mean, I think, right, we and that's why it's it's it's right. I mean, I'll say, hundred percent, it's always much harder to change an existing system than it is to basically start from scratch. The and and there is, you know, so much legacy work has been built in to actually to create either rules or roles that guarantee quality in some way in that existing system.

And and, you know, I'd say, to me, always the the biggest crime of a of a quality assurance professional is to always just then add stuff. Right? It's like, we have it this way. Oh, but we know we gotta do this. You know, we're add something on top.

Right? Add something on I I always joke that my my simple check on that is, okay, how many signatures are required on a document or a record? And if it's, like, more than three, I know that this has been like, oh, yeah. There was a cap applied for this at some point, so I get out of this, and there's cap applied for this at some point, that, this, and that. And then, actually, you know, I will be somewhat vague unintentionally, but I recently we recently did a filing, that was two thousand four hundred and fifty two pages.

Right? And I'm I signed off on it as, like, the responsible person. Right? I'm the PRRC and all those things. And you can't tell me or you can't ever have said you have expected me to rehab read all two thousand four hundred fifty pages and ensured everything was a hundred percent accurate in there before I shipped it off.

And so so those are the kind of things that in leg and I say legacy systems were built because of all these things and and and, and optimized for a way of working, I'll say, right, which was, if nothing else, maybe more more deterministic in nature, more humans in the middle. And and so all those things, you know, get you down. You know, at some point, you you may almost made a choice at a fork in the road, and you have to keep going that one. Right? And you can take it to a logical conclusion.

But now, right, as any person who's developed software, at a certain point, your legacy software code base, you just can't wring it out of it anymore without basically just now having to bolt things on the side, you get this clunky way of doing things. Actually, my there's a commercial I was listening to. Actually, I I wanna say it's ServiceNow, actually. I'm sorry. Apologies to ServiceNow.

But, you know, ServiceNow has, like, a commercial they do. Oh, like, oh, yeah. No. Hey. We are the one application that can do everything for you.

They can do IT. You can do HR. You can do payroll. You can do, you know, stuff like And and I still fundamentally believe, like, no. That can never be the answer.

Right? The answer can never be it's one perfect application. There's no way you can build the one perfect application. It's the know, the human brain, let alone a company, can't ever optimize for all those things. And it's just getting worse.

Right? There's, like, data ops, ML ops, LLM ops, CICD testing, manual testing, fuzzy testing. Like, it's it's just getting more and more, of a feature space to to manage a TPLC, a total product life cycle. Yep. No.

Exactly. And that's part. I I guess any anytime I try to I'll say I mean, just put my brain around everything that's happening under me, like, I just get overwhelmed. And almost like you say, I guess, start getting stressed out. I'm how can I possibly have control on this?

Like, it's way too much. And so to me, that is a work power. You have to build and then think about how do you design systems, to take care of that. And I'd say most legacy systems still at this point have kind of what I'll say a they've they're they're at a linear growth scale linear efficiency scale, I'll put it that way, where, like, okay. Things will just, you know, get a little bit more efficient, bit better linearly.

And, okay. If you need more, you just I I have, like, more people at it and then go that way. And I've told my team right now. I was like, you cannot build you cannot propose to me anything from a design of a system perspective that's that has a kind of linear scale with people. Like, that that cannot be the thing.

Right? From a process, from a system From perspective, from a systems perspective, be it risk management, be it human factors, be it whatever, it's gotta it's gotta have a way to grow. And for us, grow is, like, exponential growth. Right? Because it's okay.

Sure. We launch something, and, yeah, and right away, maybe it's you know, I'll take complaints. Right? Complaints, when we first launch off, okay, we might be in the hundreds, we could very quickly be, like, in the millions. I mean, given, like, our user base and the kind of things, and this so there's no way, like, oh, yeah.

Well, I'm just gonna hire more complaint analysts to go read every single complaint and determine if it's reportable, nonreportable, if it's serious, it's whatever. Like, that that cannot be the solutions anymore. And I feel like most of these old systems, at best, are sort of, like, are tacking on kind of AI agent on the side or anything along those lines, but it's not really, like, natively integrated into it. Almost none of them, I say, like, write back. It's, like, the thing.

Right? I mean, it's like, okay. It gives it almost, like it gives you something not not for you to go copy paste or whatever. Right? It's not like Yeah.

It's not it's not actually working in the system. It's just kinda, like, looking at it and reading it and giving you its opinion. Yeah. And I think that that's part of the challenge. And and, you know, I've spent a lifetime working on post market surveillance with AI, and there's just the potential is, I think it's obvious.

And I think as as someone whose family members were impacted by post market and pharmacovigilance related problems, the the potential for AI to solve that and accelerate the pace of those processes is I don't think we're the the even the beginning of that journey because I don't think most companies even comprehend how much it can accelerate. Like, I think we're saying that you could do a root cause that used to take years, or you could do it in in days. Like, that's the difference, if not hours today. Yeah. I mean, I somewhat, you could say it's probably a bad joke to make, but I, you know, I do make the joke to my team sometimes.

Like, okay. I just did that in five minutes. If I did that in five minutes, why do I need you, right, in the sense of, like, because I asked them to do something, and, like, I I was like, oh, wait. I was able to knock it out here in five minutes because of the using these tools. And so it means, yeah, I need them then to not do things the way they're doing them now or traditionally.

It's like, no. No. You need to actually have done, like, fifty of these or have scaled in such a way that drives a different complexity scale. And one thing that's always stuck with me that I can't remember when I first read it, but that to me, like, the difference almost between the medical device industry and the pharmaceutical industry. The pharmaceutical industry, like, is almost much more mature of, like, having a risk benefit analysis where, like, they actually accept no.

Yeah. Some number of people will die because of some adverse event, think along those lines because that's just the nature of, you know, human variation, physiology, who knows where they are in their life or whatever. But, like, the risk benefits are are are there. Right? And so the benefit it's the benefit's positive, so you do it.

In the medical industry, we still, I think, try to much, like almost, like, say, oh, no. We can't we have to pretend nobody can ever die with this thing or nobody can ever get hurt and all those lines. But then, like, you know in the world it's gonna happen. And so then how how are you building systems that really understand that at a much more mature level? Like, you know, well, what is but at what point is it too much?

Like, no. No. This is not that's not right. And and to your point, like, obviously, you have, like, mod databases, but mod is, you know, I'd say, is a pretty poor signal on that. Right?

The the number of reporting to mod is pretty low, think, along those lines. But for example, back to, like, okay. What does the consumer electronic industry do differently? I mean, Meta scours, Amazon reviews, Reddit, any Instagram posts like that, just for anything that talks about any of our products, and all of that data comes back. To read all that data again as a human being, it's impossible.

Right? Think about the number of people who just rant on subreddit about anything and everything. And it's but we ingest it all. And then after that, right, you can there's still valuable signals you can pull out of it. And, yeah, are you gonna miss some stuff?

A hundred percent. Right? There's no way if I have a million Reddit posts talking about meta products, I'm gonna catch everything in those things even with an AI agent. But I'm gonna catch things that I wouldn't have caught. Right?

And so it's not like this ignorance is bliss kind of, I think, temptation that many of us have. Like, well, let's not look. Because if we look, then we have to do something about it. Like like, no. No.

Let's look, and we'll do our best. And, actually, we'll we'll do something about the things that are really important that really bubble up. Are we gonna miss something every once in a while? Yeah. And then, you know, okay.

We'll figure out maybe can we get better about how to find it. But, like, it's not like we failed because, well, you looked at it, but you didn't catch this one thing that somebody said on this one, you know, Amazon twentieth review kind of thing. Right? And and I think it's critical, and AI is such a good tool for that. And when you think about using AI in these regulated workflows with people, human in the loop, how, you know, how do you you know, where do you think teams struggle with that with, like, you know, transitioning AI to be part of a GXP workflow?

Yeah. I mean, obviously, I just I my I'll say my experience and from what I see out there, I I think we're still in the early days. Right? And so people generally, I think even I'll say there's not even, like, I'll say a good education out there, like, how how to use these things well. In part, obviously, because they're changing so fast.

I mean you know, what Gemini can do now versus what it could do even three months ago is ridiculous, and and there's so many other tools out there. And so in part, you know, there's, I think, just out of, like, hey. How, yeah, how do you really integrate these things more naturally? I think there's also yes. From from a tooling perspective, it's still too much, like you say, the the the hurdle or the the friction of, oh, I have this I have this AI agent over here, I gotta copy paste back over.

Right? It's not really natively integrated into the the ways of working. Like, you know, most companies probably out there still use Microsoft Office, and it's like, know, started just, like, naturally write writing into into a Word doc or whatever from from AI or or even Gemini with Google Docs. Right? It's getting there, but, like, there there's a little bit there.

So I think that's one piece of it. Second, I think, yeah, is the is it one of I say, legitimate real fear of, okay, AI is gonna tell put something wrong in here, and then it's gonna be wrong, which the yeah. But, again, I I I still argue, again, if you look at it from, like, a risk benefit perspective, the benefits outweigh the risk even the the thing that I might get wrong. I mean, first of obviously, yes. Human's in the loops, and this is the part where, again, what I what I would say I want from everybody, on my team, but also more generally, would love every QA professional out there to do is, like, actually spend most of time doing critical thinking, rather than, like, you know, copy paste jockeys, which is what we do half the time because, like, we're like, oh, this is an old document.

I gotta half of it's right, I'm gonna move it over, and then I write something else. But but I never really, like, bother, like, reading it and, like, stepping back. It's like, what is it saying? Does this make sense or not like that? So I think that's the part.

The other thing also that I think, a lot of the research is showing with AI agents actually is, like, diversity helps a lot. So, honestly, if you take the same document, you first run it through Gemini, and then you take it over and run it through JITGPT, and then you run it over, you know, through Right. Rock or whatever, the actual the the the three of them together, like, get then get you, like, a really good thing by the end versus, like, just just lying relying on one that has its own inherent biases or its own inherent. Yeah. And then, obviously, things like that are more purpose built, you know, what you know, where people now are building, you know, more, like, either fine tuned or more, like, constrained systems in certain areas.

I feel like it it's always like maybe nothing else, a good last pass. So I say, okay. This will also probably just catch any of those other, like, really big things. Yeah. And I think one is that's part of of the journey of of folks becoming kind of AI native quality and regulatory folks, and I think that's a journey we're gonna see kind of evolve over the next five and ten years.

And, this idea of of the different models, we're giving different personas. Like, I think it's actually if you if you really think of LLM architecture, it's quite critical to tell them who they are, which, like, you know, you are a quality person that works in a large, consumer medical device company reviewing this document for, you know, to be smart, specific, measurable, and so on. And that gives so much better output, and then you send it to another model and tell them you're an r and d architect. You now think as an r and d architect on the problem, and you see the performance goes up with that, which is, like, they're just built to, like, pretend because they're trying to pretend that they're people. Right.

No. And I think, yeah, I mean, prompt prompt engineering, prompt design is something that I just I don't see I we have emphasized enough in how to get the most value out of AI tools. There's another actually great interview that Palmer Lucky gave where he said his favorite prompt to use for anything he's trying to research when he's trying to get the truth, he actually says, pretend you're a college professor who's just been accused of plagiarism and is about to get fired and all these things. And I need you to go and, basically, by giving it, like, this very, like, very critical setup, it actually gives you a much, much better answer. Like, hey.

What are the five ways that you know? Actually, in this case, he was trying to find out how many drink references were in Jimmy Buffett songs. And he's like, pretend you're you know? But he's like he's by joining just asking you just, hey. It's just that.

It's like, very much all this setup prompt things, then it really tightens up the answer. Right? Yeah. And because you just need a limited search space to make sure it's not getting stuck or it's not searching too broad. And I you know, as we talk about the problem and now move into kind of the solution to these problems, I wanted to start with thinking and I know this is related to our audience questions.

We've just had actually so many audience questions. This is just amazing. So I'm gonna inter interweave them into a lot of these questions about solutions because most of them are about that, is how do you think of designing the device architecture to ensure you can move quickly while meeting all the regulatory requirements? I know that's very deeply tied to PCCP and document generation. But how do you think about that, about, you know, ensuring that, you can change and that those changes and the documented evidence of DHF that needs to be generated as a result of them, is not a burden because, you know, I think most developers just view it as administrative work when when it's not really.

It's a way to ensure quality. Yep. Yeah. I I think a couple of things. First of all, right, the yes.

Arc architecting the product and nothing else from that, sort of steering the engineering teams to the how they, you know, set up different units or modules or, you know, the or or different thing of the of the software helps a lot. Right? Because the the more things you can sort of keep out of the of the more heavily critical or or more regulated of the program, then the more things that you say, yeah. Hey. We can run.

I mean, I I was just calling, like, design control light. Right? These things okay. We we obviously, we still need to trace a certain level because it's it's gonna affect the origins of the total product. And, obviously, we need to still change history and things, but, you can't really break anything, right, if you, or or have any any you're not gonna trigger at any high severity line on the on the risk management file, by messing around with these things.

But these areas, no, these are really key. These really matter. And so how, you know, how to separate that out. Because, yeah, you could say, from a business strategy goals, I'll say letters to file are what you want. Right?

You you you obviously, you wanna be compliant. You don't wanna screw that. But, basically, the more things you can do without having to now put, you know, a regulator in the middle that you have to, like, okay. Send them something. They have to look at it.

They have to give feedback, stuff like that. The the better. Right? And and the way you do that, though, is by saying, okay. Yeah.

These are the things that are really critical. I'm I'm gonna put these things in very nice, well defined boxes and then have, again, clear interfaces to these other areas that are less critical. And so I can focus on those. And then, yeah, then, you know, you can say, if it can't be a letter to file because still, it's gonna affect one of those critical systems or or more significant ones, then we have predetermined change control plans, obviously, are now becoming, you know, we'll say, like, default on, like, every five ten k, it seems like. Yeah.

And and I think that's a good thing. Right? I mean, honestly, I give I give huge kudos to the FDA of how forward leaning they have been in this area. I mean, yeah, it is a ton of credit, honestly. Yeah.

Ton of credit. Yeah. Because this is something that, honestly, if you ask me five years I mean, I was at one of the values when the digital pre cert pilot program first came out. Right? And it and it seemed, like, very aspirational and nice idea, and then kinda didn't go anywhere.

But, actually, it became the PCCP in many ways. And and, really, this this idea of recognizing that, yeah, you build better products by iterating on them and changing them quickly. And that, again, now bottlenecking it at there's there's no however many thousands of companies out there and the FDA only has so many employees, like, that just creates a natural choke point. And so can you can you take away that choke point to encourage the better quality, not to basically say, we're just not gonna look. Right?

Right. Right. Right. And then I think quality comes with with speed in in many ways and then with, putting responsibility on manufacturers. Like, I Paul Jones, our our kind of VP of regulatory who joined us from FDA, he always says, you know, at the end of the day, the people who make it can ensure it's safe.

Like, as an external person, how can you what can you audit in a massive company in in a week or two? So we just need to give the manufacturers the tools to do that. And I think PCCP is one. I think the new, computer software assurance guidance that came out in September is a very quick one. And, you know, if folks wanna really understand FDA, I think page, kind of twenty five to twenty seven there states some really hard facts about what they expect and don't expect in in CSV and CSA going forward.

And, they they just wanna see a lot more, practical approach, risk based approach to it because they're taking such a risk based approach. Yeah. I mean, honestly, to me, the the CSA rollout was probably one of my favorite ones from FDA ever in a sense, like I mean, it's not exactly a direct quote from a butcher, but it's like, they literally said, like, you guys as manufacturers have been wasting your time joining thousands of pages of documents that actually are not driving the quality of anything, and we don't care. And yet somehow that's where we ended up as an industry. And and it was to me, it's it's probably one of those best examples where they really recognized and said, no.

Hey. We care about the quality of the product, you know, obviously, health and and and safety. And this thing that you've been doing, we know is not really the way you do these things, and and you're generating this thing's almost like a false artifact. The and and instead, we're not gonna we want you to do things better. Right?

And and that's gonna say that they're the they really are putting it back because they know, hey. You're responsible. You do this thing right. And that's the way I I used to tell my team at the end the It's like, okay. Is what you're doing driving better quality of the product?

And if because if it's not, stop. Right? I can I definitely go on a full rant about kind of, like, training requirements and, you know, how we how we have interpreted training requirements, as an industry to what really intent of a of a training regulation is around, like, hey? Do your people know how to do things right or not? Where, yeah, we, like we know we we we have become we become so afraid of, like, yeah, the the negative observation to some, you know, detailed technicality on an audit that when we we add all this overhead and friction to the system and say, no.

What okay. Is somebody needs to know how needs to know these fundamental principles to do their job well. Can we talk we taught them to them. Do they know them? Can we just show that, yes, we have taught them and they know them?

And then let's move on. Right? And, you know, I can I can go on a whole rant about training to SOPs, but that I'll I will spare, everyone? No. And I think we're we're about to go into this question that's a little broader about it is which is, you know, when I I talked to a lot of folks, also in my experience kind of leading AI development in in the space.

When most teams try to address this validation documentation challenges, the way we do it is by putting more pressure on engineers to contribute to compliance, hire more quality validation, regulatory folks who are around this compliance space, and shipless often. Right? It's like kind of double whammy. That was not the approach you decided to take. And so I wanna ask you, what are the principles that guide you and the team and how you protect the velocity while building regulatory readiness and safe innovation?

And I love that word safe innovation because in medicine, it's like the safest first. Yeah. Just innovative. Yeah. The and I think I even saw this on one of the questions, so apologies a little bit.

But one of my, least favorite expressions that we always try is like, oh, we can establish a culture of quality. Right? Because if we have a culture of quality and that's modeled from the top, then everything will be great, which is usually just code for the engineers aren't doing what I want them to do, and so, therefore, they don't must not care about quality. And, like, I've rarely met an engineer or a product manager who actually doesn't really want to build an awesome product. I mean, that is the reason why they're there, and this is and that is what they love doing what they're doing.

And so so that can't be that that can't be the answer, or that can't be the what's wrong here. And and I and I find often what's, this is actually something that, I'll give a shout out to. The guy's name was Joey Allison who I worked with at at Sumo Logic who first, he said this to me, and I've stuck with it ever since. Like, people will always default to the easiest, laziest way of doing anything. And so the way you want them to do something is not the easiest, laziest way to do it, then you're not gonna you're not probably not gonna get it at least a hundred percent of the time.

And so then, really, what I I focused on was that, like, how do we try to make the the right way in terms of, like, what we need from, hey. We need documentation. We need validation, all these things. How do we, like, make that as close as possible, the easiest, laziest way to do it? You know?

And and one, simple example, right, then and and I'm working with you guys. Right? Like, okay. We are a Google Docs company. We work in Google Docs, and so we need to be able the and, you know, people write Google Docs all day long.

Google Docs needs to be naturally integrated into our QMS tools. If it's not, it's always gonna have to jump into something and jump into something, and you now nobody's gonna do it. Right? I mean, the second I mean, I bitch about mouse clicks even just in sometimes I give you guys feedback. Like, hey.

I know there's too many mouse clicks to get to, like, what the thing I need to do. It's gotta be And and that's literally I mean, mouse clicks, which are maybe taking me five seconds versus thirty seconds. But you do it enough times during the day, and then and they can know you're gonna skip them. Right? You're not gonna do it.

It's like, no. That's, five mouse clicks to get there. Forget it. Right? I'm not gonna I'm not gonna bother.

And so that's where I see the the the what you need to strive for and what I really keep pushing at is, like, yeah, through two things. First of getting back to, like, first of all, where are people actually working, doing the real work, and and how and how do I meet them there as much as possible? And then for these extra things that I have to do because there are certain regulations that need to have to happen, How do I make those things as smooth as possible? Right? And and smooth, right, so you know, suppose, like, load times on on pages.

It's mouse clicks. It's, you know, it's how much additional, like, editing you have to do after the fact, all these little things that at the end of the day is like, hey. If I can make those very, very, inoffensive, right, or move them to the background, you could say, then we actually have conversations about actually what are the real risks of the product. And engineers love to talk about risk and what how the thing could fail. Nothing else.

They drive me nuts because they think of fifty different ways that something can fail. I'm like, guys, now you're, like, overwhelming. I don't need the fifty. I just need the five. And so so that to me is where, you can really meaningfully achieve still, I say, yeah, you wanna have a good quality management system, or you don't wanna get ripped apart and audit by by your auditors or whatever.

But it's something where now it actually did they the day to day conversations and the day to day activities you're having at work are actually very much, like, on the value of the product. Like, hey. This is is this thing did we test this thing right so that it has the right safety profile? Did we you know, do if we had an issue with this test method, what happened with it? What are the have we actually captured all the right risks here?

We step back and look at them. Because then, like, you say, putting that on paper, quote, unquote, is, like, actually then pretty short or pretty smooth and pretty easy. It doesn't take a oh, man. Writing a change order, crap. That's, like, half a day to, like, write that change order.

And I'm, you know, I'm gonna just wait and stack, you know, fifty changes before I do it because it's such a pain the butt to write the CO. It's not worth it for only, like, one small change. So we're ready to be able to make, like, a two line encoding. Right? You may you may wanna make a two line change.

You make a two line change. You push, commit. It's gone. In our QMS, we make that thing be like, oh, yeah. It's a it's a five week, CEO process, and so it's not it's not worth it for one line for a one line change.

You know, if you had made that one line change in the moment, it would have been, like, way better even the next day. Yeah. And it to me, it's all about change. Right? Like like, companies, we, as as people who make safety critical products, have been taught for a long time that change is not a positive thing.

And I think we need to come to the realization change is the only only thing that's gonna happen to these systems. Like, if you wanna have AI in your system, if you wanna have software, if you wanna have compatibility with any other device, you will have to be able to change often. And that change needs to be part of the way you run your processes and the training and the tooling and the infrastructure and the self documenting kind of world is just so critical. And I think you see it the most in the way people are trying to write requirements. Right?

Like, LLMs, it's actually interesting because in some ways, they're, like, highlighting how, InCoast like kind of prompt structure is so important, having an actor in use cases and all these details. And in my view, especially for the first time you do it, writing requirements is one of the hardest things in device development because it's hard to understand how to detail it enough to make it useful for for other folks and and definitely for folks new to regulated industries where they're used to writing things that are a little less structures. And I wanna ask you, you know, I know we've talked a lot of it in the past is how do you see AI supporting teams, in writing requirements and other artifacts, as the systems get more and more complex? Yeah. I I personally have started using it quite a bit.

You know, you mentioned the inCos and Right? I mean, that was one of the first things I, I asked you guys recently to, like, say, hey. Help me build this into, like, the prompt model and and, an agent in in Ketryx to look at it. Because because, yeah, at the end of day, I'd say, especially, right, if you're a smaller company, or as a startup like that, yeah, you have an you're right. You have a really good idea, and you have a few folks that nice have a have a good idea of the the space, but they're not even they're not experts, and they're never gonna be experts.

And so you need you still need, either yeah. You can go hire, you know, five other consultants that kinda bring you along, or you need, if nothing else you say, the wisdom of the the wisdom of the Internet and the wisdom of the world out there through AI. Because and even just, yeah, to your point, how to write requirements. Right? Requirements and this is something I I guess I still think about a lot.

Like, I see like, if you think about it from a QMS and compliance perspective, there's gonna be the language and the style that you need to have internally because that's the way it's aligned to your culture as a company. It's aligned to the way you work. It's aligned to lingo you speak like that and those things. That's where you wanna be there in that way internally. And then there's, like, the language of, I'll say, the external, stakeholders.

Right? Whether it the FDA, they'd be notified bodies. Because they're they're not in your company every single day. They don't do all things. So you can't just give them this.

Right? Even though this would be this is super compliant. This is great. It's all there. But it's almost like it's like, you know, reading the matrix natively.

Like, you remember Neo and the ones and zeros. It's like, no. No. They are you know, they're out there. They're seeing fifty companies a year.

They see I mean, they need, like, a they need, basically, the novel version where you wanna tell them the story. And so you need to be able to, like, map across those two things. And so, yeah, so, like, AI agents in the middle literally say, okay. Here's how you work internally. Here's what how they need to see it so they they understand what's going on in your world without a one year boot camp to, like, oh, yeah.

This is what we say at Meta. This is how it works. This is how it like that. That's like to me, that's critical. And then even, right, honestly, there's still the basic things of, like, yeah, is your requirement testable?

And if you have a QA engineer that's gonna test it, are they actually even testing what you want them to test? That most of us I'll say, you know, I can run my mouth pretty fast, but my brain goes even faster. And so let alone am I, like, taking the time then and I type even slower. And so I might let how do I really make sure that everything that I've thought about and noodled on and all you need is get really you get makes you spat out, first of all, and then really distill down to get what are the the core ways to structure this thing so that then it becomes either testable or becomes designable or anything along those lines. And and I know you you guys are are using, our stuff quite a bit for that, the LLMs.

Like, how do you think we compare the Ketryx AI to, like, out of the box LLMs or other tools you've used? Yeah. I would say so in the context of of medical devices, I I mean, obviously, Ketryx, I think, so far, it gives me, like, the best answers. Right? I mean, into your well, you know, like, let's say, I mean, I think a lot of the the preprom thing, so to speak, you've already I think you've already designed it in in many ways such that it really is using much more, you could say, the the standard medical device lingo.

Yeah. And right. And it's that it's doing that while reading my entire project file because it's literally right there and it's reading through it rather than me having to, like, say, don't go to Gemini or go to ChatGPT or, heck, given our own Meta LLM, where then I have to, like, almost, like, teach it and bound it much more before it gets me to somewhere else. Use case, which is still complex. Exactly.

Whereas instead and that's what I said. Like, literally, like, the this example of this justification for, like, type testing for software. I actually first put it into Gemini, and I was like, I really don't like that answer. It doesn't doesn't make any sense. And then I kinda I I went back into Ketryx.

Was like, wait. I should have put it as Ketryx. I put it in a Ketryx inside my project, and so first of all, I even contextualize it to my project and say, oh, yeah. These things. And then they actually just edit.

But even there, like, the the external research said, oh, yeah. Here's some precedent you can point to and all these things out there. It was really much more, you can say, bounded or at least that they, you know, had the right level of blinders to, like, say, no. No. You're not looking at the world of what's on the web.

You're looking at it in the context of FDA, medical devices, and in the context of our product and this particular product and what it says, and it gave me then a pretty darn good thing that yeah. I know. I've moved that into my document. I I adjusted it a little bit. Some things that I thought maybe I was like, I had made this get a little too far.

I need to make that big a deal out of it and sorta shrunk it down. But, yeah, no. I do think, Ketryx AI, for the the domain specific things of design controls, so far, at least, has been my favorite tool to use. That's awesome. Thank you so much for, for sharing that.

I think we've spent a lot of time trying to think how do you constrain it, how do you fine tune it to this problem, how do you have the right evals to check that it's still doing this, because the use case is is just so structured and so different than and then many others and has this unique language. You know, I one of the things I'm interested in is what's been the reaction inside Meta as you've introduced this new approach and and a way to build medical grade software? I'm sure there was a lot of fear about it in the beginning. Like, I know when I moved from tech into life science, I was like, gosh. This is a very different style of working.

I can imagine a lot of people will be challenged by it, so it was very exciting to start working with you all. And I was wondering how is the reception to that. So I would say, in many ways, kudos to my my peers at Meta. They have been much more open to it than I was expecting. And and I say, back again, where we are, we are showing and bringing back value to the work actually then actually gets them to really become believers in it.

Because, like I said, if it just felt like overhead and, again, some people do I'll say that, right, you always have people who just self select that. Like, no. That's not for me, and then they'll think we have matters big enough. They go work on some other projects in some other area and and so be it. But for for the people that still because the product they're working, they really believe in the product, they wanna work on that product, and they're okay, so, say, paying the tax or the additional burden of of being a regulated medical device.

As we as we have been able to show some real real quality value to the product either because of you know, we did I'm try so I'm trying to say somewhat generic. Or whether it's, like, some cybersecurity analysis that we've done or, yeah, some, oh, yeah. We're trying to do tracing to these test cases and these test cases then they actually they read the test cases and they're like, woah. Woah. No.

Actually, that doesn't that's not testing what I wanted to test, and and they, you know, they modify, I think, along those lines. I'd say, again, you will always get, and, and there's a few other folks on my team that I'm sure, at least some of your team members can name who are very, free to give feedback on, like, when things like they like, they this thing's not is not smooth. Like, getting back to Zach, but, like, if you make something right. They want and and the way I can say, like, they actually they want to use the tool, and they're trying to use it, and they really like the tool. And so now, actually, they're focusing on not, like, the core functionality of it.

Like, hey, guys. I don't It doesn't look as good on my phone, or it doesn't it doesn't go as fast as I needed to go or anything on those lines. That's where I think you still see, well, to me, that's the testament. When somebody actually still seeing, actually, they're they're getting some real value out of the tool, not just something that they are, like, being forced to do, is when you know you have you gotten got got it right, so to speak. And I'd say for the most part, just about everybody who I have at least now working on regular basis not right.

It's never gonna be everybody. Again, back to the it's never a hundred percent. It has been at least happy with it, enough to to say, okay. This is not this is not so bad, or it's worth the it's worth the effort. In other areas, I feel like the where, we're still having challenges, actually, something that, actually, there are some other legacy tools that kinda just make it, you know, we have to almost, like, rip out a bunch of process, and then get them to come over.

And then people who are invested in the in the existing process are like, they're a little more hesitant. Yeah. But I think that the mission of making medical device medical products is just so amazing. Like, we have to figure out how it how to make it easier on people. And it actually kind of leads me to to basically my last question, which is if you're advising someone in your role at another medical device company starting this journey or another regulated company overall, what key piece of advice would you share, and what do you think our industry can do to build better products for for patients, for consumers?

That's a big question. I mean, I do think and then this, I'll say, I'll go back to even, like so as so NanoStim, I was the first quality hire. Right? I was a leadless cardio pacemaker, class three software hardware, the whole deal. The and I had I had a great I had a great boss there who who, I'll say, kinda coached me this as well, established a QMS from the beginning, but that that QMS better be, like I said, well integrated and smooth operating because then because it's very hard, and it's a it's a massive pain to just basically, like, try to dump a bunch of information, like, at the end.

Right? Mean, many companies are like, oh, no. We'll avoid it. We'll avoid it. We'll avoid it until, okay, now we're getting ready to submit.

We'll load everything in. And that is such a brutal process. You can do it, but it's just brutal. And you honestly you always screwed up or you realize actually, oh, wait. Now we probably could have fixed that seven months ago, we never did it.

So I'd say that. It's like, basically, first of all, you know, look at the tools that that your your company wants to use or is using, figure out how you can best, integrate into those. And so it might be a different different, systems or different QMSs out there more easily or more difficult to integrate into whatever that you're trying to do. So really pay attention to that. Pay attention to the integration and how how natural it feels, and then make it as easy as possible for people to start writing requirements, you know, as they think of them or how to start things.

And then you can can just you know, don't be afraid of, like, just revising, revising, revising, which usually means don't put fifty approvers on everything. Right? Keep it keep it light. Keep it keep it keep it very high highly, accountable for the person that is signing off. Yeah.

And how do you how do you make it easy for them for that person to to make a decision, I think? To me, it's all about, like, how do we make sure the right decisions are made by the right people at the right time and they are proving and it moves forward? So, Lucas, I know we can stay to however much time you have afterwards to answer some questions that folks have, but I just wanted to thank you so much for sharing your experience, your insights, your time, sharing a little bit about Meta's approach to how teams can prove fast and stay compliant and keep innovating all the time. I wanna thank everybody in the audience for joining us today. If you have any questions, the Ketryx team is available to answer.

Shoot us emails, check our website. You know, there's a lot of feedback kind of everywhere. If you write something on the user, it will come back to you. I'll answer some question now. And and if you enjoyed the session, I invite I invite you to join our webinar, which is Jira for IEC sixty two thousand three hundred four, where we'll go into the detail of what regulators expect from teams operating under IEC sixty two thousand three hundred and four, what are the common backgrafts, and what are the real world implications for Jira.

And, Lucas, I just wanted to thank you again so much for the time and the effort and the partnership, and, we can't be more excited than than the journey we're on with Meta. And I think it's wonderful just the scale. So thank you. No. Hey.

So, obviously, I've I've enjoyed and and loved the the partnership so far, and a lot of the even just the the feature requests of mine that you've, paid attention to or accommodated. Obviously, we have, we're very excited about the road map that we have here at Meta. Right? I intentionally did not talk very much about the actual products that we're working on because we have not launched them yet, and so those are not things that I that I can talk about. But we do have, some very cool things coming in this space.

And, and and like you said, honestly, when I first stumbled into the medical device industry in two thousand and four, I was like, oh, yeah. It's like gives you a great reason to get up every morning and come to work because it's very much it's you know the products you're working on are fundamentally helping people in a positive way. And always every somebody has a a cousin, uncle, a mother, a father who's affected by whatever you're working on, and so it really creates great meaning. And that's where where I feel like if if we remember that and then we we build really I mean, systems and processes that can really help then us to build the absolute best solution for those things, then everything else kinda takes care of itself to a certain degree, or or a lot of things can take care of themselves much more easily. Yeah.

And I know also, Lucas, there's a lot of roles open in in in your team, so I I recommend folks go to the Meta website. Go, check out the the QARA and other kind of compliance roles, and I think, one of the best teams out there to join. And, you know, I'm proud to be part of it as well. Yeah. I appreciate that.

Yeah. Definitely, search under medical devices if you go on the website because, otherwise, could have run, like, a thousands of, you know, job posting throughout all of Meta. But I I try to always put medical devices in the header of my job descriptions just to make it a little easier to find. Yeah. Wonderful.

Thank you. So, I'll stay here to answer some questions. Lucas, you're welcome to stay for whatever period you can, and then, I know you're quite busy. So one question I'm seeing here is, do you think pushing traceability to the right of the v, effective, effective design review and efficient requirements initiative might be degraded. So one is we're talking about pushing it to the left of the v, more traceability done earlier.

I actually think it's just gonna make things, more efficient, more effective, effective, better reviews because you're gonna do it all the time as things come up and not towards the end and definitely not towards the end after everything is said and done. Like, where I see a lot of teams struggle is they produce traceability at the end, but then realize they have gaps. They have things that are not passing tests, and then they need to do a whole rework, which is completely outside of the program timeline. So I actually think the more you automate it, the more you do it early. It actually increases the, you know, the whole thing and makes it Yeah.

I get to me, I I I almost feel like it's saying left or right of the v is even, like, a the wrong terminology. To me, like, it just is. It's almost like the base. Right? There's, like at the day, again, I very much use a a kind of a systems thinking to everything.

And so to me, everything is either an input or an output or a dependency on something else. And if you don't have that defined and controlled all the time, like, you're definitely gonna it's definitely gonna screw up. Right? Essentially, it's one of those things where you've now isolated things that need to be coupled. And so there's yeah.

The it's like it needs to be just default part of what you're working on is, yeah, what is I I've I was like the the SIPOC model, you know, suppliers' inputs, processes, outputs, customers, and you did almost, like, think of anything from that perspective. Okay. Is this coming from something else? And if not, okay. You're the first in the line.

What is it doing? Where is it going? And is that next thing already there, or do I need to, like I just put a placeholder out there for it because I need to go ahead hit it? And that needs to be, yeah, real time all the time. And you to your point, yeah, you you discover so much by actually looking at traceability.

I think, to me, always the challenge of traceability is just more about the visualization. Right? Because now it gets so big and so complicated. It just even just again, for our human brain, we just can't think of that many spaces on dimensions, but it just needs to always be there. And if nothing else, where I would love to continue to see systems get better is, yes, how to how to really visualize, help humans visualize, almost like to say the the meta level and meta with a small m.

But, then, and then at the same time, like, flag, right, or or, you know, kinda, like, think about kind of hotspots or areas. You know, actually, you'd also just, you know, drill into this and go look at this over here because this might be problematic in some way, or this relationship seems kinda weak or doesn't really make any sense, like, all those Yeah. And I I actually think that just, like, coding tools, like AI generated coding tools are just gonna make this much harder. Like, whatever how hard agile was, you know, five, ten years ago for the regulated industries, Like, coding tools are just gonna make that pain point a hundred to a thousand times worse because they're gonna move so much faster now. Oh, yeah.

You know, let me end with one question that I I'd actually I I I'd love to hear, and it's from, Tanima Ghosh, who's well known leader in the space. How is Meta conducting mock audits for AISAMD to be inspection ready? Do you have any thoughts about that? Because I know you guys are basically, like, doing a bunch of things every day, right, reviewing stuff, getting ready for it, and reviewing the traceability. Like, you do internal audits almost every week in a way.

Yeah. That's why it's it's kind of a I'll say I I don't know if complicated is the right term. Right? But, yeah, by the to me, again, like, in internal audits, it's one of these where, again, I I sort of see, okay. Yes.

From an outside external regulator perspective, they know that you kinda need to do them or see them in a certain defined way to to show and to be able to kinda have the narrative. But but to, like, your point, right, I said at end the day, it's like it just needs to be, like, an always, like, real time thing that's happening and that's always kinda out there. Right? I I do yeah. For me, I I I'll say I don't have a great answer, but it's something that's still actively working.

It's like, okay. From AI products perspective and AI and and AI product performance, and especially, you know, the way, we started talking about it more is, okay. Really, with AI, the big things you really wanna watch out for is, like, egregious errors. Right? Like, the okay.

Errors within, like, you know, the almost, the one sigma kinda spread in the middle. Okay. It's not great, but, right, nothing's brazen. It's like the the tails that, like, are just, like, really bad. And so how do you really kinda keep checking on those, you know, clipping those as much as possible that I'll say we don't I'll I'll say we don't have a great solution yet.

We're working on it. I definitely that's what I'd say. I've been spending a lot my time reading primarily in this area around controls and ways to have continual audit or quality of more AI based tools because it is one where I feel like, yeah, so far, the pace of development on the let's say, the product side is going so fast, and we have not kept up as well on the on the audit side. We're more so at this point, we're just kinda like, we're hoping it's fine or anything. The what is back to we don't we don't we have no we're not scaling, I think, the audit quality part of it as quickly as we are we have scaled the product part of it.

Yeah. And I think because it's in in many ways, it's, like, inherent to how it works for y'all. Right? Like, you're producing traceability and checking and auditing it all the time. Like like, the core part of audit, right, this traceability analysis is done for you all every every minute.

Yeah. To me, that's what I said. To me, it's not so much that that that part I feel like, yes. Right? To your point.

Right? I mean, that's I feel like almost any single day, right, I can log in to Catcherix and go to my releases dashboard, look at the dashboard, and it'll show me everything that's there, not there. So, like, that that part is easy. To me, the part I worry more about is, actually, is on we have AI in the product itself. And so now that basically, that that that thing, and, you know, we're pushing code all the time, and lots of different teams are pushing code to it or just learning from different things.

Like, did it meaningfully change on me without me even knowing? Yeah. Yeah. That's what the eval sets for. So I know one last question I was asked just right now by by Travis is, you know, we mentioned the requirements engineering is a hard thing to do.

I'm in the middle of trying to change processes from developers that are used to user stories to a more agile method. Any recommendations, resources about concrete examples of what good requirements and design specifications would look like? So one is, you know, kind of the Enco's handbook, I think, is is the the highest bar. I actually think, in the twenty twenty three premarket submission by FDA guidance for software device functions. They give some more examples.

They actually don't necessarily expect that level of rigor, and I think it depends on your device class. If you're building a pacemaker, it's probably more appropriate than if you're building, like, a web app for nutrition. And I also think that, Travis, I know I know, y'all are users, so I'd warmly recommend, put it in the Ketryx either agent or assistant, and I think you would be pleasantly surprised how we can take a user story or even a large group of user stories and turn it into requirements. Yeah. Actually, honestly, that was my my first sort of question response and be like, why are you trying to change them?

Like Yeah. They they fundamentally what is it that the way they're working right now doesn't work for you? Right? And is it because yes. Okay.

It's a nightmare from a auditor perspective or a submission perspective. Then okay. Then then then don't change them. Figure out how to how to how to get get you to that part of it. If, I mean, if they're building crappy code using user stories and you think they'll build better code or better products by switching, okay, then have the conversation, there.

And yeah. And I think using different, you know, things like InCos or or the FAA guidance and and, you know, Ketryx AI agent on to start coming up either with scrubs of those or examples of them is a good way to go. But I would almost even first start with, okay, why why do you need them to change, and and what's what's driving the change? And if it's more like, I say, I'll call it a a you problem of you then to your next customer, then you figure out how to massage or convert it back to thinking like, hey. How do we work internally versus how do how do we need to show ourselves externally?

It's more about the translate you need you actually should be focused on, like, building a translation engine, not on getting the change. Yeah. And how do you make sure that eventually the testing that you do for it has these acceptance criteria and the structure? Like, for us, for example, you could generate a story. The story can then impact the requirements tree and modify the requirements tree.

Okay. But with that said, Lucas, thank you so much for your time. I appreciate it. Thank you everybody who stayed and listened to us. Sorry for the technical difficulties at the beginning, and I hope everybody has a wonderful holiday season.

And and then I see you soon and keep building. Yeah. Hey, Noah. Thanks, Ed. It's my pleasure.

Obviously, it's always great, chatting with you. I have a lot of fun with our conversations and nothing else. Little interesting to do it live with other people watching other than just the two of us at a coffee shop, but always always a good time.