---
title: "How to Use Agentic AI to Systematically Prevent Deviations"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/820kxq1v5p"
content: auto-caption transcript, proper-noun corrected
---

# How to Use Agentic AI to Systematically Prevent Deviations

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/820kxq1v5p)

---

Hi, everyone. Welcome, welcome. Thank you so much for joining me today. I'm gonna go ahead and share my screen and get this party started. Hopefully, all of y'all can see my screen now.

So welcome to today's session on how to use agentic AI to systematically prevent deviations. I'm Jen Dixon, client operations manager here at Ketryx, and I'll be walking you through how agentic AI can help QA and RA teams maintain control as AI becomes part of everyday development. We'll talk about how AI agents can operate inside your existing roles, Jira existing tools, Jira, GitHub, Azure DevOps, and enforce the processes automatically and prevent deviations before they occur. Now a little bit about me before we maybe I'll jump into this later, but like. I've done this role myself very much manually, you know, enforcing your QMS within Jira GitHub, and it's definitely a time consuming task and I'm really happy to present that we can do a lot of it now automatically.

This session will cover best practices. Use real life examples. Lessons learned from teams that have already embedded these tools, these agents into regulated workflows. Alrighty. So let's go ahead and get started.

So before we dive into the content, I want to introduce Ketryx. You're probably wondering what Ketryx is. Ketryx is an AI native compliance platform built purpose built for life sciences and life sciences teams to deliver products Faster. We are interoperate your tooling across various teams across your organization and apply your QMS rules across it. So that's something I definitely had to do manually across multiple systems.

With that data, we can also automatically generate the compliance evidence and documentation you need. So whether that's a five ten ks and IND, we're here to document that and generate that deterministically or or with agents. And with our AI agents, can automatically trace your requirements, assess change impacts, validate test coverage, and also flat compliance risks in real time. For quality teams, this means less manual chasing of evidence and more confidence in audit or submission readiness. And for R and D teams, the benefit is more time spent coding or working on your developer native tools and not necessarily doing your manual documentation.

So I'll briefly introduce myself and my co host Bailey, who I'm very excited to do this webinar with. So I come from Canada, so I did my bachelor's in engineering and I've also been a licensed engineer Ontario for about a year now. I started my career at Snapdive Medical where I was working in manufacturing quality and that was a really great experience to understand the physical analog for a lot of the practices we do day to day. I then moved on to become a director of quality and regulatory at Omnition Neurotechnology. Very different is a Sam D company, so that was very interesting to apply all of your ISO thirteen forty five principles, but for a software only environment.

And then now I am a client operations manager here at Ketryx. Internally at Ketryx, I'm the quality and regulatory person for the entire client operations team. So whether, you know, whenever there's an issue around quality regulatory new standards that come out, I'm here to sort of be a voice within the company and make sure that we're following what the regulators expect of us and our clients. And Bailey, please, would you like to introduce yourself? Absolutely.

Thanks so much, Jen, and hey, everyone. So excited to be here today. I'm Bailey Cantor, our director of solutions. My background was originally in physics at Chapel Hill, and then I went on to be a systems engineer at Raytheon in the defense space. After that, I was a back end software developer, as well as a scrum master in the healthcare space.

So my technical background, combining regulatory requirements written for the defense, as well as implementing it directly in the source code, showed me how challenging it is to be agile in a regulated industry. And so I'm so excited to talk with you all more today and how you can use adjunctive workflows as well and embed that directly into your SDLC. Thank you. Thanks, Bailey. One thing I love about working with Bailey, we've worked together now for several months, is just how well she is at listening to all of our clients' problems.

She can work with any client from any industry, and she understands exactly what they're going through because she really understands those pain points. So I'm sure everyone at this point has heard a lot of the hype around AI agents. Let's quickly jump into what agentic AI is, how it works, and state set the stage for why it matters. So across life sciences, AI agents are already reshaping how teams operate. Certainly, you know, even outside of life sciences, we've definitely gotten prompts and ombuds from from, you know, various GPTs or whatnot.

Messaging apps and things like that. But but in life sciences, there's even more disruption because of all the automate automation that needs to happen. So analyses and now analysts estimate that these systems could unlock up to two forty billion dollars worth annually through faster time to market and automation. I'm sure if you've had drugs you're waiting for or technologies you're waiting for, particularly in the life sciences industries, it tends to move quite slow and automation could unlock a lot of additional value just by getting the product to market faster. About fifty five percent of today's workforce hours could be automated by AI agents, so freeing up teams to away from those repetitive tasks and more on those sort of critical tasks where you need a lot of accountability.

We're already seeing measurable results. So one major insurer achieved a ninety percent reduction in documentation time using AI agents. Imagine insurance, finance, even law firms are all using AI agents. There's a lot of documentation to go through. This ninety percent figure reflects that what many of our customers, including Medtronic, Harflo and DeepHealth are seeing in our platform.

So imagine that same kind of efficiency applied to validation, traceability or complaint processing. You know all that post market data that's available out there. There's so much of it. Why don't we get the robots to do some of that work for us when it's really repetitive? So my question now is, you know, how do we bring that potential into a regulated environment without losing that control, right?

That's something that we're always concerned about. We want to demonstrate control. We want to show that we're meeting the regulators expectations. And that's where AI agents can really solve this issue. So we just looked at how AI agents are reshaping life sciences as a whole, and this is what is the transformation is inside the medical device environment.

Imagine AI teams AI helping teams generate design controls or code automatically. So think about, you know, the breakdown of like this high level idea, but into individual tasks. You might have someone who's a code specialist, a trace specialist. Well, now we can have agents who are instead that specialist so it can surface relevant information instantly with natural language. That's something we know that that agents are good at.

We can summarize dependencies and even monitor projects continuously for coverage gaps or redundant work. These are the types of real life practical use cases, so you'll see several of them on the screen right now right now that already how products are built. So faster iterations, pure errors, and stronger traceability throughout. Now, Bailey, I don't know about you, but certainly in the earlier roles within my career, a lot of my work was around just reporting what has happened in the past week, right? Or summarizing things like here's all the new requirements.

And though it was a good learning experience, I know that it was very hard for me to take on more critical work or more interesting work because I was kind of bogged down with all this, like, competitive stuff. Did you encounter that as well, Bailey? Yeah, exactly. I remember filtering through hundreds of requirements. We actually use Doors, which is another tool in itself, and having to command F and find a requirement or go talk to a SME who was an expert.

I had really no way to bring all of that context relevant information to the forefront and centralize it for me. And so some of the features you're showing on your screen is actually how you can build those agentic workflows in AI, not just in one tool, not just in Git and Copilot, but throughout your entire SDLC aiming for compliance and regulatory completeness. Yeah, for sure. And I know when I talk to folks about it, people are definitely worried, could it replace human judgment? I certainly think that there's ways to use it correctly, like any tool, where you're not offloading that accountability, right?

You're actually just utilizing it, utilizing the power that it has. So again Exactly. Yeah. And it's keeping that human in the loop too, and that's key here. And we'll definitely show that throughout this whole this time here today.

Absolutely. So let's give teams the intelligence and visibility that they need. Certainly, you know, had to do that in the past. It's just part of my daily chores, and now I feel like it's part of just my working persona, my professional experience, my professional benefit. So let's quickly talk about the agenda today.

AI agents is exactly what I'll explore. And, you know, Bailey's actually gonna demonstrate some of the agents that we have in house and how they're, you know, making our day to day workflows better. Again, we use Ketryx to build Ketryx, and it's certainly benefiting a lot of our customers. So first we'll look at the challenges that come up when AI operates inside a regulated workflow where every output needs to be validated, traceable and explainable. Then we'll cover strategies for maintaining control, including how to combine generative and deterministic AI and I'll get into the details later about what that difference is, Both creativity as well as reliability.

That will that will a five step framework for implementing AI agents in a risk based compliant way. Might be building your own tool internally where you might have agents integrated into your workflow. So we'd recommend you pay attention to that part if you are. And finally, we'll close with real world case case studies and some short demonstrations about AI agents today. Folks, if you are you know, we'd like to engage the wider team here at the attendees.

So if there's anything you'd like to see, please let us know because Bailey is ready to show you exactly what we have in store or what we can achieve with it. Alrighty. So let's start off with a poll here, a little pulse check. So As your organization begins adopting AI, maintaining alignment with your quality management system can be a little bit can mean different things. So from design controls and validation to change control and oversight, this poll will help us understand where you expect the most effort to go as AI becomes part of your workflow.

So go ahead and select all the areas that apply. We'll take a look at the results in just a moment. So I see I have twelve minutes past the hour. I'm just gonna go ahead and say at thirteen we'll review the results. So where do you expect the most effort to maintain QMS alignment as your organization adopts AI?

Bailey, I'm not sure if I recall. Did you implement any tooling where, you know, you kinda had a series of, like, butting heads of maybe quality or regulatory teams where, you know, you really had this wonderful tool and then suddenly, like, no one wanted it in a way, but they wanted the results. Right? Did that ever happen to you? Yeah, absolutely.

I mean, especially just working with challenging different stakeholders who And I mean challenging in the sense that we all have our own goals and initiatives that we're trying to accomplish. Regulatory and quality, we're really focused on ensuring that we're following our QMS and our SOPs and approvals, and working with the software team intimately. It was about how can we develop this code as fast as possible? And we often were at ends when trying to achieve that. And I know, as an innovator myself and working on a previous software team, I want to use AI as much as possible, but we really need to be strategic about how we do that and how we use validated AI.

Alrighty. So I see we have the results of the poll now. I think folks should be able to see it on their screen. So so where do we expose? So near the top of the list.

So validate AI tools meet documentation, audit trail requirements, and output quality, managing AI generated content to avoid incomplete, inconsistent, or noncompliant artifacts. We'll certainly touch on that topic just after this. So regulatory alignment's an interesting one. I was thinking that maybe most of the folks on this call are coming from the regulatory side, so I know that there's a lot of global requirements around AI and maybe aligning all those requirements globally will certainly be a challenge. I know that even smaller countries, I think I remember Chile has like AI specific requirements and just managing all of that can be really tough.

And then it looks like change control is, you know, also up there in terms of things that folks are worried about. And human oversight is on the lower end, which I'm a little bit surprised by. So ensuring reviewers stay in the loop as AI output volume increases. Bailey, any thoughts on like why that might be or maybe any comments around our top performers on the poll? Yeah.

I mean, it makes sense to me that the regulatory alignment is up there, right? Validating AI is really a challenge. It's commonly not understood today, and we don't have clear guidance on exactly how to do that, especially when you implement generative AI or AI learning algorithms? How do you validate that? So that makes sense to me, and we'll definitely show you how you can use that in the platform.

And maybe the human in the loop is lower because we can always have, you review that output and then take action to review it in the system. So definitely interested in learning more about these and then also showing how Ketryx can solve some of these in the demo today. Awesome. So jumping ahead into maybe some of the regulatory aspects here. So I saw that change management was kind of in the middle.

I'm a little bit surprised by that because change management continuously becomes a challenge in our ever changing world. We're moving very, very fast. Thank you all for the poll responses. It's important to remember that AI in devices isn't new. So the FDA actually approved the first AIML device back in nineteen ninety five.

So think, you know, basically before everyone was on the Internet, that device, the Papnets testing system, AI pathologists in reviewing cervical cytology slides. So, you know, all that manual appending the slide on there, reviewing it, that's where they applied AI. So nearly thirty years ago, the FDA was already open to the use of AI. What's slowed adoption since the isn't really the regulatory resistance, it's the change management aspect. How do you adapt your quality management system to support AI development?

How do you adapt your design controls, your your standards to that that new world. So AI introduces new variables. Right? Models evolve, data drifts, validation becomes more of a challenging process And it's more than just a one time event, right? There's options out there for continuous learning.

Managing that kind of change within regulated frameworks across tools, teams and documentation is really what makes adoption difficult because you almost have to solve those risks as problems like multiple times across the entire workflow. So the challenges are here. The regulatory alignment, data integrity, clinical validation, all those requirements, they're not new problems, but they're amplified when AI is part of the workflow because there's so many unknowns. The opportunity now is to modernize how we handle change of control so innovation doesn't have to slow down every time AI or automation enters the process. So the FDA did is trying to help us.

Right? So they did put out a draft guidance back in January twenty twenty five. I doubt it's gonna get finalized anytime soon, but as always, these draft guidances are represent the latest thinking of the FDA. So take a look at it if you haven't already. The agency isn't saying slow down.

In fact, it's saying be structured. Right? Use those existing life cycle processes and policies to help you build AI that's safe. So you need a robust QMS that covers the entire product life cycle, including the post market phase. It says, hey, you gotta have strong data management.

So that might be kind of new. Right? I think in the past, if you're building you know, you're validating and verifying something, you kinda just grab the data you need and say, okay. Yeah. We use that data that it makes sense And we validated the results were good and wonderful.

Okay. And maybe next time you grab the same data or different data. If the FDA really highlights how you need to really be sure that you're not moving data or reusing data in the wrong places, right? Segregating your datasets It calls out that UI UI and labeling is extremely important. And and I think we've all experienced that before.

Right? So there there might be when you've maybe gone to an online website and you're thinking, I'm talking to a human in this chat right now, but it turns out it's actually a bot. It's actually an AI. And that lack of transparency is the exact situation the FDA wants to avoid for all medical devices, right? They want all of their users to know if there's AI power behind whatever that functionality is.

The FDA also stresses that AI specific risk management is something you need to take into account. So imagine, you know, you already have cybersecurity specific risks. You already have human factor risks. Now you need to think about what is the additional hazard or new technology that you're introducing here that invites these hazardous situations. And they probably is gonna be, you know, model drift is likely going to be one of those things.

Data bias is gonna be one of those things. And so you know, how do you manage that risk? And also, can you also manage that risk through a predetermined change control plan? Bailey, I don't know about you, but you've interacted with a lot of customers. I'm sure their PCCP comes up a lot when you have these conversations.

I find that a lot of folks might actually misinterpret the purpose of a PCCP. They're like, oh, let's just add it onto our submission and avoid future submissions. I think it's a really great ways to actually showcase how you are continuously maintaining the safety and effectiveness of this device because you have a PCCB. Like, it's actually safer because you've gone ahead and planned those changes. Do you find that that's a common misinterpretation?

Yes, definitely. And it's one of them is a challenge in creating your PCCP. What is the designation? What is the workflow that we're going to establish for implementing change in our system? That's the first step in my mind of creating a clear PCCP.

The second part is executing the PCCP. How do you align your PCCP with your tool stack and embed it into the correct workflows to make sure that all changes in your system are coordinated in a controlled way and that you're not skipping any corners or you're not doing approve or missing an approval where you need one. How do you enforce that PCCP? And then how do you align that with being agile? So, yeah, lots of challenges around that, Jen.

For sure. And certainly, how do you make sure all these approvals go through? Like, what is the process you wanna utilize? Chances are you wanna just use your regular quality management system process or an automated or a deterministic process to do that. Right?

So embedding these directly is a really great way to leverage those controls as like evidence towards our PCCP and you can ensure continuous controlled innovation. So again, lifecycle management, nothing new here. Make sure that you're applying your thirteen forty five. You're applying you know, if you're doing software sixty three or four. There is another standard out there that's relatively new.

So ISO forty forty two thousand and one. So this might be the one you'll encounter if you do, like, I put Google, like, what is the AI management standard out there? I always say that this standard is a great one to think about as, like, a North Star. So it talks a lot about AI specific practices, methods, things to consider for any AI management system. So it's not just constrained to medical devices, actually useful in a lot of different areas.

However, I see it as a really great starting point of where you should be, how you should think about your organization, but not necessarily something you wanna hold yourself accountable to. We have yet to really see a lot of the benefits of utilizing the standard, I do wanna put it out there that, you know, there is a way for you to think about, like, what is the overall framework we wanna use? So what is the governance framework? And then maybe you can build your practices in alignment with that framework and reuse what you already have. Right?

Like, don't feel like you need to build a forty two thousand and one quality management system that's completely separate than your ISO thirteen forty five quality management system. Yeah, and one thing, Jen, I'm thinking about that regulatory experts might find really frustrating or hard to manage, how do you keep up with all of these standards that come out? How do you keep up with the new regulations that are dictated by the AIEU Act or forty two thousand and one, you always want to make sure you're in alliant and not out of compliance. So it's definitely a challenge there as well. For sure.

Yeah, there's a lot of ways to get more passive notifications when these things come up. Obviously going to wraps is a great way to keep up with the latest and greatest regulatory changes. You can always come to our webinars. We'll always try to notify you when something big is happening. But yeah, it is a full time job to keep up with all the regulations and the latest interpretations.

So we talked about the hype. Now let's talk about, you know, ways that AI hasn't succeeded yet, right? So a recent MIT study found that ninety five percent of AI pilots fail. Certainly, when I read that, I wasn't necessarily super surprised. It's certainly it's only alarming that it's a whole ninety five percent.

Think of all think of eventual waste out there of, you know popping in by, you know, effort and not really getting the benefit you want. When you dig into the why, though, the reasons are very are not so surprising. It's just anyone who's actually worked in a regulated environment before or has a strict, you know, operational excellence workflow that they need to achieve. So the first reason why is that there is they found that there was a lack of measurable business impact, right? We see a lot of really high level lofty ideas of how AI is going to change the world.

But when it comes to working with teams, teams want a very specific business out out impact or a KPI that they wanna hone in on, and many of those implementations didn't really focus on that. So many iProjects never make it out of that prototype, you know, proof of concept mode. Instead, they're generating, you know, kind of interesting outputs, but not nothing that anyone can back. You know? Second, they found that there was poor integration into workflows.

So maybe the AI was something that that lived completely outside of their existing toolset, and they just told folks, hey. Like, every time you do x y z, make sure you go to this other tool and you do some work there and then go to this tool again, which I think Billy you and I have experienced before is not fun and is not efficient in any way whatsoever. And quite often you make a lot of mistakes, right? When you switch tools. AI often sits Oh, yeah.

Sorry, Jen. I was gonna interject there. Yeah. Absolutely. That context switching really slows you down.

Teams just want to stay focused on their preferred tools and the way that they work. For sure. And because it's sort of like sitting on the side, it's very quick to you're like, say you're having a really tough day you just wanna stay focused. You're very quick to, you know, put that off to the side and, you know, not really work with it. I don't know.

Has anyone recall the, like, Clippy before? I don't wanna use Clippy a lot when they were using Microsoft Word. I know that, you know, that worked for like the first couple of times you open the application, but afterwards they're like, no, thanks. I don't need Jenny to focus up. So jumping back to the study, I'll not talk about completing any longer.

The third reason is the learning gap. So many eye systems don't adapt over time. They retain feedback or they don't retain feedback. They can't adjust the context. They maybe don't learn a lot about how you like to learn.

So there's also, you know, are they keeping up to speed? There's a misalignment, and this is something I think daily you and me talk a lot about. It's like, you know, we have something coming to us with a problem. What's the use case that they'd like to resolve? Tend to prioritize high visibility areas like sales or marketing.

While some of the biggest ROI return on investment often comes from improving internal process heavy areas like quality and documentation. And finally build versus buy trade offs. I'm sure you know when chatty PT came out and they made their enterprise version. There was like a thousand ships cast of different folks making their own internal AI, maybe training it in house. There's a lot of effort that goes into refining it.

So projects built fully in house tend to stall due to limited resources or expertise Cause you know, you didn't really hire for that role. Right? Prompt engineering has almost become its own profession. And whereas if you externally develop something that isn't domain specific, it might not fit the use. So you're actually looking for something in between, right?

So something maybe externally developed that has resourcing behind it that is domain specific solution and it tends to be and what you're looking for especially in the regulated space is something that's already validated and is maintained continuously, right? If you do something in house as soon as that person that expert if they manage to get something that works as soon as that person leaves, which, you know, of course they might get poached because, know, they're a fantastic AI ML engineer. As soon as that person leaves, it's really hard to maintain that, right? So for quality and regulatory teams, these lessons are especially relevant. AI adoption succeeds when it's measurable, integrated into your daily workflows and built to evolve safely over time, not when it lives in isolation as a pet project.

So I think folks on this call probably at this point have all heard of AI slop before. It's only going around the social medias and you know it's definitely a cause for concern. So for those who don't know. AI slop is when you run into you have low quality generic and often incorrect content produced by AI, and you're kinda, like, surrounded by it. Like, there's so much of it.

I think there's lots of people who are using AI daily for maybe just their entertainment, but also sometimes for their work. So when AI implementations start to fail, sometimes it's because or one of the signs that it's failing is because of all this, like, AI slop that's created. Right? All these, like, unintelligible outputs. You've seen it before.

The outputs are double polished. This kind of slop immediately turn off. Right? So you see something like this. You're like you're like heavy.

All of us, it doesn't make any sense. I'm gonna move on to what I know is right and correct. So it tends to be it tends to show up in a few predictable ways. So things that don't make sense, repetitive outputs, frequent hallucinations, so they might get something wrong at first and then get it right if you prompt them again and then get it wrong again, depending on, you know, the time of the day, the weather. So that is also something we don't like in quality and regulatory.

And because of this, we actually lose a lot of time. We might potentially lose more time if we didn't utilize this AI at all, right? Because you're spending all this time manually reviewing the outputs. So what are the mitigation strategies? So the good news is that these issues aren't inevitable.

There are symptoms of unstructured use and there's ways to prevent them. So one of them is human oversight. So ensure the experts remain in the loop for review and correction. So you never want the AI to be generative content to be just produced willy nilly. Like you want someone in the loop for that And you want a master prompter.

So someone who can help set, you know, precise expectations and constraints for the model. This is something that we have expertise in house at Ketryx. But if you're building your own tool, you really wanna make sure you have someone who can refine prompts for you. And lastly, data integrity. So grounding the AI in domain specific data reduces errors and keeps results relevant.

So think about your AI is a child AI almost and it's only grown up with books about patents. For example, I was talking about patents the other day. If they only get exposed to patents chances are they're only going to provide you patent type data outputs, right? So you only want to expose it to data that you expected to learn from. And lastly, the most effective way to mitigate these risks is with AI agents.

What does that mean? So we're agentifying it. So we're pairing down the amount of data each one of these AIs is exposed to. So we have systems designed to pair generative innovation as well as deterministic controls. Some of them are only going to generate one type of data and some of them are only gonna have deterministic processes that they're gonna enforce.

In In other words, agents don't just create, validate the document and maintain traceability as I go and they specialize in those things, right? So maybe it's validation of required requirements is reviewing test cases. That's how we move from move from AI. This is a bunch of noy noise, something that is reliable and safe because each of those agents are checked for their own validation and performance criteria, much like humans. Alrighty, so how do AI agents balance generative and deterministic?

So I've mentioned that a few times. Let's start with the generative task agent. So this is the creative side of the system. I'm sure, you know, you've had friends in your life were extremely creative and come up with these crazy great ideas. Some of them are good.

Some of them are bad. Hopefully mostly good. It executes individual tasks like drafting requirements, writing user stories, and summarizing test cases. It's fast. It's flexible.

It's capable of producing a lot of output quickly, and this is what we typically see when we think about AI these days. But it's on it on its own, it doesn't really know whether that output is correct, compliant, or approved. So think about, you know, a very enthusiastic intern. I think we've heard that a lot at RAPs. That's where the next type of AI agent comes in.

So the deterministic process agent runs in parallel. It enforces your quality system around that generative work. It ensures that every change request, every requirement and every test falls established workflows with proper traceability, human review, and synchronization across tools. So again, that pain point failure of like syncing things across tools, we have that agent now to do that work for us. So while the generative agent creates is creative, the deterministic agent validates and controls.

Together. Bridge the innovation and structure bridge. They bridge innovation and structure. So allowing teams to move fast, but also stay compliant. So think about your left hand agent, the generative agent where if you're looking for something where eighty percent of the time they're right is good enough, then go for your generation.

If you're looking for that six sigma, you know ninety nine percent quality assurance, your deterministic agent is the way to go. Now that you've seen how generative and deterministic agents balance each other, let's talk about how that works in practice. So in the next section, we'll break down how these agents interact step by step to maintain control with within regulated workflows. This is where we move from concept to execution. How agentic AI actually delivers systematic reliable control to day to day use.

Now if anyone on this call is working on an AI agent themselves for their internal tooling team, maybe their their regulatory team, this is where I really like you guys to pay attention. So the so the first prompt, obviously, you're using, you know, your l l LLMs is what the agent what you do to tell the agent what you want. Right? So you can phrase it as like, you know, review all the software application specifications and flag any that need more detail. I know that you know within thirteen forty five there's a requirement for all requirements must be unambiguous, testable, traceable.

You know there's like a very clear checklist for a lot of requirements and sometimes your org has specific checklist items so it can review all of those things. The quality of the prompt really matters, and that's why I think that prompt should have that checklist embedded in the prompt. Think of it as setting the direction. A strong prompt focuses the agent's analysis so it delivers the insights you actually need. And once the prompt is in place, the agent automatically begins the analysis.

Once the prompt is submitted, the agent moves into the analysis phase. Here's what it, this is where it really adds value, right? Is moving, is utilizing all of the hardware, GPUs, the CPUs, and it has that computing capacity that you maybe don't have time for and it's performing that analysis. Of course, you can tailor that analysis. That's something that you can definitely customize depending on how you're building Next comes the recommendation phase.

So this is where the agent turns its analysis into something actionable. Instead of automatically making the changes though. Again, we're working in a regulated space. We're working with that very enthusiastic intern. You want to make sure that the human is in the loop to sort of review those recommended changes.

And the goal here, especially if you're under the AI act is that it needs to be as transparent as possible, right? You don't want you want to really empower the end users with the information they need to make the right decision. So that human review, very, very important. Most important throughout the entire process. AI agents will never take an action on their own.

I'd say we're not at this stage yet for some processes where AI agents should take an action on their own. However, we are starting to approach that and what I encourage everyone on this call this webinar to do is think about, you know, have we gone to the point where we're human reviewing a bunch of things and most of the time it passes that review? Now think about what is the impact of saying, what if we just automated and let the agent make those changes themselves? What is the impact, the risk and benefit? That's potentially where you might be able to even have further savings, but I will admit, like, we're not quite there yet, but I hope that we'll be there with the accuracy of our, you know, models improving.

I hope we'll be there soon. And then the last step, failure to synchronize. Getting it to sync across all of the systems is something that we deeply deeply want. This is something you don't necessarily have out of the box off the shelf with most models, right? It won't necessarily copy and paste directly into your source systems, and that's why I really enjoy using our catch X AI agent because it works within our systems that we already use.

Alrighty, so I just wanted to highlight a case study that we had recently. We actually have another one that's coming out very soon, which I'll talk about in a minute. So this case study. So our example comes from a company that's a global leader in clinical research and digital clinical research digital platform. So they work under GXP, but a lot of their the folks who are coming to audit them are working in that life sciences world.

So they provide infrastructure used by major pharmaceutical, medical, medical device manufacturers and they run compliant clinical trials. Their platform connects hundreds of researchers. Research sites worldwide and supports millions of workflows each month, helping teams collaborate efficiently while keeping studies on track. So think about all the documentation that comes out of clinical trials. Because they operate in a GSP environment, every release end to end needs to be complete, validated, and they have to have the documentation ready for the world's largest manufacturers.

Their QA and RA teams are spending countless of hours manually compiling that validation evidence and generating reports. And so what we were able to do is by implementing a requirements management agent, we were actually able to automate the validation and document generation across different versions of the product. So think about that AI coming in early that requirements management agent coming in early looking at what those new requirements are and making sure that the validation documentation is generated. Again, that's also part of the agents that generate the documents and making sure that validation was complete. So the agent was continuously tracking the changes, linking the related requirements and generating audit ready documentation, turning what is sometimes a multi week process into something that is within the next one or two days, a continuous background process.

So the results are powerful. The documentation time dropped by eighty percent. Change management became a seamless process across multiple systems, and there were things like redundant or conflicting requirements were automatically resolved. That wasn't something, you know, that comes in at the eleven eleven hour. I'm just starting to set myself like, oh, no.

We just realized we actually have we have three duplicate requirement. So they gained both speed and confidence proving that automation can actually make compliance stronger, not weaker, when applied to the right structure. And one thing I will say is that we do have another case study on the way I just was speaking to our AI team around, you know, doing the same thing, but actually with code as your source information. So getting the the code that's coming straight off the presses or straight from Git and generating documentation from that And so Bailey, I'd like to hand this over to you for AI agents and demonstrating that. Wonderful.

Thank you so much, Jen. You explained so much, and it was really relevant, especially me coming from the regulatory space of systems engineer. And I'm sure many people on the call resonated with some of those pain points you brought up. Now I'm gonna walk us through a demo of how this actually gets executed in practice. How can we apply some of these ideas that we've been discussing in a platform like Ketryx?

Before we do that, I'd love to give this poll that actually allows you to interact and decide which use case you are most excited to see. Given time, I'd love to explore all of them, but I'm just curious to know which one is most relevant to the folks on the call today. So I'll just give you all a second to review. And maybe while we're doing the poll, we'll just quickly talk about this question we have coming from Q and A. What are some ways to make sure prompts are clear and specific for regulated work?

So I always like to structure prompts with, you know, here's all the contextual materials and I want you to use these specific links And, you know, try you can always ask the question a way of like, if you are a quality regulatory person, how would you respond to this question? Right? So really providing that context. Bailey, was there anything that you wanted to add there about ways prompts are clear and specific to regulated work? Yeah, absolutely.

So, part of it is what you feed the agent? The prompt is so important. And there is a period where your Ketryx works with your teams to create valued agents, and there is that refining period. That refining and learning period is so important because the agent can learn on your QMS on the data within your system. And that's another key way that we make sure that we're not taking any generalizations.

Every action and recommendation is taken from the data within your system. That ensures that we have a complete audit trail for every recommendation that the agent or AI features within the system are suggesting to you, that we could say, where did this come from? And it'll say, this came from this SOP in your QMS. And that's an example that I'll walk through today. So making sure that all of the agent suggestions are based in real data is key.

Wonderful. So, it looks like we have some responses here, with the highest being producing a final submission document for source material. Of course, if we have some time, we could definitely review some of these other ones, like review SOPs, triaging a complaint, creating a new requirement. But I'll definitely keep it up start with the producing a final submission document from source material. Wonderful.

So now I'm going to actually share my screen in just a second here. I need to get that poll out of the way. Alright. Let me share desktop one. Wonderful.

Are you all able to see my screen okay? Yes. Great. Thanks, Jen. So, for those of you who haven't seen the Ketryx platform before, welcome.

This is how we'll show you how you can execute some of these ideas we've been discussing around agentic workflows. We're mainly going to be in this insulin delivery monolithic system project, which contains our entire SDLC, all the activities that we need to produce the evidence of compliance from design controls to test executions. And you can see here where we're doing that work. In this case, this team is developing and writing requirements directly in Jira and tagging or contributing to that evidence directly in the source code. So we're able to actually integrate directly down to the source code where you may write software item specs, or you may have automated tests that contribute to the documentation that you need to submit to a regulatory body.

Jumping into the insulin delivery system here, we're greeted with the all items screen. You can think of this as your change control board that allows you to see all the changes in your system, no matter where they live. Again, in this case, we're pulling items directly from the Git, so the source code repository, and you can see we're tagging them as software item specs and test cases that live directly in the source code. We also have requirements that live in Jira, and complaints are being managed in Jira as well. And down here, you can see we have some requirements that are acting as risk controls.

With Ketryx configurability, we can configure many different item types to support the way that teams are already working, from requirements all the way to post market activities. You can think of Ketryx as that overarching layer that sits on top of the tools you're using today to ensure we have a connected system. This gives regulatory and quality folks the ability to have clear insight into the activities that are taking place all the way down into the source code and can guarantee that we have compliant and validated workflows as we're working. This ensures that compliance that I mentioned, making sure that items are being moved into a controlled state with Part eleven approvals embedded directly into workflows. And here, you can also see the state of items, so ones that are in progress, resolved, or closed.

I'll quickly jump into Jira, and then I'll generate documents before we jump into AugenTig workflows to really lay the foundation of how Ketryx works with themes today. Jumping into Jira, you can see the few things that Ketryx adds on top of Jira to allow Jira to be a tool that you can manage requirements in, because natively, it doesn't do a great job of that. One of those ways is by embedding this approval workflow directly into the Jira item, as well as this local traceability that allows you to easily understand upstream, downstream, and items that would be impacted if I make a change to this requirement. And this is all interactive, so you can build traceability directly within your source tools as well. The third thing Ketryx does is it helps you build a robust audit trail.

This is extremely important in general, but also when you start to build in agentic workflows, because you need to keep track of every single change that's made to your system. And here you can do that by using this icon over here. We can see the item approvals, so who needs to approve the item, as well as a robust audit trail that shows me every state change that this item has been through, as well as you can see controlled records. In this case, I don't have a controlled record here, but maybe I'll jump to this item. Oops, I'll go back here.

I'll jump to this item in Ketryx and show you what that history looks like when we have a controlled record. Filtering down a controlled records, you could see we have that audit trail. This is clear red lines and green lines that show you exactly what's changed. This ensures that while your teams who prefer to work in their dev tools, who prefer to work in Jira, or prefer to work in the source code, they can do that, while us experts in the quality and regulatory workspace can ensure that we're keeping track of all of those changes and we know exactly what's changing. We also ensure that they're moving it through a workflow that's in line with our compliance or our QMS.

I'll jump straight to our release dashboard and go to this two point zero release that we're working on. In this case, we have a few more controls here that we can embed in our workflow, well as visibility into what's going on in our system. We have some waterfall like activities that we've created in our system because although we want to be as agile as possible, we know that some things are just inherently waterfall, like making sure all of your requirements are gathered and in a control state before you can start your design reveal. And using milestones is one way to enforce that process directly into your workflow so that developers can't start committing code or committing changes or making changes to code while your requirements are still being worked on. Going to the documents for this release, this is all the documents that I need to generate and provide as evidence that we have our SRS here, our design, our test report, traceability matrix.

All the evidence of clients that we need can be generated with just a click of a button. What we find today is that teams are spending a lot of time from code freeze to actually deploying that to production, gathering all of this information, making sure you have zero gaps in your traceability matrix. But with Ketryx, we give you real time view of all of those documents and where you may have gaps in traceability as you're working instead of at the end. Shifting left to these processes really helps you to work agile and quickly. So just like that, we were able to generate all of our documents.

I'll actually open this as a PDF for you and show you exactly what that document looks like, for example. In this case, we just generated our SRS. So I showed you that requirement that lives in Jira. And this is us ingesting that information and showing it to you in a templated document here for you. We have a robust templating language that allows teams or that allows us to template this document so it looks just like the document that teams are generating today.

And we pull in all of that relevant information into this document for you to view and generate with just a click of a button. And what's kind of neat about that, an SRS is a software requirements file. If you have different requirements files, you can actually have different agents review each of those requirements differently. So for software, maybe you care more about error handling. So you might have a reviewer that looks at whether that's built into the requirement, but maybe for hardware, you'd have a different agent.

Absolutely. That's a great use case, Jen. Now that we've laid a foundation for the Ketryx platform, I'll just show you one more key feature here, which is traceability. This is making sure that we have all of our full traceability from requirements down to where they're implemented in the code, as well as tested. And here, you can see that we integrate directly with the source code, automated tests, to show you where you have passed test results or failed test results.

And we want to make sure that we have test executions that are passing. So you can have an agent that reviews that and checks and requires you to all of your tests to be in a past state before you can release this project. This is the And this beat icon allows you to see up, down traceability, where you can have agents that work in between each of these columns here to make sure that you have complete requirements implementing this user requirement and so on. And it can also generate those requirements for you. Let's interact with the assistant now with just a few minutes left to show you how you may do this.

So let's ask it, for example, as a regulatory and quality expert, What? Oh my gosh. Should I ask you. Wonderful. And clearly, you can see here that it doesn't really matter too much my spelling.

But here, you can ask the agent things like, What should I ask you? Or the assistant, Things that I can ask you that are relevant to me in my role. Of course, these are questions or context that's in line with your QMS and traceability. So here, if I expand this, you can see that it's thinking through all of the items in my system and being able to show me things that I should ask the assistant to get a better understanding of what's going on in my product and where I'm at and ensuring that this is ready to get ready for a release date. So this is important questions that you can ask it, you know, without using or having to go to a SME to understand where you're at in this project.

For example, one key thing you could ask the assistant is identify a risk that could be introduced. Hi, let's just use the name here, Katie Too. So I'm going to ask you, taking a risk based approach is extremely important, especially as you're trying to comply to rigorous standards that, or when you're using AI as well, standard like the AI EU Act. What a lot of these guidances say is that you really need to take a risk based approach. And so being able to interact with the assistant that can take a risk based approach that's in line with standards like sixty two thousand three hundred four-fourteen thousand nine hundred seventy one is a way that you can interact with the Assistant and actually have it calculate and suggest where you may have risks based on everything else in your system.

And so then we can go ahead and create a formal risk in our system together. For now, I'm actually going to jump into agents and explore how you can use agents as well to interact with your system. In this case, we're actually gonna jump into this requirement expert quality review agent. This is an agent that is set up to run perhaps daily that looks at all of my requirements and makes sure that, you know, they are robust or they're not missing acceptance criteria. So it's scanning over all of those requirements.

And as someone who previously worked in a requirements management tool like Doors, there could be hundreds of requirements that I have to sort through. And I'm not sure if a requirement already exists. And so being able to use an assistant or an agent that can do that and follow SMART criteria for evaluation, we can actually look at all of our requirements, have it filter down on the ones that are relevant to us, and then review suggestions with the agent. So, you can see that this agent is requesting that we add clear acceptance criteria and measurable requirements directly into our system. And so, of course, I can review this suggestion and keep that human in the loop to decide if I actually want to take action on this item and review it and implement it into my system.

Similarly, we can review some of these other recommendations that it suggested, like edit the field description, make sure that we have a comprehensive suggestion that's in line, again, with our QMS and activities. My Wi Fi has been a bit slow today, so apologies for not that response loading. But while I was thinking, I actually want to interact with the assistant here and ask you some key things about my QMS and SOPs. Because Ketryx has a EDMS feature here, we're able to store all of our documents that are relevant relevant to our product. Like here, you can go into the QMS, and you can see all of our plans, policies, and procedures that are guiding the development of this product.

So I'm gonna ask the assistant, what change procedure should I follow for making changes in my system. And, Bailey, if I can just jump in here, like Yeah. This is something I wish that I had when I started working. Like, that first one you had up where it was like, you know, what would a quality and regulatory expert like, what questions would they ask? Like, training on how to think like a quality and regulatory expert and then now having direct access, like search access to these SOPs, these plan documents, that would save would have saved so many hours of my time.

I had to go chasing people down for those answers. Exactly, Jen. And it's not just about The first part is giving you visibility in the context of what you need to do as an expert. You can just summarize the current plans that you have today. But once you have that information, how do you actually execute it and use it in your system?

That's another challenge. And so because the assistant can read that information, it can then also suggest, here's where you can actually take the action in your system to execute on what you know as an expert, I think that's really powerful. So here I just ask what change procedure should I follow for making changes in my system. In this case, it actually referenced the exact change management plan, so plan one, that lives in my QMS here. So I can actually open this up and show you that plan one actually does dictate what my change management should be.

And so here, I can use this thinking and planning to actually dictate how I should allow a change request to move through my system? Only is this valuable for regulatory experts, but it's also important for software engineers or teams that are working on creating a change request. As an expert, you wanna make sure that they are enabled and empowered with the correct tools to follow that QMS. And with Ketryx, we can give them those guardrails to ensure that they are exactly doing that, that they're following the procedures that we have dictated to ensure the quality of our product. Going back to agents, I just want to call out a few more Evaluating agents that teams are using to ensure the quality of their product, like this change request expert quality review.

We just went over the requirement expert quality review, but there's some other valuable ones here, like the anomaly expert quality review and the complaint expert quality review that even looks at post market activities as well. This is key to, you know, not only writing requirements, understanding what we need to do, but taking it the full circle of your SDLC from writing requirements to observing post market activities. This really puts the power in everybody on your team, from quality experts to developers to test managers, to ensure that we're enforcing a process with agents to help us along the way that create a high quality product, and mostly a product that's safe. Jen, anything else I'm missing here and anything you wanna review with everyone in the last few minutes we have today? No, not.

How are you? I think we can. I think you cut out a little bit here. Sorry. Oh, my bad.

I'll get closer to the microphone. So one of my favorite things about this screen in particular is that you can definitely play around with the different agents that are accessible. So you can keep it to yourself first and try to hone in on, like, what you like about it. And then when you're ready to present it to the team, you can then make it available to everyone. We have a couple of questions coming from the Q and A.

So I think the first one is a very common one we get. So how will Ketryx ensure that the information trained will be confidential? So enterprise set up for only one company. I know on my side, when we're setting up with clients, we have a zero data retention endpoint behind this AI. So everything that's typed into the chat on the right hand side, when we work with our partners, it's not getting retained or used for training.

We actually use Ketryx AI internally, and a lot of our training data actually comes from our internal data. Bailey, anything you wanna add about Yeah, exactly that. It's making sure that the data that has access to is, you know, within your own system. We aren't gathering information outside of the data that's in, you know, your project as well as guidelines from standards like six, two, three, or four. All of that is based in your data and standards and regulations when we're making suggestions using agents or assistance.

And yeah, sorry, go ahead. Any other questions, Jen, that were in the chat? I'm just queuing up a response with the assistant here while we're talking. Yeah. There's a couple.

So what are some best practices to identify, trace, and record machine learning as AI tools train themselves from the data fed? So this is more like when you're building your own product. So specifically from a compliance QMS perspective, the data that you use to train your machine learning model for your product is a design input fundamentally, right? So the data itself and how it's kind of makeup is going to change the output of your design. So you need to incorporate that as a design input.

So you should not only have the data, but you should have a meta analysis on the data. So what is it composed of? How is it segregated? You need to have traced to that level so that you can present like a bias report or justification risk management report that incorporates data biases risk. Absolutely.

Yeah. And actually, one thing I wanted to show the team today that thought was really cool was that you can actually have it, the agents and assistants make suggestions that are in line with rules that you set up. So again, going back to making sure that the output is dictated by some input, and that input can be rules that are dictated by your QMS. So I set up some rules in my QMS that anytime I suggest an item, make sure that it's written with this Hello, Webinar, and explain that it's a part of my QMS guidance. So this is just one way that we can create predictable outcomes with specific inputs.

Is it only for medical device regulations, or can this be used for drugs or biologics also? Yeah, great question. So, we work with teams that are producing many different products from complying to GxP standards, whether they're creating a digital pathology platform or lab developed tests, or a robot. We help teens in all of those different areas. Whenever you're trying to prove that your product works as you say it should, so it meets its intended use, you could benefit from using Ketryx.

What do you usually see as the biggest barrier for companies when they start using Ketryx or a similar tool? So is it getting leadership on board or helping teams trust AI and their quality processes? I know I have an answer for this, but I'm curious what you think. Yeah, It's a great question, and it comes up very often. I think the biggest hurdle we chase now is internal politics.

Introducing change and with innovators who want to use change and who are already using AI, they're doing it quickly and they see the benefit. But you're up against teams who've been doing things a certain way with legacy requirements management tools. And change is hard. And so trying to help them bridge that gap of understanding that AI and tools like Ketryx are innovative tools that aren't meant to be disruptive and are quite flexible. Introducing that and understanding how those specific workflows can map into this is key.

Agreed. I definitely say that the biggest problem is very organization specific. So it really depends on what your organization is, how it's composed, what are the goals, that sort of thing. I know that from the client operations team, when we work with clients, we're always taking, getting a sense of, like, what are the biggest barriers? We're always asking those questions and what are those business objectives you'd like to achieve?

And then we go from there. Right? Wonderful. Well, with just a minute left, I just want to take a second to go back to the slides here and key up our final our next webinar, Jen, if that makes sense. Yeah, so thank you all for joining us here today.

I know there were a bunch of questions in the chat, which I'm really excited to review and see if we can answer those, whether in our FAQ or in a deep dive session with you all later. But upcoming next, our next webinar is again with Jen and I for streamlining multi region submissions. So if you're interested, we would love to see you there. Thank you all so much. Yes, thank you all for attending.

Looking forward to seeing you at the next one.