---
title: "How Flo Is Building Regulatory Readiness, Fast"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/kuh0lbs5hx"
content: auto-caption transcript, proper-noun corrected
---

# How Flo Is Building Regulatory Readiness, Fast

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/kuh0lbs5hx)

---

Hello, everybody. My name is Erez Kaminski. I'm the founder and CEO of Ketryx, and I'm thrilled to be joined today by the Flo Health team to talk about how they've built regulatory readiness while keeping the pace of innovation. Before we dive deep into today's session about how Flo built their regulatory readiness fast, I wanted to share a few quick housekeeping rules. Before we start, we're going to record this and share the slides from today.

There is people answering questions and some of the questions will be answered live. So please feel free to ask questions away in the Q and A section. We welcome them. And then finally, at the end, we're trying to improve how we do webinars. So please, we'd appreciate if you can fill out the feedback survey.

With that said, let's get started. So I wanted to share a little bit of Flo. Flo is the world's leading femtech app trusted by over four hundred and twenty million people globally and is the number one OBGYN recommended app. It's one of the most recognized names in digital health, using AI to deliver personalized insights and improve health outcomes for women worldwide. It was named one of Time Magazine's best innovations of the year.

And I'll just share that, Roman, I'm happy to tell you that this is one of quite a few Ketryx customers that are on this list of Time Magazine's Best Inventions of the Year. It's very exciting for us because we see that what we did is very useful for innovators like your team that are building these world changing apps. I think what's remarkable is how the Flo team achieved this scale while building foundations of trust and safety, evolving from a very fast moving wellness app to a company that's more regulatory ready than ever without slowing down innovation, and you've done that in just ninety days. I believe it was eighty seven days. I think we clocked it together.

I'm in and we measured it. I'll introduce the Flo team in just a second, but I just wanna give a review of what Ketryx and Flo built together and a bit of the Flo solution. So for those here who are new to Ketryx, here's a quick overview of how it works. Ketryx is a way to connect many different team members working across many different functions in many different use cases, using many different systems and many different processes in order to generate evidence of compliance to the most rigorous healthcare and other quality and safety critical regulation, combining them with AI agents and workflows in order to generate the final internal or external submission. People all around the world use Ketryx to accelerate the regulated product delivery, a lot of medical devices, a lot of pharmaceutical apps, and other things of that nature and robots by a hundred x.

We've seen teams go from yearly releases to weekly releases in fast The Flo team met us, I guess, about four months ago, five months ago now. And after a process of selection to find the right vendor for them, an at scale fast moving company, they decided to work with us to help build their regulatory readiness. It took about two weeks from when we started to work together to plan and set up a QMS, build a common language of how we work together and how we develop regulated applications, agree on shared goals, and map their existing development workflows with the processes needed for regulatory performance. We then spent eight weeks of sprints training the Flo team, integrating with the repository's testing solutions, Jira, and other systems they use, connecting into their CICD pipelines to automatically capture their build, test, and deployment evidence and generate a DHF, basically, in a moment's notice. They're able to create traceability matrices from this data directly from their source systems, perform risk analysis, threat modeling, and execute this as part of a typical kind of SaaS and AI workflow.

We got all the design documents configured together, the QMS stood up, and we're able to review and release. So in in basically twelve weeks of time, we went from wanting to become ready to create medical devices to being ready for that. Today, they're able to produce, review, approve all required documentation through Part eleven compliant workflows that are compliant with EU and US regulation, easily verify completeness of traceability before release with the different stakeholders, and have a live QMS. This is a culmination of work of many, many different people, and I couldn't be more excited to share what we've built together. Before we share more about how Flo made this possible, I wanted to start with a few introductions to the team.

And then I'll just go back to set us up on the right slide because I just think there's a great slide to have this in the background as we talk. And so starting off, I want to introduce Roman. Roman is the chief technology officer of Flo Health. Roman has spent over eight years helping grow the company to unicorn status and now leads a two hundred plus global engineering data science and security organization. A three time CTO and head of engineering, he previously led engineering at Payfort, later acquired by Amazon, and held senior act architecture roles at companies including Adform, EPAM Systems.

At Flo, he's driven AI and ML into core products, scaled systems to support hundreds of millions of users, and delivered award winning capabilities like anonymous mode. It's pretty mind blowing just to hear these numbers, Roman. Like, I building apps and building computer systems for a long time, it's hard to imagine how we can build a system scale up for so many users and the whole place is not on fire every day. With Roman is Timefei Savitsky, and Timefei knows that I'm always mispronouncing his name, so he'll forgive me in this time. The chief legal and compliance officer of Flo.

Tim Fei has spent over eight years of Flo and grown from the head of legal and data protection to leading overall compliance across Flo's global groups in the United States, the UK, Netherlands, Cyprus, and Lithuania. Timothy, as you can hear, doesn't really sleep that much. He is just up. Sometimes you get a text from him at five AM eastern, and sometimes it's at five PM eastern. So it's hard to know.

He oversees all legal and privacy functions, driving GDPR compliance and FDA readiness, and has supported the company's growth through multiple funding rounds, including its two hundred million dollars last year led by General and Client. Before joining Flo, he advised international clients on IT, IP and data protection at Cobalt Legal and many law firms through Europe. Welcome, Timefei. Finally, we have Simas with us here. Seamus is director of engineering and leads teams advancing the company's AI and health tech initiatives.

Seamus brings a rare blend of experience across pharmaceuticals, diagnostics, and software, previously serving as a CTO at Ligence where he guided an AI echocardiography, it's so hard to pronounce these things, SMD through FDA pre submission, and as a director of software as a medical device at AstraZeneca AZ, a company that many of us, whether we don't don't know, use all the time. Right? Like, the world's kind of healthcare relies on AZ. And AZ led global teams developing AI powered diagnostic for oncology, and his career span startups and enterprise innovation, uniting a deep sense of technical leadership with a focus on regulatory grade and patient centered software. I couldn't be more excited to have you three, with me in the room today.

So I appreciate your time. I know you're incredibly busy. And I think, Roman, you just came off a webinar a few weeks ago with Sam Altman in Databricks. Right? So you're you're just getting this this fall.

True. Yeah. It's it's a busy time. It's like Techtober and TechNovember in in in the world, and everyone is doing, like, webinars and conferences, and it's a busy time. Yeah.

It's an exciting time, and I think it's kind of one of the best times that digital health have have had. Right? It feels like the companies that really are creating a lot of value are here, and they're now growing and growing faster and finding folks who who need these type of solutions. So, Roman, to start off, for those new to Flo, what does the product do today, and who who does it serve? Yeah.

So Flo is the number one health and fitness app in the world by user base, and we are serving females audience with tailored health insights that we that we build based on the data points that the user provides us. So we collect data from wearable devices like watch or OuraRing, and then we can combine these data points and we can build, like, insights that will be easy for users to understand, and they can use those insights to live a better life. And main features that we have is, like, tracking cycles, managing reproductive health. We have a lot of functionality around, like, trying to conceive, pregnancy tracking. And, yeah, right now, it's a very popular app, and millions and millions of users rely on us to live healthier life.

And right now, we are also very excited about building, like, more features for perimenopause and menopausal users. It's not a big topic for us, but, basically, our goal is to port women's to support women's through entire life starting from, you know, early days up to, like, you know, premenopausal menopause. Yeah. And, Rowan, can I ask how many what's the percentage of women in the UK? I think it's, like, age eighteen to thirty five that use, Flo?

So I think it's about twenty percent of UK users in our, like, audience, like, age group are using Flo in UK. I think very similar number in US as well. So our penetration in, like, developed countries are very high, but we're also kind of, like, very popular in the rest of the world. We are number one house and fitness app in India probably as well. And in many countries, our app is free of charge.

So, basically, we believe that in some countries, we need to give our application for free. And, like, in many countries in Africa, in India, in Ukraine, our app is available for free. And, basically, the idea here is that, like, we want to invest into health education, and we want to empower women around the world even if they don't have enough money to pay for for a subscription. So, yeah, we believe in building, like, better future for female health through through our app for entire world, not just UK and US. Yeah.

So it's it's basically, like, one in five women that are I think it's, like, age eighteen to thirty five, you probably know better than me, something like that, in the United States is a user of Flo. Exactly. Wow. It's, it's just an impossible to imagine how a startup that was not here, you know, fifteen years ago was suddenly just supporting so many people in, you know, one of the most critical tasks of life, right, like giving life. Let me ask you, Roman.

What value did you wanna deliver to users that required this type of regulatory read readiness? So for a long period of time, we were more like a wellness app, and we were like, from day one, we we invested a lot into medical safety accuracy. We have a medical board that kind of oversee all the future development within the Flo app. We have, like, in house doctors and so on, but, like, we were more like a like a digital version of the book. We had a lot of, like, health insights and many things that don't require, like, medical regulation and FDA approval.

But as we're kind of progressing and we're building more and more features, we see that we need to go deeper into health. We need to go deeper into medical space. And for that, in order to be safe to make sure that it's all safe, it's all compliant, we need to get FDA compliance, and we need to get we need to make sure that we're clinically great. We have, like, clinical great capabilities, like symptoms evaluations, like risk assessments, different, like, health insights that goes beyond just, like, general information. And, basically, our goal is to be essential health partner for all women around the world, and that is why we decided to invest more into this space and to kind of go through this exercise to obtain, like, FDA compliance for some of our features that we are building as we speak.

Yeah. And and when we met, you know, one of the things we talked about is what's the the type of impact this would have on the engineering team you have. Right? Because it is a very fast moving kind of consumer tech company. What what how do you expect that to happen?

What do you expect the the kind of impact to be, on your engineers? Yeah. So it's it's true. Like, we're relatively small company in comparison to AstraZeneca or, like, Apple. Apple we are competing with Apple.

Like, they have Apple Health app. They have some functionalities that allows users to track cycles and so on. So, like, we are competing with big companies, but we are relatively small company. And in order to be competitive, we need to move really fast. We need to work smart, not hard because of limitations that we have.

It's not a small team, but, like, it's not the biggest one in the world. And we were worried that, like, by introducing, like, regulations, we will slow down innovation. We will slow down feature development. And today, for example, we are having, like, four hundred AB tests concurrent AB tests that we are running inside the app. So, like, we already kind of used to deliver functionality very quickly.

We have, like, these agile processes. We kind of build features really fast without compromising on quality. For sure, we have, like, many quality controls in place related to medical safety, related to privacy and security. We are ISO compliant for privacy and security. But it's still not the same as being, like, a medical device.

It's not the same as being, like, FDA approved app. And we were worried that by introducing, like, QMS, we will slow down and engineering will be impacted by this kind of, like, bigger, let's say, regulations. But, yeah, I'm not don't want to spoil it, but, like, our worries were kind of we're worried for for nothing, basically. Yeah. Again, it affected us, but not as significantly as we expected.

Yeah. And I think that's very common both for people who are coming from outside of the, like, the medical grade, medical device, you know, GXP validated industries into it because you hear the stories of how it's done today, and it is very burdensome, very expensive, very slow compared to what you'd expect software to be. And so I think that, you know, I, when I first joined the industry, heard some of the statistics of release times. I was like, is that a thousand times slower? Like, some of this, I I just can't really I don't understand why this is like that.

So I'm happy to hear that was your experience. And, Timothy, from your perspective, what does regulatory mean for Flo Health at this stage? Yeah. I think it it's it's a great question. First of all, hi, everyone.

Never had a chance to say hi, so happy to participate today. I think it's a it's a it's a difficult and easy question in the same time. I would say that for us, the regulatory readiness is something that I think resonates with a lot of the listeners or watchers of this of this webinar. First of all, building basic processes that are inherently built in the operations of the company. So, essentially, a fully operational QMS that would be embedded into operations naturally, not as an add on, not as a, like, sort of, like, like, a thing that that exists there just for the sake of existing, but very natural continuation of any work that we're doing at Flo.

And so that we can ensure that the development processes, the the quality process, they all performed in a smooth and very efficient manner. And I think, like, at this stage, regulatory readiness for us also means being smart about it, of course, as Roman pointed out. As as as as we explained and and it's actually a reality of many companies. We have a portfolio of products. Some of the products are wellness products.

Some of the products are more regulated. And, of course, you wanna your QMS to be smart and kind of adjustable and adaptable to ensure that you follow the stringest possible process, but in a inefficient fashion for the parts that do require regulations, like the ones that are medical devices, and you ensure that you continue working according to your very common and well established processes for the rest of the organization. I think a lot of organizations actually don't get it right. They they put QMS, everything is suddenly under QMS, and they you don't you don't differentiate between nonmedical and and medical devices, and therefore, you slow down the organization overall. Yeah.

And I think, like, first and foremost, patients don't want that. Right? Patient wants innovation. And I think the regulators agree with that. They want the least burdensome approach.

And many times, there is a feeling that some companies, because of how hard it is, they end up having the most burdensome approach. And, really, these practices need to be part of just developing high quality products, and ideally, all these artifacts just need to be generated from the work we're already doing. Simas, let me kind of move on to you. So I know that Flo is moving very, very fast. How often do most teams at Flo release today, and how do you think of experimentation with those releases?

Thanks. It's a good question and a pleasure to be here, Erez. Yes. I think the audience got the hint. We do move very fast, and we are proud of it.

I don't think there's any other way when you're building a globally competitive b to c app. So, you know, successful product, yes, it requires speed, quality, and how features are built and and how we communicate them. So Roman mentioned we can concurrently run hundreds of AB tests. And in a year, we're we execute more than a thousand AB tests. We release daily and yes.

Yeah. Exactly. Wow. That was also my reaction when I joined Flo. And we you know, this experimentation is really at the heart of how we develop the product.

And when we think about, yeah, compliance, for example, we in the industry, perhaps, people tend to think that, well, you have to choose between the speed and agility and and the compliance. I think in Flo, we believe that we can achieve both. And, obviously, that's that's kind of the the approach that we were trying to implement here. Yeah. And it's exciting.

You're talking about an amount of releases that is almost the amount of total releases in the medical device industry. Right? Like, it's just a vast amount of of work. And I think it's it's again, it's vast in in this critical regulated demand, but, like, this is what people are doing in b to c and b to b f all the time. Right?

Like, Netflix, right, famously releases every sixty minutes. Right? So I know me and you have spent a lot of time talking about AI and AI in medicine and what that means, how PCCPs fit in, Nick, how is the infrastructure and regulatory kind of framework getting ready for AI? And so I wanted to ask you, what's the role of AI in your product? So in product itself, AI plays a big role.

You our users I mean, I don't know if our users know, but AI underpins one of the key features of the app, which is the cycle predictions. We use machine learning models to predict cycle lengths on how the menstrual cycle will look like. At the same time, AI is not just part of the product. It's also a big part of how we work. So it's a productivity enhancement tool.

Engineers use AI to write code, to write documentation. It's a tool for content generation and content review. So AI judges, for example. So overall, AI is a big element in the of the product that we develop and how we work, and it's clear that it's not going anywhere. So we need to find ways how we can maintain this agility, how we can maintain the speed while having a an AI element being at the at the core of the product.

Yeah. And I think a lot of medical companies all over the world today are trying to figure that out. You know, recent, information published by FDA in the first half of twenty twenty five, in one month, there's been more AI submissions than in the previous in a year just a few years ago. I'm not talking about ten years ago. I'm talking here maybe three or four years ago.

Right? There was a hundred AI products. And now, I think this year to date, to the information that's by the end of May, there's hundreds already. So it's very, very exciting, the shift. And Rowan, with this pace, the level of experimentation, right, four hundred concurrent AB testing, a thousand a year, what's the hardest part of introducing QMS?

Was it more of the cultural, the process, the tooling? How are you thinking about it? Yeah. I don't think it's a culture. Cultural wise, we were there already, like, many years ago.

We were big believers in medical safety, medical accuracy. We had, like, a lot of processes. And I think even our engineers, they already followed, like, a lot of best practices that I expected even, like, by, like, FDA. But we still had to adjust processes, so it's it's impossible to become, like, a regulated to become, like, a regulated without adjusting processes. And I think another big part was tooling.

Right now, there are very few tools that kind of built for modern era. Like, we were looking for something like, you know, as as agile and as good as GitHub. You know? Like, every engineer works with GitHub and knows how to work with GitHub. But if you check the market for tooling, it's all built, like, I don't know, many, many years ago, or it feels like it was built, like, many, many years ago before Git was introduced by to the world.

So I think process and tooling were the biggest thing for us, and the goal was to introduce new processes and new tools without slowing down the engineers, without kind of, like, affecting productivity of our engineers. So we were very cautious about how we chose the proper tools and how we introduce all these new processes to to the workflow of our engineers. And then we had to think about, like, what to put under QMS and what to put outside of QMS. Like, who should work under QMS and who should work outside of QMS. Our app is quite big.

We support, like, many different life stages. We have, like, a community where which is basically a social feature where users kind of talk to each other, and it's not a medical device. It shouldn't be regulated. You still want to kind of keep innovating in this kind of social area as you were before. And then, you know, like, there are other parts of the app where we need, like, QMS and we need, like, let's say, more structure.

We had to think about this kind of carefully, and we put on the we decided to put on these those kind of, like, medical features under QMS. So it's kind of was a clever way to keep some of our engineers outside of QMS, let's say. And I think it's a very good idea to kind of think about this this way and to kind of, like, split things and to define, like, which system should be under QMS and which are better to keep outside. Yeah. And I remember when when we first met and then Simas joined maybe our second call, we were starting talking, like, what we need to do here is a multifunction device.

Right? A device that have medical components and nondevice components. The problem is how do you actually create that differentiation while you're working so fast and have an agent that kind of sits in the background and both structurally and statistically checks that that's true. Right? That we're not touching the things we shouldn't be touching.

But on the other hand, innovating in the place we wanna innovate. Right? Like, patients are not, you know, there's parts of this that that is not in their benefit for us to increase the burden for. Right? And so it's really important to be very focused.

And I always think about how even companies at your scale, right, that are so fast moving, well funded, there's just so much so such limited resources because there's so much you could be doing for folks. And then how do you focus it? And I think, you know, when we talk, there was this strategic decision in Flo to build regulatory readiness because you see it unlocking pretty massive opportunities for growth and product differentiation. Right? And, I always tell folks that, like, obviously, clinical products have a lot of, value because they're clinical.

Right? They actually treat something. The moment you get into that treatment diagnose, the moment it becomes a medical device. It's even I think any person can understand that when you treat something, it has a lot of value, right? Because it actually impacts you.

And we were fortunate enough that after, a long evaluation process, you selected us to be your partner and Ketryx was the way to make this vision possible for you. So, Simifei, I wanted to ask, what did you expect from the organization when you started down this path to meet your compliance goals? What did you stand up first and how long did it take to get it going? Our our key goal is was always has always been to be fast and furious. Let me put it like that.

And I think, yeah, we actually our key goal was just to to to build on what I've said is just to ensure that we get the foundations very, very fast because, like, foundations are important, like, critical. Like, building a proper QMS, and I don't wanna repeat what Roman said because I fully agree with that. Like, you need to really map what the QMS covers and what what it doesn't. Otherwise, you risk slowing down the whole organization. We managed to do it very, very fast with with Ketryx, essentially, as as you Erez mentioned, like, within eighty something days.

Of course, it was a huge commitment from you guys and from our from our side just to see it and really put a lot of effort into building all the processes, making sure that we help have all SOPs embedded in in different parts of the organization, that they make sense, that they implement it. There are a lot of things, for example, that are related to risk management that need to be embedded in the in the formal cycle. Risk management could have been a, like, a part of the culture for for years, but for quality management system, it's something that should be formalized in a bit different fashion. Therefore, we really appreciated the speed of of implementation, and it was, like, probably the biggest the biggest and the most critical thing for us. And from my side, I would also say that I was always a bit afraid of of of of of anything kind of regulatory related for one specific reason.

I need to hire fifteen regulatory people to create documentation, processes, technical file, blah blah blah. It could be, like, I don't know, tons of pages of documentation at the end of the day. And I can surely say that probably I don't need that many people at at this point because I do feel like with with this tooling, it's of course, you always need to have a a man in the middle, essentially. Someone who would check about the date, do some manual work as well. But overall, it decreases the heat on on the head count and, essentially, how many people you need to hire.

It keeps you agile, but but you don't need that many people to to drive some of the typical regulatory things that you can observe in other organizations. Yeah. And and I gotta tell you, Tomfei, that a lot of our work together brought together this vision that we've worked on for almost a decade, four years as a company, many years ahead of that planning this as innovators trying to figure out what if it doesn't need to be that complicated. Right? What if you don't need armies of people to do all this work?

Because again, the patients really benefit from the documentation being, you know, made in an ivory tower and all these other things, right, when you can actually make them very rigorously. And I think one of the great things we did here together in partnership with you all and with Siemens especially as part of this implementation is tying together the world of the clinical, the design architecture, and the regulatory in order to set great boundaries for a multifunction device, for a system of subsystems, and how to make sure we're we're providing the right level of risk and risk management for each part. CMS, you've been one of the most amazing partners we work with just the depthness of knowledge that you provided and your deep understanding of software of unknown providence and components and how it all works together. And so I thought it'd be great to to hear from you a little bit. How did you think of designing the architecture to ensure that you can move quickly and meet all the regulatory requirements?

Like, what were the guiding principles for you? Yes. I I do remember those times quite well, diving into the specifics and trying to almost, like, with a surgical precision cut through what is the device and what is not the device. I think my my reflection on this is perhaps engineers make very good quality people because they're able to understand, well, the engineering world and also the the regulatory constraints. And in this case, one of the guiding principles is has been basically, well, see what what the regulatory guidance even tells us.

So regulators have always been very upfront that they do expect software interoperability. They do recognize the principle of system of systems. In fact, that this is sort of where the world is moving. A lot of interconnected software modules, software systems working together. So in in radiology, this I feel like this has been very well executed, and there's a lot of precedent.

In women's health, not so many, but we are trying to lead the way. So, yeah, so one of the principles is, well, let's see what the regulators even think about this. And in fact, they do encourage this approach. The other important element is, well, what is the existing solution? Flo has very interesting and unique challenge, I feel.

We already have a great product. So we have a complex product. It's a super app. It contains multiple modes. There's a lot of functionality.

And so maybe, not such an experienced team could be thinking, well, do we turn the whole app into a medical device? Maybe we rebuild the app. And a very experienced engineering team, together with mature, quality function and a right compliance partner, can look at this problem and see, well, how can we kind of slice through the system and isolate the function that we believe is medical device that truly serves the medical device function and keep the rest of the app as nondevice. So these I would say these are, like, the key principles. See what the regulators say, have the right competencies, so right quality management system, right compliance partner, and also strong engineering, to find the boundary between the device and nondevice.

And finally, maybe one point I want to mention, know, both Roman and and Timothy shared that while we were interested in, first of all, assigning teams that work in QMS and teams that don't work in QMS. If you remember, Erez, one of the goals that we set, we said we set a a goal that we want no decrease in developer satisfaction, in those developers, yes, that work in QMS versus those that don't. And we also want to see no decrease in speed in the teams that work in QMS and teams that don't work in QMS. So, our our hypothesis and goal still remains. Even if we slice through the product and we isolate the device and nondevice function, we don't want to see basically any difference in how the teams work in terms of their satisfaction and in terms of the speed.

It's really about, like, the mindset and the problems that they solve, but but not in terms of, like, friction introduced and anything like that. Yes. It's a lot about understanding what the regulations are and where you stand, and then understanding in a risk based approach which parts of your system are, in fact, the device, which are not, what are high risk parts of the device, what are not, and then kind of rightsizing it for them. Because I I agree with you. It is the easiest thing is just everything is a medic is it everything is a class three.

And there is some simpleness to that, but there is a a huge cost to it. And one of my favorite quotes from our our VP of regulatory who wrote most of the US regulation, Paul Jones, and with FDA for a long time, he always says, you know that the when he started as a system engineer in the seventies and sixties, he said, you know that the requirement traceability matrix is a r and d artifact. It is never used by quality and compliance. It was something that r and d leaders used to say, you know, did we implement everything? And over time, it ballooned into the something that is challenging, but I I couldn't agree more.

Engineers make the best quality professionals because they're already thinking in this way. And if you read six to three or four, you realize that it's basically just saying, can you just explain how you conduct best in class engineering practices and just have the amount of documentation that you feel is reasonable for that? It's not really asking for a lot more than that. And and this brings me into this discussion, Roman, about how to think about making this transition. I meet engineering leaders every day that talk to me about both in kind of big co med tech and life science and also in emerging companies, growth companies that are trying to become that.

They're realizing the product innovation and and particularly regulated product innovation is the only thing that matters in today's market. Right? They're just so it's so competitive. It's competitive in drugs and devices and consumer apps and b two b apps. And so the speed of product innovation is what really matters.

And I wanted to ask you when you when you think of talking to other leaders who are making this transition, building regulatory readiness on top of an unregulated starting point, what should they worry about? What does it mean for engineers and development timelines? You know, are we worried about kind of less time spending coding, more documentation, delayed releases. I think those are a lot of the topics. And, what were your must have as a solution so engineers didn't need to slow down?

Great question. Yeah. You always want to keep your engineer focused on building new features and new value for end users. Because in this way, you create more value for end users. You create more value for for the company as well and kind of it creates, like, a flywheel of innovation.

You kind of you build more features. You earn more money from that. You kind of reinvest this money into building even more value for for end users and solving even more problems. So, basically, we want to keep engineers focused on that because they know how to do that. And if you ask them to do, like, to manage spreadsheets or, like, fill some checkbox kind of exercises, they will quickly become bored, and the churn will be high, like, the productivity will be low because they basically not not used to do that.

It's, like, not in the engineering nature to do these manual steps and to manage spreadsheets. And in ideal world, you need to automate as much as possible, automate documentation, and, basically, the idea here is to reduce amount of human errors and free up people time to do a creative job. Once you automate something, it becomes, like, repeatable, and it becomes less error prone. And, basically, sometimes you can even you know, people, they apply, like, common sense. They apply, like, different things to optimize things, and sometimes it's kind of good thing, but it's also might be not very good thing because, basically, they can skip some of the steps.

They can introduce, like, some mistakes here and there while copy pasting something from one spreadsheet to another. They may forget to paste or, like, to do some other silly mistakes. It's a human nature to do those mistakes. But once this process are automated, once all this compliance checks, they automate it. Once you have all these kind of things in code, it becomes repeatable.

It becomes, like, less error prone. So my suggestion is to kind of keep engineer focused on building software and automate as much as possible. And how do you think about CICD in that, Roman? Yeah. So, well, CICD is very important.

It's very important to kind of, like, have it kind of, like, integrated into all the processes that you have so you can capture all the artifacts that are created by CICD and kind of, like, you need to have this integration between Jira. You need to have integration into CICD, into your compliance system, into whatever processes you have. It's kind of like it's all goes through the CICD pipeline, and then you can use, like, a lot of things like pull request and so on to kind of make sure that all the code changes, they go through certain processes and you have, like, a special checks in place. But I think even better, Simos can talk about this because I think he's doing this, like, day to day. So he probably feels this pain even, you know, even better than I do.

Yeah. And, Simas, I'll I'll also add to this. I know that, like, you know, in AI, CICD is it's hard to explain how inherent it is to the idea of having a machine learning model, the fact that it needs to update frequency frequently. So how do you think about that? Yes.

So first of all, as an engineer, I I share a romance sentiment that engineers love automating things. That's for sure. And, one thing that engineers don't like is is, yeah, exactly repeating manual tasks. We do have a strong safety view for the regulated part of the, you know, of the development. It does involve automatic reloading tests, s bombs, lot of artifacts into CAT checks.

And we did explore the role of, like, how can AI assist us in this process. We did some prototyping that you may be familiar with. Erez, you and your team, where we looked at, can we use AI to, for example, analyze the PR, find the relevant item in Captrix? Could we even, let's say, automatically keep our design history file up to date as, like, a reflection of the code changes that engineers are making? But I think at the end of the day, still, there is, a human in the loop that that ensures the process is running, and the same thing applies to whether it's software or whether it's AI models that we are updating.

At the end of the day, it's still, you know, same CICD pipeline running for the whole feature that we are developing. Maybe some specific parts are unique to the software, some specific parts are unique to the AI. But at the end of the day, it still adds up to the same collection of the artifacts. And at the end of the day, there's still, like, a person reviewing it. So whether you're developing software, whether you're training an AI model, the process is is relatively the same.

And and, really, the focus here is about how can you automate as many steps of this process so that people so the most valuable resource of of an engineer is, of course, the, you know, the intellect. So the person can then direct intellect at reviewing the outputs that were produced and not so much about manually copying information from one place to another. I think we are now at the stage in our in our world and also with solutions like yours where, you know, manual copying of information just to keep the design history file up to date is is just becomes somewhat ridiculous, I think. Yeah. And and it's something I talk lot about and then happy for folks to participate and and give some thoughts.

But, you know, we we have partners that every release of their product has ten to twenty thousand pages of evidence. And when they wanted to move from a year release to a week release, you realize that they're talking about making between five hundred thousand to a million pages a year for one product. Like, this is an immense amount of information, and and it's not even possible for a person to analyze so much information. That's why when we started working on this neural symbolic engine that combines symbolic algebra with an LLM to help people execute these processes and bring it to people, to me, it was very clear that there this combination of the rule based computer algebra system, the AI LLM based system, and the person in their tools reviewing this work and guiding these tools and and making any decision. Right?

Because that's inherent to safety critical system is that person making a decision. The that's the only feasible path forward for companies to get to validated CICD. And I think it's it's something that people throw around, you know, we'll just do CICD on the medical device, but it requires an immense amount of resources and the right set of tooling and knowledge because it combines these three really distinct areas of, like, clinical, regulatory, and software and AI engineering, which it's it's really hard. All of them are like, you know, PhD level subjects to understand and then synthesize together. And I think with AI, Seamus, right, this this issue of frequently updating makes it very, very hard to generate so much evidence.

Like when your product has AI and you have so much data coming in and you need to update it often. Right? Like, there's no you know, any good machine learning person would tell you, like, updating once a year or even once a quarter kind of misses the point a little bit. Like, the whole point is that you're getting more data and you should update it all the time. And, you know, in the very data hungry companies, right, they will fight tooth and nail to be able to update it as fast as possible.

Right? And kind of if the machine learning team is not fighting for that, you need to question why why is that. Right? Like, that's, the the AI pirates I know are really into quickly updating things and retraining. How do you all think of of thinking a little bit about the journey of making compliant, getting the team compliant, thinking of making it just so easy for them?

I don't know if folks wanna share some thoughts on that. So, from my perspective, I think it's very important to start kind of early and to kind of not to wait to have this like urgent need to become, like, regulation ready. It's always good to start introducing, like, good processes, best practices, engineering best practices as early as possible. And then it will be possible to do, like, QMS in ninety days as Flo did because, basically, you need, like you still need, like, best engineering practices. You need CICD.

You need Git. You need tools integrated that speak to each other. You need Jira with Jira tickets. You need all this kind of, like, basic stuff. Maybe it's not Jira, maybe some other tool, like, whatever at least engineering are using, but, basically, you need a proper process anyway.

It's the only way how you can run a successful engineering company. And, also, like, think how to introduce QMS so that your engineers will not feel it. You know? It's very important to kind of introduce compliance in a way that it kind of happens in the background, and it happens automatically. So engineers can still be focused on building, like, best possible software for for customers, for users.

Yeah. And a lot of this is just commonly used practices. Right? Like, it's like this question of a root cause for bugs and making sure, change requests are tested. Right?

Like, this is what commonly great engineering teams are doing anyway. Now to make it regulatory is just about, like, making it be able to be documented in a way that can be reviewed and assured, which is also very important. Right? We all know that if you don't do that, it's hard to maintain it for a long time, right, if there's no accountability. And I know we're we're kind of getting to ten minutes before the hour, so we're starting to wrap up.

One thing is for the crowd, we'd love to get questions as folks wanna ask us some questions. We have a few more things we can go through, but always happy to take some questions here for the team. And as we do that, I I keep thinking of how do we look ahead and reflect on on what we learned together through this journey. I know that we learned a lot about how great a team can be when they're so decisive on doing this and both folks who are doing the work on the ground level, the leadership is just deciding this is what we're doing now. The commitment from you all to being kind of compliant and doing things in a way that's patient first and safety first has been really amazing for us to see.

Just that was, I think, from the very first conversation, Roman, it was like, we're trying to build a safe compliant medical device that meets everything it needs to meet and also doesn't stifle innovation for us. Erez, how how does one do that with you all? And that was kind of how this discussion started, and it was very, very lovely that we never left that line. And maybe with that, looking back over the last few months, I wanna ask you, Roman, what actually changed in the day to day of the engineers since you started pursuing regulatory readiness? And how much of your release process is now automated?

Yeah. So it hasn't changed almost like at all, and I mean this in a really good way because, basically, we are still working the same way as we worked with the same speed and agility, but now in a much more compliant and much more regulated way. And most of our, like, regulation pro compliance related processes are automated, and they just kind of as I said before, they just happens in the background. And I think it's really good. Yeah.

Wonderful. And, Simas, maybe you you wanna share a little bit of how do you think of the visibility and control that you need. I think so Roman has definitely said the right things, but there's one thing that I'm not really sure I agree. I would think that engineering teams working in the quality management system perhaps even gotten faster just just because we are building now a regulated feature and engineers are excited about this. Of course, it's difficult to compare, but in in my mind, yes, I do I do agree that the teams that were affected by this move to the QMS, they definitely did not slow down, but I would say maybe quite the contrary.

Now they also feel like they're working on something very serious. They also feel like they're building, you know, definitely this medical, you know, medical direction product. And it does serve as a you know, it's it's an inspiration, basically. It's it's why some people now push the extra mile because they want to be part of the first medical release. So overall, I would say definitely, visibility is there, control is there, and we've got that extra boost of motivation.

And and we're kind of proud that we are building something that is, you know, going towards the medical device. And I think building medical devices, right, it it just means you're building a product that can treat, diagnose, and modify the human body. Like, it can actually do something clinical to people, and then it's awesome. Right? Like, you know, you meet so many health care innovators early on in their journey that ended up doing something different than a device or something that's less patient facing.

And it's always the same story. Well, we first thought of building a device because it's kind of the most obvious thing that you go to a doctor's office and you're like, you know, I wish there was an easier way to get diagnosed for skin tumors. Right? And then they realize how hard it could be. And if it was just less burdensome, they would try to do it more often because it is very valuable and it feels great.

Like, I love working on medical devices. I think that especially when you meet patients and we were fortunate enough to work with so many different partners that I was in the hospital with my family recently and we were connected to a device that we're working on, like, in the hospital and it's pretty amazing that feeling. Right? And so we have a lot of questions now, folks, if it's okay to to just start with some questions from the crowd. One question here, and I think this is for you, Simas, but for the team is someone asked, when you mentioned releasing once a day, do you specifically mean releasing a change in your medical device function or actually releasing something that might be nonmedical device because of the way you set up your subsystems or both?

Yep. Yes. So it's it's a good question. Indeed, in Flo, we release multiple times a day. I was referring overall release schedule across the whole app.

Indeed, in the medical device case, we do not release once a day. It depends, of course, on on the things that we are building. Technically, we could be releasing once a day. But because of the complexity of the feature that we are developing, yes, we we do not release once a day. But that though what matters is you could once you get there.

Right? Yes. Yes. We could. If if we were building something simpler than what than what we are actually building, then, yes, releasing once a day would be possible.

However, the engineering problems that we are solving, they they do require, yeah, they do require, like, longer sprints and yeah. Really basically, a release schedule needs to reflect, like, the kind of product that you're developing and the kind of features that you're shipping. Another question here here is, how does the team here see the FDA PCCP guidance predetermined change control plan and machine learning operations together hand in hand. I'll just say for folks who don't know, PCCP, we have a lot of material on the website or the FDA website, is a way to preapprove change management for your device based on a set of metrics right now around machine learning models. But in the near future, anything in a device would be, you could change this except high risk components of a PMA according to a guidance FDA released about a year and a half ago.

But it basically kind of opens the door to making very fast moving changes. And how do you do you all think that ties into MLOps together? I I can I can just briefly probably say a few words from the regulatory perspective, and then guys can add on more technical level? I do feel it's a very helpful instrument, and I'm pleased actually to see that FDA really thinks of of of of such products. And it's it's truly a gesture of of innovation, I would say, to allow such instruments to be implemented, submitted, and approved by FDA.

And I do feel it opens a big opportunity for a lot of companies like Flo to explore various products, AI enabled products that could potentially evolve change over time without no no need with no need to sort of submit every single version or new product to to FDA. And to be honest, I think if PCCPs can you can be used broadly, not even for AI companies, but also for some other variable elements of of a product. And we also have some ideas in place at Flo, how we can leverage this framework for for our own developments. But, yeah, long story short, I think it's it's a very helpful element of of of of today's regulatory framework in the states. Yeah.

And I'll leave us with one last question. Just a quick round table from you all as we close on is, if you're advising someone in your role at another digital health or medical device company to start this journey, what key piece of advice would you give them? Maybe starting with with Roman? My advice will be, to treat, regulation as a growth enabler and something that allows you to deliver more value to end customers and kind of makes you special and you kind of you can compete with other companies because of that. And if you do it right, it becomes your competitive advantage.

Because, basically, if you're in a regulated environment and you can release two times faster than your competitor, you will be twice more successful as a business as well. So and you deliver much more value to to to the users so they will love your product as well much more. So my advice is just think about this as an about enabler. Okay. Timothy?

Do rely on compliance by design. I really like compliance by design here at Flo, and we are, for example, ISO twenty seven zero one certified and and things like that. So the best compliance is the one that embedded in the design of the organization and different systems that allows people to kind of almost never observe this this rigid requirements, but still follow them because the systems are built in this particular fashion. And, of course, reducing manual labor as much as possible. Think that's that's another one.

Cool. Simas, any last thoughts to share? Yeah. I would say, yes. Automate as much as possible so that engineers can focus on building things, rather than manually documenting things after they've built it.

Yeah. I couldn't agree more. Well, gentlemen, thank you so much for the time and thank you everybody who listened to this conversation. It's so delightful to be here together after this meeting kind of not too long ago and planning this and executing together. So I wanted to thank you all so much for your partnership and your support and your desire to build things for patients.

And I look forward to many years of working together. Thank you for the time. Thank you. You. Thank you very much.

Bye Thank you.