Eisenberg: Hi, I'm Kate Eisenberg, and welcome to this special mini podcast series. So for this third episode, we're focusing on our Distinguished Engineers that we have with us today. So as I just described it to a colleague, the heart of our technical team behind Dyna AI. They've been absolutely instrumental in both getting us going with this project and, you know, continuing the forward momentum as we go. So I'm going to let them introduce themselves starting with Erica, going to Omkar, and then we'll get started.

Lesyshyn: Thank you. Kate. So as you mentioned, I'm a Distinguished Engineer focusing on the search and discovery platform for Clinical Decisions. And I became involved in this initiative after it became clear that we were going to productionize this.

Kulkarni: My name is Omkar Kulkarni. I'm a Distinguished Software Engineer, just like Erica. I specialized in search and machine learning AI, and I have been involved in this Dyna AI project since its inception. What initially drew me to this project was a strong interest in exploring how generative AI could be leveraged to deliver the knowledge more effectively to our end users or customers.vAnd that goal has been the driving force behind the work on this initiative.

Eisenberg: I'm just going to follow up on that, actually, because I feel the same way, that I think the generative AI is a technology that is allowing us to do what we were trying to do anyway better and faster. Is there a technical problem that, you know, you really wrestled with more than other, you know, work you might have day to day, or something that was kind of the most satisfying once you were able to address it. Does anything stand out?

Kulkarni: Yeah, definitely. I mean, one of the biggest challenges in this Dyna AI project was especially the latency. You know, generative AI responses, especially for the broad queries, can take a long time and users often expect quick answers. And to improve the response and overall UI/UX experience, we implemented streaming. This was a bit challenging, but significantly enhanced the user experience and was a satisfying achievement. One more challenge that I can think of was accurately understanding and categorizing the user intent, you know, specifically in medical domain, as you know, where precision is very, very critical. At the same time, you also need to prevent the hallucinations. So designing a system to handle this complexity was tough. But again, we as a team ultimately did that, and, you know, it was very rewarding.

Eisenberg: Erica, how about you? Same question. Does, you know, does any particular challenge stand out to you?

Lesyshyn: Yeah, I think for me, one of the biggest challenges was achieving a high degree of clinical quality. When you think about, historically, the way that the engineers and the clinical staff have worked together, you would typically see that perhaps engineering would work on something and then have regular touchpoints with clinicians. But in this case, we really needed a deep synergy between the technical staff and the clinicians. And so since we're not clinicians, the engineers, it was refreshing for us to have this constant iterative collaboration on a daily basis.

Eisenberg: So, Erica, when you think about the arc of the work here, how would you feel like the team has navigated that tension between getting that clinical quality where we want it to be and moving quickly? Omkar mentioned speed of responses, but speed of development is important here too.

Lesyshyn: Yeah, so I think this was a significant cultural shift. It was almost an opportunity for us to let go of the working styles that we had in the past and come together in a different way. I think luckily the Search and Discovery team had already had some close relationships and partnerships with clinicians. So it wasn't a typical team where you had to go through storming or norming or forming. A lot of those pieces had already been established over the years. And I think one of the challenges that we faced from a technical standpoint was receiving feedback in a uniform fashion. We had some clinicians that were very technical and some that were non-technical. And so it wasn't until we really came forward with an automated testing framework and a ticketing system to capture feedback where we were able to move quickly and kind of balance how feedback was getting captured and how we were prioritizing and working against those initiatives.

Eisenberg: That was a real maturation step. I think when we formalized that system. Omkar, how about you? How do you feel about kind of the clinical quality piece versus speed or, you know, maybe they're one in the same.

Kulkarni: Yeah, I think I completely align with Erica, what she said. I mean, clinical quality has always been a critical concern, you know, especially due to the risk of hallucinations. And in medical domain, precision is non-negotiable. We must ensure that, you know, users always receive accurate, reliable answers, right? And delivering high quality, you know, is very essential but it also introduces the complexity that could slow down this development. And to manage this, we collaborated, again, closely with our subject matter experts. We set some clear expectations, and we leveraged some framework like Ragas, you know, a testing tool to keep us in line. And this allows us, allowed us to strike the right balance between, you know, the maintaining development speed and also ensuring that, you know, we achieve a higher clinical accuracy and quality.

Eisenberg: I feel like that's something I've learned from the two of you is that complexity needs to be a real consideration. There's a lot of things we could do, but if they're too complicated, we may not want to or we may not want to maintain them. So I think that's been an important technical perspective for me. Can you think of something where you steered us in a direction that you're like, yes, that the you know, I'm really glad we did that.

Lesyshyn: Yeah. When I first joined the team and we started productionizing and moving this forward, one of the critical pieces I felt strongly towards was ensuring that we had a solid architectural design to support both rapid prototyping, but also quickly migrating that to production quality code. And so it was kind of finding that balance between, you know, how much code are you willing to throw away until you get to something that you're comfortable with, versus, you know, spending too much time up front. So I felt strongly about getting that architecture in place that's more plug and play oriented, that allows us to do some rapid prototyping and then quickly be able to migrate that and then move forward. So I think that for me, the foundation upon which we worked was really critical.

Eisenberg: Omkar, how about you? What, you know, what are you most proud of when you think about all the work you've put in or all the decisions you've recommended we make?

Kulkarni: Yeah, I think the architectural foundation is really, really important, but when I think about one of the features that we implemented, I'm particularly proud of implementing follow up questions. You know, after reviewing many real customer queries, it became clear that users don't always phrase their questions clearly or completely. And since Dyna AI is fundamentally a knowledge delivery system, accurately understanding the user intent is crucial for generating the right response. And follow up questions help refine that intent, you know, allowing us to deliver more precise and relevant answers. And this feature significantly improved the system's effectiveness and leading its implementation was one of the highlights for me.

Eisenberg: Very nice. I remember that, you know, there was a good period of time where you were really, really it was not an easy problem as an impression. And you really kept at it. And I remember when we started, you said this is going to be a tricky one. And then you came out with it and, you know, that was really successful.

Kulkarni: Yeah.

Eisenberg: So, you've both alluded to this a bit, but I'm really interested in hearing your thoughts on how did this team feel different than past projects? I know for me, this was a different type of experience. And I'm interested, from each of you, and we can start with Omkar about, you know, how has that experience been for you? And, you know, how did it feel different from your other work or other teams or projects you've been on?

Kulkarni: Yeah, I think Erica has already touched upon this particular aspect. And, I think what sets this Dyna AI project apart from other projects is a Tiger team approach. And the team brought together product managers, developers, clinical SMEs and business stakeholders into one tightly integrated unit, you know, and this structure enabled fast and direct communication and quick decision making, you know. Compared to my past projects that I've worked on, this collaborative and cross-functional set up significantly made a difference in both how we approached it with terms of effectiveness as well as speed.

Eisenberg: Absolutely. And, Erica, same to you. What has the experience been like for you?

Lesyshyn: Yeah, I would echo Omkar as well. I think having that solid framework in combination with the team structure allows us to move very quickly. We were able to move the traditional barriers that you would see associated with communication styles or with technical limitations. So I think it created that Goldilocks zone for us to move very quickly and pivot and change direction.

Eisenberg: All right Erica, I'm going to go to you again. So, getting a little bit cheesy here, but what's something that you saw Omkar do during the project that really impressed you about the approach he took?

Lesyshyn: So I think, Omkar, that we've probably been working together for about eight years. And it’s Omkar’s dedication to following an idea over the long haul. It really is a marathon in his mind. He has been focusing on this initiative for many years, and the technology really wasn't there. But boy, once it achieved that level, he, you know, jumped on it. And so I think that trait is something that I've always admired and appreciated and saw throughout the whole project. And even today, you know, on a weekly basis, you see him continuing to revisit things that maybe weren't where they needed to be in the past and continuing to push, despite setbacks or despite being told, you know what? We're not ready for this. So it's that passion to continue and that curiosity that I just admire so much in him.

Eisenberg: You know, it's interesting because that really echoes back to what you said, Omkar, at the beginning, that this technology let you do what you were trying to do anyway better and faster, and that sounds like what you're reflecting here.

Kulkarni: Well, I don't think I'll have enough time and words to explain that in this particular meeting about Erica. But, you know what really impressed me about this project was how effectively it was managed. And, promoting, you know, a fail fast approach and mindset during the research spikes and, at the same time, continuously pushing for innovation, you know. And under her guidance, the team stayed ahead of the curve. That level of organization and forward thinking, you know, made a really big impact on the project success. And that's really important. And, I'm so much impressed with Erica on everything that she has done for this particular project.

Lesyshyn: Thank you. Omkar.

Eisenberg: Amazing. And, I mean, I would obviously echo that as well. I think, Erica, your steady hand every day, you know, in guiding and shaping the work, it's really added up to something special.

Kulkarni: I really cannot imagine the success of this project without Erica’s leadership.

Eisenberg: All right, and we're going to go to each of you before we finish up. Omkar, what's one word you would use to describe this journey?

Kulkarni: I would say it's trailblazing.

Eisenberg: Very nice. And Erica, how about you?

Lesyshyn: Fun. I think we just had so much fun. So much fun, right? It really feels like you're just in a playground and just having a blast. So, yeah, it was a lot of fun, and it still is.

Kulkarni: That was more apt, by the way.

Eisenberg: I think everyone has said a different word that I've asked, actually. Thank you both so much. I'm sure you have much work to do, but I appreciate your time, your insights you're sharing with us today.

Lesyshyn: Well, thank you so much. And we appreciate the opportunity.

Eisenberg: Sure, this was fun.