The Enterprise AI Playbook: Strategies for Scaling AI in Quality Engineering | Testμ 2025

Testμ Conference 2025
63

Honestly, this question came up in the panel too, and the way the speakers put it really clicked for me. When it comes to UAT and use-case testing anything that involves real human behaviour AI can help a lot, but it’s not something you hand the entire responsibility over to.

Think of AI as your amplifier, not your replacement. It’s great for doing the heavy lifting: generating tons of realistic scenarios, replaying user sessions, analysing patterns, even simulating user traffic to see how your product holds up. That part, AI does brilliantly and at scale.

But the moment you step into areas like “Does this flow feel natural?”, “Will a user actually trust this step?”, or “Is this accessible and comfortable for everyone?” that’s where humans still matter the most. No model today fully understands emotion, trust, frustration, or those tiny UX decisions that shape a real user’s experience.

So the takeaway for me was this: Use AI to speed up and expand what you can test, but keep humans at the center for subjective judgement and the real-world nuance UAT demands. It’s a partnership AI handles the scale, humans handle the sense.

64

Absolutely, I was lucky enough to attend the panel session at TestMu 2025, where Dror Avrilingi, Janakiraman Jayachandran, Khimanand Upreti, Mobin Thomas, and Vikul Gupta shared some fantastic insights on scaling AI in Quality Engineering. One of the most interesting points that really stuck with me was the discussion on the top pain points that enterprises face when trying to sell these “magic” AI tools.

To sum it up, the main roadblocks are:

  1. Poor Data Quality/Availability: Many enterprises struggle with the quality and availability of their data. Without good data, it’s tough to make these tools work effectively, and that’s a huge hurdle for teams trying to scale AI.
  2. Lack of Integration Standards: Another major challenge is the absence of clear integration standards. Whether it’s APIs or data contracts, businesses often find it difficult to get these AI tools to seamlessly integrate with their existing systems. This lack of standardization makes things unnecessarily complicated.
  3. Cultural Resistance and Governance Concerns: Lastly, there’s a cultural shift that needs to happen. People in enterprises are often hesitant about AI, especially when it comes to governance. There’s a fear of losing control, which makes organizations slow to embrace AI fully.

What’s interesting is that these challenges tend to hold things back far more than issues with model accuracy. So, it’s not always about how smart the AI is, but how well it fits into the enterprise’s existing systems and culture.

These insights are definitely things to keep in mind when you’re looking to scale AI in any enterprise environment!

65

When it comes to measuring AI’s success in testing, it’s not just about how much faster or more efficiently we can work. Sure, productivity is important, but there are other critical areas we need to keep an eye on to ensure AI is adding real value. We need to track things like ethical compliance and bias reduction. For example, are we seeing a reduction in biased outcomes? Are we ensuring that our models are auditable?

Security is also a big one AI can help us prevent incidents, but we need to measure how effective it is at keeping our systems secure. At the same time, we should be looking at how AI impacts customer experience are we seeing fewer defects that directly affect customers? We also need to think about the bigger picture: things like uptime and customer satisfaction (think NPS scores).

Finally, let’s not forget about model governance. Is our AI drifting off course over time? Are we making sure we can explain why certain decisions are being made by the system?

These are all important metrics to track if we want to make sure AI is working for us, not just in terms of productivity, but in building trust, security, and long-term value.

66

Absolutely! I had the chance to attend the panel session at TestMu 2025, and there was a great discussion about how enterprises can successfully scale GenAI in Quality Engineering (QE). One key takeaway from the session was how important it is to carefully select the right use cases when starting with GenAI, especially in QE projects.

The panelists, including Dror Avrilingi, Janakiraman Jayachandran, Khimanand Upreti, Mobin Thomas, and Vikul Gupta, really emphasized the importance of starting small and low-risk. It’s best to begin with tasks that are high-frequency but low-risk, like detecting flaky tests, automating test case generation for stable modules, and streamlining the triage process for test results. These are processes that tend to occur often and are critical, but they don’t have as much of an immediate impact on your most sensitive areas.

The goal is to prove measurable ROI early on with these smaller wins. Once you’ve demonstrated success with these use cases, it becomes a lot easier to justify scaling AI to more complex, higher-stakes scenarios. By getting those quick wins under your belt, you also build trust within the organization that AI can truly make a difference in improving QE workflows.

This approach ensures that enterprises don’t rush into high-risk applications without first proving the value of GenAI, setting a strong foundation for long-term success. It’s all about being strategic, starting small, and scaling thoughtfully.

67

Absolutely! During the TestMu 2025 session, we had some great insights on aligning AI initiatives with tangible business outcomes. Here’s a breakdown of what was shared on how to make it happen:

The key is to define your success metrics right from the start. Think about the metrics that matter most for your business – like reducing incidents, cutting down cycle time, or driving cost savings. Once you’ve got those in place, it’s important to track them regularly. Setting up your pipelines to measure these KPIs is essential – this way, you’re not just guessing how things are going, but actually seeing the impact in real-time.

Another important part of the process is having monthly business reviews. Get the right people in the room – the owners of each metric – and hold them accountable for the progress. This way, you’re always on top of how the AI initiatives are driving real business value.

It’s all about setting clear expectations, tracking progress, and staying on top of results. That way, AI doesn’t just stay a tech buzzword, but becomes a key part of hitting your business goals.

68

During the TestMu 2025 session, Dror Avrilingi, Janakiraman Jayachandran, Khimanand Upreti, Mobin Thomas, and Vikul Gupta gave some great insights into scaling AI in Quality Engineering, and I wanted to share my take on one of the questions that really resonated with me: “When should you scale AI in testing, and how long does it usually take to get there?”

Here’s the thing: It really depends on having the right data and strong support from sponsors. Based on what the panel discussed, you can generally expect about 6 to 12 months to get your AI system operationalized and up and running in a meaningful way. The real tipping point, though, is when you start seeing AI insights directly influencing product decisions not just the analytics side of things. That’s when AI shifts from being a nice-to-have to an integral part of your testing process.

So, if you’re already at the point where AI is guiding key decisions in your product development or quality strategy, then it’s time to think about scaling. The sooner that happens, the sooner you’ll see the real benefits of AI at scale in your testing workflows.

69

When it comes to scaling AI beyond the pilot phase, one of the biggest mistakes companies often make is overlooking operational readiness. It’s easy to get caught up in the excitement of a successful pilot, but without solid groundwork in place, those pilots will just stay as experiments and never transition into real, scalable solutions.

The key here is to invest time and effort into things like defining clear SLAs (Service Level Agreements), having rollback plans ready in case something goes wrong, keeping an eye on costs to avoid any surprises, and conducting thorough security reviews to ensure everything is safe. And, perhaps most importantly, it’s crucial to establish clear ownership across teams so everyone knows their role in the process.

When you get all of this right upfront, it’s much easier to scale and see real, sustained success beyond just testing. This foundational work is often the game-changer that sets the stage for long-term AI adoption in your company.

70

Absolutely, I was at TestMu 2025 where we had an amazing session with Dror, Janakiraman, Khimanand, Mobin, and Vikul discussing “The Enterprise AI Playbook: Strategies for Scaling AI in Quality Engineering”. One of the key topics that came up was finding the right balance between human testers and AI-driven testing.

From what I gathered, the general idea is to start off a bit more conservative, with humans doing the heavy lifting think around 70–80% of the work. This helps build trust in the AI systems while still having strong human oversight. As you begin to feel more confident in the AI’s capabilities, you can gradually move towards a 50/50 split, especially for areas that are lower risk.

However, it’s crucial to remember that humans should always be the ones making those critical decisions and giving the final sign-off. The human touch is irreplaceable, especially when it comes to judgement calls that require a deeper level of context or expertise. So, the trick is to ease into AI testing while keeping humans in charge where it counts most.

71

Absolutely! I was fortunate enough to attend the TestMu 2025 session on “The Enterprise AI Playbook: Strategies for Scaling AI in Quality Engineering,” and I found the insights shared by Dror Avrilingi, Janakiraman Jayachandran, Khimanand Upreti, Mobin Thomas, and Vikul Gupta really eye-opening.

When the topic came up about which skills are becoming essential for data scientists with the growth of AI and automation, the panelists pointed out some pretty key areas. First off, it’s no longer just about the basics of data science or building models. MLOps, which is all about deploying, monitoring, and managing machine learning models in production, is crucial. This is where data scientists need to focus on maintaining and improving models once they’re live, which is a whole different ballgame compared to just building them.

Another skill that’s becoming more important is observability. It’s not just about making sure your model works; you need to keep track of its performance and behavior in real-time to spot any issues or opportunities for improvement.

Data engineering is another big one. Data scientists can no longer just focus on analysis or modeling. They also need to understand how to handle and prepare data for models at scale, which can be complex with so much data coming from various sources.

The panel also mentioned domain fluency. Understanding the specific industry or application you’re working in is more important than ever. A solid grasp of the problem you’re trying to solve ensures that the AI models you create are not only technically sound but also practically useful.

Lastly, explainability is something that can’t be overlooked. As AI becomes more integrated into business decision-making, being able to explain why a model made a certain prediction or recommendation is critical especially when you’re dealing with enterprise-level decisions.

To wrap it up, the panelists emphasized that while raw modeling expertise is still valuable, skills like MLOps, observability, and understanding the business domain will be what truly sets top data scientists apart as AI continues to scale. It’s about making sure the models are reliable, explainable, and aligned with real-world needs.

72

Absolutely! During the session at TestMu 2025, when the panelists shared their experiences with adopting AI in the SDLC/STLC, they highlighted some real challenges that many of us can relate to.

One of the main issues that came up was dealing with brittle pipelines. This is something I’ve seen firsthand when you’re integrating AI, things can often break down unexpectedly, especially with the complexity of new tools and processes. Another issue that the panelists mentioned was noisy outputs. When you’re relying on AI, it’s easy to get overwhelmed by the sheer volume of data or results that may not always be relevant, making it tough to identify what’s actually useful. And of course, there’s the problem of unclear ownership. In a large enterprise, figuring out who is responsible for what, especially when AI is involved, can get confusing.

But here’s the good part: they didn’t just stop at pointing out the problems. They also shared how they tackled these issues. The solution they proposed and one that really resonated with me was to deploy AI incrementally. Instead of doing a huge, all-at-once rollout, it’s better to take smaller, manageable steps. This way, any issues that pop up can be handled without a major disruption. They also built adapters to make sure the new AI tools could integrate smoothly with existing systems, reducing friction.

Another key takeaway was governance establishing clear rules and processes around AI use. This ensures there’s no ambiguity around ownership or accountability. Finally, they emphasized the importance of automating validation pipelines. By having clear owners and SLAs (Service Level Agreements) in place, they made sure that everything runs smoothly, and the team can track progress easily.

So, if you’re working on integrating AI into your testing or development cycles, these steps really do make a difference. It’s all about starting small, staying organized, and making sure you have the right processes in place to support growth.

73

I was actually at the session where Dror, Janakiraman, Khimanand, Mobin, and Vikul shared some really interesting insights on AI in Quality Engineering, and this question about whether AI will replace current automation testing tools came up. From what I gathered, it’s highly unlikely that AI will completely replace the automation tools we use today, like Playwright, Selenium, and other test runners.

Instead, AI is more of a powerful addition to these frameworks. It helps by generating tests, prioritizing tasks, and triaging results basically streamlining the process and making it more efficient. But the core toolchains and orchestration systems we rely on will still be around for the foreseeable future.

AI will definitely enhance how we work, but it won’t be taking over the entire automation process anytime soon. It’s all about collaboration between AI and our existing tools, making the testing process smarter and faster!

74

Absolutely! One of the key takeaways from the session with Dror Avrilingi, Janakiraman Jayachandran, Khimanand Upreti, Mobin Thomas, and Vikul Gupta at TestMu 2025 was the idea of treating AI as a service. The rapid pace at which AI solutions evolve means that the model you develop today can quickly become outdated, sometimes even within a month. So, what’s the solution? It’s about constant monitoring and iteration.

First, monitor your models for drift meaning, you want to track how your AI’s performance may change over time, especially when exposed to new data or environments. This way, you can detect any performance drops before they become a big issue.

Next, automate triggers for retraining. Instead of manually revisiting your models every time, set up automated processes that prompt retraining whenever it’s necessary, based on performance or new data inputs.

It’s also really important to version your models. Keep track of every iteration this helps you understand what works and what doesn’t, and gives you a clear picture of how the model is evolving. Plus, make sure you’re implementing small, incremental updates to avoid big shifts that could potentially break your solution.

And don’t forget the importance of ownership. Assign product owners to your AI models and set up Service Level Objectives (SLOs) to measure their performance. Think of it as managing a product if you let models run without ownership or clear goals, they might quickly turn into unlabeled experiments. Keeping them properly managed ensures they remain useful, reliable, and up to date.

In short, treat your AI like a living, breathing part of your ecosystem. Regular updates, constant monitoring, and clear ownership can help you keep your solutions on track and ahead of the curve.

75

When you’re scaling AI in Quality Engineering, there are a few key signs and metrics you should keep an eye on to really know it’s working. From what I gathered during the panel session, the first big indicator is seeing fewer production incidents – basically, the number of issues making it to production should go down. Next, you’ll want to track the defect escape rate. A decrease here means fewer bugs are slipping through the cracks and making it into the final product.

Another thing to look for is how much your team is actually adopting the AI suggestions. If people start using those AI-driven insights more often, it’s a sign that the AI is becoming an integral part of your testing process. Stability is also crucial – you want your models to stay consistent, with minimal drift, so the predictions remain reliable over time.

Of course, there’s the ROI – if you’re seeing a positive return, then you’re on the right track. But it’s not just about numbers; you also need to look at the bigger picture. For instance, how quickly are you resolving issues (Mean Time to Resolution, or MTTR)? And are customers noticing a difference in the quality of your product? Positive changes in these areas can really highlight the impact AI is having on your overall business.

These are just a few of the key metrics to monitor as you scale AI in your QE efforts, but they give you a solid foundation to gauge success!

76

Absolutely, I can totally relate to the concern raised here. Implementing AI in a project can feel like it’s piling on more work for the associates, especially when they’re dealing with all the extra reviews and decision-making that come with it. But here’s the thing—we can definitely mitigate that.

First off, it’s all about improving the signal-to-noise ratio. Essentially, you want to make sure that the reviews only focus on the important stuff, rather than getting bogged down by irrelevant noise. One way to do this is by setting up thresholds and ranking mechanisms, so only the truly critical items make it through for human review.

Another practical step is automating the sanity filters. By doing this, we can filter out the simple, low-risk issues that don’t require manual intervention, allowing the associates to focus on more high-impact areas.

Also, using confidence scores is a game-changer here. These scores help us route the right issues for human review based on how confident the AI is in its predictions. If the confidence level is low, we can automatically flag it for review, but if the AI is highly confident, we can just let it handle things autonomously.

And finally, always be tuning and refining the models. AI isn’t a set-it-and-forget-it solution it needs to keep getting better. By continuously tuning the models, we can reduce those pesky false positives, making the whole system more efficient and less overwhelming for the team.

The key takeaway here is that AI should help lighten the load, not add to it. By fine-tuning the way we use it, we can make sure it’s actually taking tasks off the associates’ plates, not piling on more.

77

When it comes to validating AI-generated test cases, there are a few key steps you can take to make sure they’re reliable. First off, start with some basic static checks—think of it like a quick inspection to ensure everything’s in order. This could be things like linting or static analysis to catch obvious errors early.

Next, I recommend running those tests in a safe environment like a sandbox before you throw them into production. It’s like doing a dry-run to see how they perform without the risk. You can also compare these AI-generated test cases against your human-written ones to see how well they cover different scenarios. If the AI is covering as much (or more) as your manual tests, you’re on the right track.

But don’t stop there! It’s important to have a “gate” in place meaning, you don’t let the tests go live unless they pass a certain threshold. It could be a simple pass/fail system, or you could take it a step further and involve some human sampling to make sure everything checks out.

Finally, as time goes on, keep an eye on how these tests are performing. Are they catching bugs consistently? Or are they a bit flaky? Monitoring the catch-rate and flakiness over time gives you that peace of mind that your AI-generated tests are reliable and improving.

So, it’s all about a blend of automation, human oversight, and continuous monitoring to make sure AI can really step up to the plate when it comes to testing.

78

During the panel at Testmu 2025, we discussed some important lessons learned from AI initiatives that didn’t go as planned. A lot of the failures we see come down to a few key issues: weak data strategy, lack of integration with existing operations, misalignment of KPIs, and, of course, resistance to change within the team.

What I personally took away from the discussion was this: Don’t wait until you have the perfect AI model to start thinking about the bigger picture. It’s crucial to focus on the groundwork early on things like owning your data, establishing solid governance, making sure your outcomes are measurable, and setting up a strong technical foundation (aka the “plumbing”). When these elements are in place, you’re more likely to get the results you’re aiming for, rather than just chasing after model performance metrics.

So, while models and AI algorithms are exciting, it’s the operational and strategic setup that often determines success or failure.