Now on Now: How we increase developer productivity with GenAI and DevOps

Hey, hello, and welcome to Knowledge24. We're happy that you chose to listen to our session. This will be a session where we share with you what's new, what we have learned about developer experience, developer productivity, and how we're building an internal development platform in 2024. I hope some of you were able to attend the Knowledge23 session where we outlined the foundational, the ideas we put into and this chat is a continuation of that, what we have learned in the past year. How many of you have developer experience on the platform? All right, then you're at the right theater. My name is Andrei Shpakov. I have been with ServiceNow for three years. I'm on a team that's responsible for developer productivity, and that's ServiceNow's digital technologies, DevOps is being part of that as well. And, with me, I have Tom. - I'm Tom Abraham, technical product manager of developer experience, also been at ServiceNow about three years, yep. - The agenda, we have about 20 minutes. We also realize that we stand between you and the events right after, the welcome reception, so we'll try to make this talk entertaining and also informative. This talk is also part of the Now on Now series where an internal team within ServiceNow, we talk about our experience, how we use the platform, how we innovate on the platform, and how we also give feedback to our product teams so they can build the next release better with some features that we have developed. So about the agenda, we'll first revisit our philosophy, what do we think works best for developers and their experience in the platform. We'll talk about challenges we saw when we looked at the day in the life for developers. We did value stream mapping exercise for everything our developers do from the inception, work assignment, development, testing, also deployment. There were a few findings that we are able to share. Also, we'll talk about where we are right now, where we were last year, and where we want to go with GenAI as well. We'll finish up with some deep dives into Test Assist, how does it improve code quality for us? And also our DevOps tool that we call DevNow CI/CD. And we'll finish off with some key takeaways. All right, so to kick off any developer experience initiative, we want to start somewhere, and this quote summarizes it well. We want our developers' life to be frictionless, so I often use the metaphor of a death by a thousand paper cuts. It's painful. Some of the pain can be removed one by one paper cut, and we want to do that. To simplify processes, tooling, and approaches, how little they are as well. We're excited about GenAI. And nothing can remove pain of repetitive tasks better than machines themselves. So GenAI, we're very excited about that. And we're already seeing some evidence internally, when we talk to our customers, we see some evidence as well. It's a game changer for us. What do we do as software engineering organization? We deliver value to our customers by delivering software. And we want to build better software and build it faster. So we have tools, we have processes to help smooth day-to-day activities, and increase productivity also using GenAI now. So what does developer productivity mean for us? Let's talk about our scale. We have 400 developers on the team, on digital technologies organization. We have hundreds of apps, we have dozens of instances, live instances. So even the smallest incremental change translates into huge time savings for our organization. And we go about it by optimizing our workflows, processes, and environments as well, to drive the higher throughput and stability to our instances. And I would like to emphasize also, it's about balanced improvements in all of those. We don't want to focus only on quality, so speed suffers, or prioritize speed, then the quality suffers. So we want to make balanced incremental improvements in everything. So how we set up our work, we roughly divide every developer experience effort into three lanes. So our Enable track enables our development community with knowledge, it's everything with developer boot camps, education, brown bag lunches, so we can build the relevant skills, especially GenAI skills right now. Simplify is all about our flow and shortening the feedback loops into the flow. So we, as developers, we try many things, and then see if it validates our assumptions or invalidates our assumptions. So the shorter the feedback loop, the better, and the faster we get the value. And Equip is about building the best tooling available for our development community, and we'll do a deep dive into a couple of those. So when we look at developer productivity, we also look at it through the multidimensional lens, inexperience, speed, and decisions landscape. We also built the four main groups of developments, DevX solutions that are using GenAI. So what does experience mean? It's how developers interact with our systems. That's what they see, that's what they type into, experience should be good. What does speed mean? Time-saving techniques, removing those paper cuts, removing those roadblocks or speed bumps. What does decisions mean? It helps the developers with pointers, recommendations, order-generated metadata and code, that's all about that. All of those phases are important as well. This is another emphasis on the balanced. Not only people, products, technology, but also experience, speed, and decisions. We also have a partner session, another session that focuses primarily on Code Assist and Code Scan. I encourage you to join that session. It will be on Thursday. All right, so to get started, let's look at the day in the life of our developers, the value stream mapping exercise, Tom. - Thanks, Andrei, so now that we've covered our approach to developer productivity in GenAI, let's look at some specifics at some of the tools we are developing having increased developer productivity. Here are the daily tasks that a developer needs to complete to do their daily job. Overlaid, we can see all of the tools that we are giving them to help, and you can see GenAI is being used in some of those tools as well. The overall goal is that we get through this entire process as quickly as possible, and we want them to do it as many times. So repeating it multiple times quickly is more important than doing it once slowly. So how do we improve? So we looked at our cycle time. And, on average, it was taking over two weeks for our developers to get their development tasks completed. We also had only 65% of our deployments fully automated. Meaning there are many cases where a pipeline wasn't being used for production deploy, because of deployment step needing manual intervention. So we identified two areas that we believed that would make the most impact in improvement. The first is manual testing bottlenecks, and the second is deployment delays. So to fix the bottlenecks, we have with testing, we identified it's time consuming to create tests, they're often being done manually, and our tests aren't covering all of the changes that we are making to our applications. For delays in our deployments, we identified manual steps in our deployment pipeline, and we found we needed flexibility in how it is configured. As we can see, we improved our actual cycle time by four days. That's four days that a developer has back into the actual coding tasks they're doing, instead of the toil or tracking down somebody needing to approve something or manually deploy. We also a saw significant improvement in our automated releases. So 73% is now fully being automated. There's no actual manual step that's being done in the actual production deploy. So now let's look at some specific things that we did for testing with Test Assist. So Test Assist is our AI solution that generates tests primarily using ATF. So it generates three different types of tests. We generate unit tests, which are tests at the code level, functional tests that are manually being done by a PO or some other QA, and ATF test cases, taking those functional tests and automatically running them. Let's quickly look at what some of our unit tests would look like. So you can see that there's a code file on the left here, and this is auto-generated test code using Jasmine framework in ATF. And so you can see that we are, you know, describing each of the functions in our test, and then putting expected inputs and outputs. So this is getting down to the actual code level, you know, lowest level tests that we can make. And here's an example of a functional test that we created just from the description and acceptance criteria of our stories. And generative AI is just automatically creating all of these test cases that would be done manually. And, finally, from these manual test cases, we are now automatically creating ATF test steps. So from those manual tests, we start on a, you know, at least a set of test steps that can be executed automatically. So that's how we're looking to improve our testing coverage. But let's look closer at our deployment process with DevNow. So what is DevNow? DevNow is our custom app that manages our build and deploy process heavily focused on our taking feedback from our actual developers, as well as other DevOps tools that we've seen in the industry. We focused on self-service, making everything self-service for onboarding, as well as everything that they're doing in their day-to-day work. Reduced the manual steps as much as possible, and made as much as we can observable, and that's from like a development environment or just where the developers are actually working. As expected for any CI/CD solution, our quality and security checks are done automatically and our testing is running after every commit. And a key thing that we focused on is we were using other third-party tools previously, like Jenkins, to automate part of our deployment. We are now doing everything within a custom ServiceNow application entirely. We're using parts of the actual product, like the GitHub spoke, we're using flows to actually do a lot of the orchestration and integrating directly with Agile and change management. But, again, it's all housed within the platform itself. And now let's take a quick look at what some of these actual experiences look like. So code promotion is simple and configured specific to each application. The environment or instances to deploy to are simply selected, and a set of automated steps will automatically run as part of the deployment. So you can see a deploy update set may run, as well as running automation or a test specific to a specific environment. Or when you get to stage, a change request may be created automatically and running other quality checks. We make all of the automated steps visible in our development environment and in our Git repositories, so developers no longer need to jump between instances and URLs to get the information they need. And we automated more steps for the actual release of change requests. So if there are additional steps needed, like running a script or importing XML, we're able to do that based off of these change tasks, and we are doing those in those pre-deployment or post-deployment steps. All of this has combined and led us to a better experience in the day-to-day of our developers. - All right, let's look at some numbers. The value that we are gaining by using these two solutions, just these two solutions. So we looked at time savings of the environment promotions, so from development through the testing, through staging into production, it's about 2 1/2 hours that we give our developers back. Increased frequency of SubPROD deployments. That goes, again, into the fast feedback loops. We give developers the easy way to validate or invalidate their assumptions by deploying into SubPROD so they can check on those instances, what works and what doesn't. If it's easy to do, if they can do it faster, that results in a better code. As for the Test Assist, we have seen since the three months that it has been running in our environment, we have seen more than 2,000 net-new unit test cases created, which are not trivial to do. And developers sometimes prioritize the actual code development over the test development, so our solutions help with that, give them the foundation and something to work with. Not to start with the blank slate, but get something from GenAI and improve it over time. And so, essentially, 700+ script include methods were covered by unit test cases in these past three months, which is a good number we think. - And let's take a quick look at some key takeaways. So one of the key takeaways is improving developer productivity is not just a technical solution. So the example of our DevNow solution, it replaced the previous legacy solution, but it was also a change in our overall process, not just the actual tool. Second GenAI tools, such as Test Assist, it's helping with the time-intensive tasks. So saving a little bit of time on bootstrapping that test is just, you know, it's still value add even if the actual test generated is not perfect. And, finally, our DevOps and orchestration solutions are tying everything together, so these are really making the impact for our developers. So it's not just having Test Assist, but also running that test and kind of bringing it all together within one pipeline automatically. - All right, that's it. So we are happy to answer any questions next to the bar. And we also have a booth, Now on Now, when we can show you the demos, how it works in real life, and we can answer any follow-up questions if you have. And with that, thank you very much. (audience clapping)