[Music] welcome everybody wherever you're at around the world um if you want to drop a note in the chat of where you're from we'd love to see where that where that is as we get started today we're going to be talking about the automated test framework and some of the updates that are coming through with that uh today our our guest speaker speaker is yinka oini and we'll hopefully be joined by John Lynn my name is Adam Stout and we all work on the outbound team uh outbound product management team covering the ATF and platform in general as we're going through today uh we may have a few forward-looking statements so I want to make sure we've all read the safe Harper statement please don't make any purchasing decisions based um on things that are not yet released and what we're talking about our focus is on what's coming out or what came out in Washington um and before uh but in case we do let anything slip out please don't make any purchasing decisions based on that if you're interested in what we're talking about today and other things related there are several other webinars in our Academy series and live live at service now um go ahead and grab this uh this QR code or the or the form you registered on in the community and you can get access to all of those uh this session is being recorded and we'll be posting it on YouTube and into the community uh probably in a week or so um but as we're going through today we do like this to be interactive so please feel free to uh drop your questions in the Q&A panel so we can make sure we answer them or any comments you have uh feel free to put those in the chat as well and with that I'm going to hand it over to yinka to give us an update on ATF thank you Adam um let me share my screen right now um hi everyone um good morning to you from wherever you are right now um so today's goal will be the we're going to understand what's new in ATF and um how we can improve your testing strategy as the case may be and we're going to go over some of the changes that happened within ATF and um some of the new features we adding to ATF in Washington DC um like I said our agenda today is we're going to go over the update then we do um a couple of demos and we will also share some of the additional resources that available to you for ATF um if you want to catch up on some of the previous um videos or previous Academy on on ATF you can use the link down here which shows um the the ATF test generator and Cloud this was the academy we we recorded during when we launched um ATF test generator and Cloud so this is more like um a build up on this particular um Series so um I'm I'm just going to go over test generator again for some of the um new customers or new users of ATF or new users on service now that don't know about the the feature called test generator now um as you can see on my screen right now what we have for test generator is an intelligent tool that generates ATF test by analyzing your instance behavior and when you see instance Behavior this is a tool that goes over your table and uses the last six months the last 6 months data you on your table to generate um different test based on what you can understand basically you're saying oh do you want to validate um some inut or do you want to create some input so that's what the test generator does for you and um one interesting thing about this is that it doesn't come with the with the family release it's available on store so what that means is that you don't have to wait for your next upgrade or for for a new family release before you have test generator on your instance and one interesting fact about this also is that for you to run the test generator and Cloud you need to it can be executed on the service now um Cloud infrastructure so when let's say for example you want to schedule a test or you want to run a test overnight when you you're not with your device you can always schedule um this test on cloud rner what that means is that you can run your test without having your device open uh um I'm going to go a bit into test generator some of the interesting so the test generator and Cloud are two two different um feature in one in the sense that the test generator is is run on on top of the cloud the cloud is an infrastructure on its own so I'll talk about test generator bit just to give us some some insight um The First Time released this was on Tokyo patch 5 so for some of the old customers you know about this and you can also attest to how we've able to improve on our first version of test generator we've added a couple of um interesting updates which I'm going to share later on for the folks that um probably stopped using it the first time we released it um I mentioned Also earlier that this is an intelligent tool we we try to we're trying to automate test creation or test authoring as much as possible and with this to we're trying to say oh you don't have to bother yourself with manual test you have an automated tool that help you generate different edge cases when it comes to to to your tables or your records or your forms as the case may be um another interesting aspect of test generator is that we been able to expand our coverage Beyond just um tables right now you can decide to run or to generate test based off of forms and also service cataloges you're going to see some of that later in um the demo then for the cloud Ral um this is an interesting aspect of of the test generator on cloud R feature itself we we had customers talk about um headless browser and how it's it's difficult to set up and how it's also hard to maintain we add all this concerns and we come up with what we call the cloud Runner so with the cloud Runner means you are running all your test on the service now infrastructure and one thing you have to understand is that I know questions have come up about oh does the customer data get copied on service infrastructure no it doesn't what what we do is we we take a snapshot of the past six months for for when trying to generate the test and we create test based off of that we do not copy your data we have um a couple of links that will tell you um our data policy when it comes to Cloud Rena and and yes we have a question about about test generator and the data uh so the question that came in is that test generator uses actual data which we just discussed is can I use test generator on new applications I'm creating a new custom app yes you can um as as long as the the custom app has a table you you're referencing test generator can help you generate that in fact we we we've gone ahead in the latest update we've gone ahead to to ensure that you can actually save a preset for the scope of the application you're generating like you want to generate test for so regardless of what you're trying to trade or or build as long as the application has a table that we can access you can always generate Bas of that but it will need data will need data so if you don't have enough data as long as you have some data for you to work with you can always generate test so there's no restrictions on custom tables or out of the box tables or custom Scopes or out of the box scopes or stor Scopes there are no restrictions there but we do have to have data to generate the test on so if I have a brand new application it's not going to create feature tests for me it's going to do regression tests and if there's no data then we won't be able to generate valid tests and if I have just a couple of Records the test may not be any good so it's more of a go forward that's correct that's that's why I said earlier that um it it analyzes your your your instance behavior and to analyze your instance Behavior you need a couple of data generated using the the application great thank you yeah interesting question so um um we've also had comment from customers that oh when they install um test generator it comes with two default Lanes that's what we did but they they found out that over time they need more Lanes or they need more more Trad as the case may be for them to run different tests let's say for example you're trying to run seven test if you have the two default Lanes the seven test will be killed one after the other so that means one test is running on the first Lane second test run on the second lane and they get killed up so they can run so um we we we decid to say oh for you to have more parallel test running at the same time you can you can get additional lanes because it comes at a very low cost so if you're interested in that you can always reach out to your your account executive or your to help you then we can go forward from that so um I I think I mentioned some of the the updates we've done on test generator and cloud and I just want to run you through some of the things we've done um the the the 1.1.1 was the version we released after the first one back in November 2022 and we had some feedbacks where we had to fix a couple of bug and stability fixes did that but if I'm going to touch on the the major one or some of the interesting updates we we've add so far in May 2023 we added the the ability for you to generate test based off of scope like oh this is the application you want to generate test of then we also um we noticed some some script ances with the progress bar so we also fixed that in May 2022 the 1.26 version then in August 3 2.04 version you can find all this in the the version when you go to the service down store we eventually give support for service catalog and this is an interesting tool you can you can go try it out once once after this Academy is done um we also fixed a couple of stability um improvements on bugs also then in November 2023 which was a couple of months back we we we G the ability for you to do something called condition Builder so for example let's say you are interested in having a select number of users to be used to generate test or to be used to validate a couple of test you can use the condition people that select to to to limit the users you're interested in you can also use a condition condition Builder to tell you oh to define the tables you want to create this test within for example if you're interested in a couple of Incident Management table or some change event table you can use a condition condition Builder to to set this limitation then also in February 2024 we did we we customer said oh they want to be able to to um generate test from previous um what it called previous cope they created so we give the ability for you to be able to save save your preset in a particular manner so when you go up there when you go to the H you would see the ability for you to save so when you save that what that means is that if you're trying to run the same um set of tables again or the same set of users again or the same service catalog items also again you can go back there and just generate the test again my manager likes to Adam likes to say oh um test generator is a disposable um tool like disposable generates disposable test so let's say for example the past months there is a different um instance behavior that you that some some of your customers have done on on on your application you can always go back to that scope you created earlier and generate a new one with that it analyze the new changes or the new validation that has been done and create something similar to that and and one interesting thing is that the the latest version of the the test generator and cloud has a backward compatibility to Tokyo PCH 5 so if you're on Tokyo PCH 5 you can still use um the 2.25 version and yeah we a couple questions um that that came in um the first related to some of the stuff we've talked about uh and the the first one has to do about moving just want to clarify with the update sets when you generate tests um and move them between instances do you do you or can you move your test between instances uh via update sets or some other method or do I have to generate tests on each instance if I have my my Dev and my QA and my staging do I have to generate the tests on each three of on all three of those instances or can I move the tests around yeah interesting question you can you can actually move the test around but if those in instances share the same table or the same data set there's no point moving the test around you can always just generate again based off of the scope like you already know the the with the condition Builder you already know the users you're trying to limit limit it to you also already know the the the tables you're interested in so you can use that same scope on your other instance but you can move the test around so if I have so scenario I might I I might be in is that my staging instance my my my last instance before prod has all my data is a clone I might generate my test on that and then move them down to my Dev that has a lot of clean data um or that has has a lot of yeah that that that has a lot of sanitized data doesn't have my full data set so you may want to generate you want to generate the tests on the instance that has the most data if they all have the same data it doesn't matter and if they um and then move them down so I might generate them in one instance move them to where I'm going to do most of my testing onto my Dev instance but it's whatever makes sense for you and really similar to wherever I put my ATF tests um but you do you really really want to have data that's the the better your data the the the tests that you generate are only as good as the data that you give it to generate the tests um and I'll plug into it that I definitely do call these disposable tests you are going to regenerate them from time to time I'm not necessarily going to regenerate them every day every week but probably after every when my data changes my process changes because there's no point in me to get feeling good about the data that this works for my instance from two years ago I want it to work on my instance now I want it to work for this upgrade for this process um so don't feel bad about throwing away these tests and regenerating them we don't charge for generating tests it it takes a little bit of time depending on your data but the you know I run them keep them for a bit throw them away and get the new set these are about fingerprinting your instance and regression tests and you want to generate them as your data changes a couple other questions um can I use these to test portal pages so um right now we are expanding support for service portal so um at the moment you can't use this portal Pages well the and I'll clarify the test generator doesn't work on Portal Pages we don't generate at test for portal for service portal today but the the cloud Runner does run everything uh does run anything ATF can run and ATF canest silver portal yeah so we don't have Auto generation yet but everything else we're talking about today does run with service portal so just think of your Cloud as different um virtual machine machine HED on a different device so you're not trying to run your own browser but if you have service portable that you run typically on ATF you can run them on cloud R but you can generate um service portal test and and Y I I don't know off top of my head do does test generator and cloudrunner work on uh personal developer instances yeah it does uh no it doesn't it doesn't because um the the the the infrastructure we're using right now does not have um that capability right now so doesn't run you can't run T gen on on PD yet okay think I know that's something we've been we've been talking about how to do that um ATF does run so most of the other things work test generator doesn't really make sense in a lot of ways for PDI because you need to have data and generally you don't have real data on pdis um so but it does not work if you're trying to get familiar with it it doesn't it doesn't work quite yet yeah all right I'll let you go the rest of this and then we'll we'll I'll interrupt you with some more questions yes perfect so um I think I mentioned this earlier also about um getting addal Lanes if you look at the the there's a cable article down here and with this cable article we we've had instances where customers say oh they want 10 additional lanes and we created this cable article for customers to know what are the limitations to having additional Lanes like for example um what's the number of nodes you have on your instance that will determine if you can go up to 10 or 15 but the highest the highest possible you can go up to about um 20 lanes so you can you can have 20 lanes on your instance and run your test as fast as possible so um you can use this KB article just to get more insight into how this works then so um now we're here to talk about what we have for Washington DC release so um this is a much to about um topic and we finally finally came up with this in Washington the support for configurable workspace and right now we we did some analysis and we found out that customers mostly interact with the forms and we're trying to prioritize okay this is what customer need first we want you to be delighted with ATF so we we we gave support for forms for configurable workspace and if you look at the the lower part of this image here you can see all the test steps that are supported within configurable workspace um later for for the next release we're trying to give support for least because that's the the next used component within the configurable workspace but um um this is what we have for for Washington DC and some of the interesting thing about this is that it's activated by default like it's you don't have to say you want to go download something it comes with ATF by default so once you you have you have upgraded to Washington DC you can run or you can generate test that are for configurable work space forms and I think the next thing also is um we we're trying to enable test designers to test customizations Beyond service now Legacy UI the call UI and some of the interesting things you've done on on on um next experience be want you to be able to test them we want you to be able to to see um see some validations understand what customer what what you are trying to see get insight into how your test is being run so um I'm going to show um after this I have a demo will run through how this works then you can ask some questions if you if you have them then um our next major feature on ATF is what we call the performance testing and this is an interesting aspect of this because um as you both know as we all know rather um regression test can be can can can be a bit tricky in the sense that when you make customizations or when you make changes to some of your pages um the chances of them causing a bit of delay or increasing the the test time is is um is is high so with with performance testing we relas this tool to help you pinpoint or understand okay this is the particular component that's causing a bit of delay in my test if your test is used to run for like let's say 30 seconds and all of a sudden you notice your test is running for 5 minutes or running for 2 minutes you can use this tool to pinpoint what's causing the the extra lag in test execution and also by default this comes with the with the Washington DC release you can always find this in the in various part of of the of ATF looking at image right now we have it in the top right corner we also have it beside the own test in the middle of the image so there are different ways to to um to um run this and one interesting thing is you can only run performance test on cloud rner because we need um the way this works once you see the demo you understand why we we are we are limiting the ability for you to test that on cloud rner um the the the when you run the test it runs about 11 times and we do something called warm up on cloud rner tries to ensure oh the first run is is okay with not each and be now run the next 10 on cloud rner so with for you to run performance test you need to have installed cloud rner and ured you've activated or selected a cloud user or or an user for that um so the the the the whole point of the whole T behind this is we're trying to help you discover customization based performance issues which are often ignored by developers and um they're they're they're not observed ignore it is a harsh word they care they care but we want to identify those right let's let's find it before we go to PR instead of after yeah yeah yeah that's correct so I I I think I would also want to mention that this we should not confuse this with low testing this is not this is not a low testing tool is strictly performance you want to see um how your customization is is improving all the all how your customization is improving or or the delay that is that is happening within your customization right time for demo and we can take some questions before demo yeah Adam well I I'm just gonna add um one or uh a couple things one my Q&A just actually cleared out if you asked a question that I hadn't that we hadn't responded to yet can you ask it again so we make sure we cover it um if you had asked in the last last couple minutes if you could just put it again if we didn't do it um there was a question that came in about um uh recording test recording and I uh Mark I think you you asked a question about test recording we don't offer that today um in the world under our Safe Harbor in the world of AI we're looking at kind of to jump over recording uh there is an app in the store app in the store there's an app and share that the the I forgot the name of it but there is a test recorder for ATF it records ATF tests it makes it make records what you do makes ATF tests that's a third party not supported by by service now but it creates ATF tests those tests are the exact same tests and they will run in cloudrunner they they'd be usable in in the performance testing and the other note I was going to put get about the performance testing yinka talked about it but it's only available for cloudrunner and and one of the big reasons for that is because that way we don't have to worry about network issues since cloudrunner is hosted in our data center we're going our data service now data center to service now data center where your instances um that works and we don't have to worry about any network problems before then so we're more likely to catch Real Performance issues rather than just some Network latency somewhere between between you and our data center um and the uh there was comment in there about the test recorder yeah the test recorder is not ours I I it's been around for a long time and I I'm not sure on how active it is I could believe that it's not active um we are working on Solutions uh to kind of go beyond that there's some really really good things that we'll be talking about later this year hopefully as as everything comes to fruition um but I definitely hear you loud and clear that uh a recorder uh is important um I don't have poll open on that but we have quite a few attendees in here and I'm going to keep the chat from this so if you would like to see an ATF recorder if that's important to you just put that in the chat and then we will be talking to our our uh our product development team a little bit more closely and if and the more evidence we have and the more people asking for it the more likely it is for us to to get that prioritized so if you're looking for a recorder let me know in the chat and we will follow up on that um okay then we have a couple of more um oh one quick question before before the uh before the demo about the the test data for the Gen for test generation so we mentioned six months is there a way to change that um to to tune that could I make it three months or could I make it 12 months for example so I I think that's an interesting question and something we look into we right now you we don't have the ability for you to to change that value or change that number but um we can look into how how what's the possibility of that and why we should do that yeah okay great I'm ready for a demo okay think our audience is as well let's go good um let me pull that up starting here from my instant homepage I am going to navigate to the ATF test module I'm going to click the new button to create a new test and I'm going to go ahead call it whatever I like from here uh we'll arrive at the standard ATF test form I'm going to go ahead and add some test steps so we'll click add a test step and what I want to do is I want to open an incident form in service operations workspace so we'll start by searching for the open a new form step and I click next now here is probably the most important part of this demo which is the ability to choose from different workspaces or different versions of the platform UI by default we have selected here in form UI standard UI uh which basically refers to any pre-ex UI experience interface um sometimes called ui6 also U now in this case I actually want to choose service operations workspace you can also note that there are other workspaces available here to select from and for for table I'm going to go ahead and put [Music] incident then I'll submit to create the step and we can see the step has been added to the test I'm going to go ahead and add a second one and now I want to fill out this form and so the step to do that is set field values note that the mandatory Fields have already been populated and I'm going to go ahead and just fill these out with any value just for demo purposes finally one more step here I want to submit my form I'm going to click submit a form I'm going to confirm that the step will assert that the form is submitted to server successfully you may notice that on the subsequent steps this field is already filled out we actually fill it out using the value from the prior step which means if you set it to a workspace in the first step you can expect that all subsequent steps are going to automatically have this populated to the uh matching value is the first step from here after my form has been filled out and submitted I want to click a UI action and what we're going to do is we're going to click the assign to me action and what this does is it's going to assign the instant to my active user which in this case because this is kind of a trivial test it's just going to be system administrator so I'm going to go ahead and select assign to me uh When selecting UI actions make sure that you choose one that's relevant for what you're testing so in this case this is a service operation workspace version of this action I'll submit this step and finally I'm going do one final step which is do a field values validation to assert that the assigned to value on the form is set to my current user which is system administrator so from here we can go ahead and click run test to bring up our browser picker and this is I'm just going to run the test locally and what we're going to see is we're going to see the instant form opened up in service operations workspace we will see it populate short description and Coler we going to see it submit and finally we will click the assign to me button and validate that it is assigned to system administrator and it actually goes pretty fast and going back to our test form we can see that the step or that the test did indeed succeed so that's an example of a very uh kind of basic trivial use case in Asia workpace okay um do we have questions for that I'll just note as we're going through so we we ran that in the interactive mode which is the the classic mode for ATF where we had a we ran the test from our local browser that's really important that's not going away because if I need to test it with specific plugins that I have or I need to test it with Safari or Edge that's great today our cloudrunner runs on on Chrome and what would happen if I had run that with cloudrunner was I would have only seen the first tab which shows my progress and the actual execution would have been done in the service now data center on our on our browsers um but effectively it's the same thing it's just a matter of whether you want to use our browsers and want to have a tab or you want to have a tab open and run them what you're looking run with very specific browser configuration yeah that's right looking for your comments to see if you have any questions before I move to the next demo I don't see any questions coming in so I think we can go on to the next demo I think this is the really exciting one for everybody to see um add an again feel free to put in questions along the way whether we're about this or anything else we've talked about today yeah all right so hey one one question came in is it possible to log the result so um we we all we always have test logs within um ATF and um once your your test have been run you would see a different tab different button that shows you all the test log so you can all all the test run on ATF are always logged you can always see the test log after you you're done run any test and the default is to take screenshots when there's failures um there's an opt some in some cases I have to have screenshots for everything uh there's an option on how to do that but when I test passes the results the the there's a log that it passed but we don't keep every step along the way yeah or it just as pass so it's pretty simple but particularly when there's failures that's when you will want to save that screenshot and go back um and then what you'll do is go back and if that's not enough you can debug the test you can persist the results to really drill drill into it but generally one of the great features of ATF is when you run the tests you don't have to worry about leaving a mess behind you because it will roll back the transaction the transactions and remove it'll undo the things that it did as part of the test um which is what we want the vast majority of the time but I will normally run my Suite anything that fails I'll go look at the screenshots I go look and see what was wrong and then I'll generally debug those keeping the the data so that I can dig into it to really understand sometimes the failures are really easy to understand screenshot tells me Ah that's broken I can go fix it um but there is the debug mode now as well which was added in the not too distant past uh which will allow me to keep that data persisted and not roll it back immediately so I can go dig into it find the real problem fix it and then run my test again and know everything that is happy and smooth that's correct oh and yes you definitely don't need to be there for the running of the test cloud in the past you didn't need to be when you were using local your local browser you had to have your laptop open though and occasionally run into time out issues and other things with cloudrunner you're just going to schedule it and review the results you're going to see a lot of green be happy um it'll be great and if there are any Reds then you'll be able to then dig into it so the failures again you'll see the log this these two steps or these two uh tests failed right th this step failed which caused this test of fail which called the sweets of fail you'll be able to get to that really quickly and then generally what I'll do is debug that test and watch it interactively go so I'll watch what what is breaking um and then I can go figure out is it is it suppos to do that or not right sometimes when you get a failure we use the word failure which is not I don't like it to myself it's we we noticed a change in behavior and then it's up to me to look at it to decide is that change in Behavior desired or not um if it's you know I can't submit the form anymore probably not but if I can't submit the form anymore because a a new required field is missing well that's actually correct I need to update my test um so I'll do that but the running of my thousand regression test overnight is not something I'm going to be there for it's going to run while I'm sleeping yeah and and one also interesting addition also is um once you have the the test generator and Cloud installed or activated um on when you go to um shedule test automatically by default you would see run it in Cloud right now run it in Cloud so basically we're saying oh if you run want to schedu your test it's best you run it on cloud R now because you don't have to be there or you don't have to just wait by a computer to have it run okay so um I think I can go to the next demo now so um this demo is for performance testing um all around on cloud right now so um we're going to start right now it's a quick demo of the performance testing tool in ATF coming up in the Washington release so here I have a demo instance uh running Washington release um the main focus of the tool is to help customers find performance regressions especially earlier in their development cycle um a lot of times uh performance is an afterthought when when doing development and a lot of the issues arise once they uh hit production so this tool aims to help customers find those regressions sooner in their development cycle making use of all the the great things that ATF offers already go ahead and click uh Cloud Runner and accept the acknowledgement that uh we will be pausing the instance for the duration of the test uh this is going to happen if you run a test a suite or a schedule so it's the same for all type of performance tests when I hit run test the instance will pause and it will give a 10-minute cooldown period to allow any jobs to to finish executing and also to allow the memory and and the the instance to go back to a steady state before starting the tests um I have disabled that for demo purposes for now but in a actual instance that would be the case so once you hit run test we issue ass system pause and we wait for a cool down of 10 minutes I'm going to go ahead and click run you'll see a total of 11 runs in here so the very first one is a warm-up run and 10 sequential runs so all of these execute sequentially actually the warm-up run is so that we do any server side caching um so that we warm up the cache basically on the server site so that the actual Runs run through one through 10 um have a better uh cash localization and again this is all in efforts to stabilize the results so I'm going to let this run but I'm going to skip forward in the video so that we get to the result this run is done here again just to recap we pause the system we waited for about 10 minutes for the instance to cool Downs for the cool in Period basically now uh we ran one warmup and 10 runs so all these execute sequentially and now we're done so I'm going to go ahead and click go to result and this is going to take me to a new table introduced with this with this future is called a performance run table very simple table it basically just acts as a container for all our runs here so we have a total of 11 runs you can see this one's marked as a warm-up and Step results here now you'll see Buton right here and a link as well as here to to uh the UI actions to compare the performance results so this is the part where it's the most useful for customers so once we have performance runs and we want to compare them I can go ahead and do that so I can compare a test against another test right so the same test or I can also compare a suite versus a site or a Suite against the test as long as that test also exists in the suite so for this demo I'm just going to compare to previous run that I had done so I'm going to click here you're prompted with this model here I'm going to select it uh from this same test right here so I'm I'm going to select from test but a previous run that I had done for basic UI test go ahead and click that and I'm going to click okay now this is going to take me to this new form called performance comparison so once you have multiple runs you can always navigate to The Run and to the performance run and click the compare button but you can also find that under performance profiling in ATF uh you'll see basically an entry for both of those new two tables so the performance runs themselves or a performance comparison so you can also create a performance comparison by just clicking new here and selecting your runs um but from a ux perspective it's probably easier if you're already at a in a performance run to just hit that we have a couple of questions for that so if you do have any questions feel free feel free to drop those into the Q&A panel we'll get those answered anything related to uh test uh the performance testing uh test generation Cloud Runner anything having to do with ATF um we'll answer those if not we'll wrap up a couple minutes early and do want to thank you for your your time and questions we had a couple great questions coming in there were quite a few great questions and a very active chat thank you very much um and again noted uh about the request for the progression about about a sorry Recorder about a test recorder for ATF we'll take that feedback back a few other questions about a road map and what we're going to handle in test generation and these are great sessions I appreciate you coming and sharing your thoughts and what is important to you um we may uh try to follow up with you on some of these and uh a question did come in yink can you see the question in the Q&A about the UI test not sure if I miss this but what kind of things are included in performance bics you I test um I think can we have more details about what this question is so yeah if you can give us some examples in that one Kristen will will try to answer that one uh next question y do you know I I I was trying to answer this in the chat but I was I wasn't sure do do we have do we know when we have scheduled or what we're planning for um service portal I guess it service portal test generation we have test generation for the catalog item um but I don't think that's actually in the portal that's just the catalog item by itself through the through coreui service portal test generation backlog so we're looking at um having that in the next two releases we're looking at having it in yukama um saveable notice so uh we we are trying to also get all the requirements and understand what customers really want from this but we're looking at having that come out in yukama great so we're looking to expand what's going on um the uh uh another question came in about Cloud runner for each instance in the lanes can you explain how the lanes work if I if I have four instances um do I need to buy four to get started do I need to buy uh four ATF packs or how how does that work okay so if you have four instances and you want to run Cloud R all those four instances all you have to do is just go to the store install Cloud r on each of those four instances so what that gives you is eight eight Lanes or two lanes per instance basically and if you need if you need more more Lanes to run your test you can get additional five lanes per pack so what we're saying is each instance comes by default with two lanes yeah so so there to get started every instance has two two lanes ready to go um no charge those are those are ready to go use those if you are maxing them out if it is taking you if you're running your full set of regression tests and it's taking 12 hours a day to run your regression tests then we should look at adding some new Lanes what comes through but um for most of us especially when we're getting started two lanes per instance is is more than adequate to get through it take a few minutes the alternative today is that I run these tests manually in my browser and I'm running them single threaded you can only run as a user a single user you can only run one set of tests at a time cloudrunner is doubling that and they don't have to take a break to go get some coffee um so cloudrunner is going to put you in better shape than you are today U particularly because I can run it overnight and it can just kick it off at two: in the morning and run through my tests and have them ready when I come in at at 8 um so use it and then if you're maxing it out then let's talk about it but you don't need to buy anything to get started yeah that's correct uh I think Christian also responded to a previous question so um the basic UI test we saw that we run the performance test on um was just one of our test our test within ATF and it has a couple of test steps So within those test steps we we we have a test step that that add um to validate the particular component so it's the basic U test you saw was just one of our um test step within ATF um when you say underlying activities with we the changes that happened was that one of the components we I think we added a new image in the component and that increased the delay or that increased the time for the test to be run so um regard the basic UI test is just our own test just our own test does not it's it's not um it's not unique or it's not you you won't find it in yours unless you have something like that I think that that should answer that question so it's I I don't think there's there's any underlying activities I can explain concerning the basic U test so I think there's another one what is the timeline on cloud R plugin not breaking inbound scripted rest API not using certificate based pa okay now this is interesting I think we need more insight into this also aam do do you I I actually I I don't know that one um we'd have to follow up on that if I'm assuming that that there's a case in a prb we could we could take a look what's going on but I'm not I'm not sure why it would be breaking with that um or what cloud I have some ideas of what's coming on uh but uh uh we'd have to look at that inside of a case of what's going on on but I I'm not aware of yeah timeline for that specifically um hey a comment came in in the chat that I want to make sure we address too um and the the comment was when test tests are successful is it possible to send the update sets to prod so I I definitely recommend that when you're once you have your tests don't keep them off in la la land put them in prod get them into your app repo get them into Source control um get them into prod so when you clone they come through the test results we don't move up today the test results are probably are going to go away when you clone you're going to get rid of them they're just helping us make some progress um if you need to retain your test results uh indefinitely and you have clones then we we need to address that and make sure that the retention periods are there um the test runs have a table cleaner so they do clean themselves on on all the instances eventually but the test themselves whether you're uh the test themselves I want to get them in prod test generated tests I'm probably not going to put in prod because when I clone over I'm going to want to regenerate I mean a clone event means I'm getting I'm doing a clone because I'm getting new data and I want to regenerate so my test generated test I probably will throw away and not move to prod anything any other ATF tests I am probably going to move to prod so that they're retained over clone um I I've seen some people who just keep their tests and update sets and and lo them in after a clone um I I don't see the value in that I would just again make them part of my app send them in my app repo uh have them go make sure they get in a product whatever the case is so when I clone down I I have my test ready to go um and for for performance test does it identify which business rule or where the logic let's think business rule or flow or something that's going on does it identify which one it is or does it just say this test take took longer yeah does it does like gives like a Pinpoint location of where the the delay is coming from or where the the the issue is coming from so it does give you the particular test steps tells you okay this is a component that might be causing the delay so it gives you like an an interesting Deep dive into the issue right which is why which is why we have it go run on cloudrunner so that we eliminate network issues or network latency and variance and it also um it's why we try to qu the system right when nothing else is running so that in the in the real world today testing is quite difficult performance getting a performance test because what else is going on so that the performance tests are are designed to try to eliminate as many other factors as possible and highlight where we might have a potential problem um it I I don't believe it's going to catch a 100% of our problems because some there might be some interaction with something else that's going on right somebody else's script might be using your resources for what comes through um but it's going to give us a lot more insight and pinpoints help us alleviate the simple errors right and as we as that gets smarter as everything else gets better we we'll improve it um but it's going to give us something where we've had nothing before and give you something to look at and focus on and make sure that we're we're not having any unforced errors go out yeah um and then as the followup to the updat set um update sets moving automatically is not something that I would see us doing because I don't want to ever mess with your prod uh even though that would probably be safe to do and you could do it um it I would want somebody approving it or or some some workflow to make sure it's okay um we're not talking about it today in our road map um as we go through but there is um a lot of of thought being put on about how does ATF and how to many things how should they operate through our instance Stacks today our instances are very isolated prod doesn't know about its sub prods the subr doesn't know about its prod but especially in the in the in the use cases we're talking about well I want to I want to ensure I've tested on a subr and then and then the code can move to prod the test might move up I want to see my test results um moving in prod right I want my test results retained in prod even though the test ran on a sub prod we're putting a lot of um thought and design in how do we better handle those use cases how do we better support that um so those are some of the areas we're investing in so we'll see hopefully see some improvements in that um in in the in future releases yeah um and it's something basically with devop as Paul you commented about devops yeah H how do we handle that it's similar into our if you're familiar with app engine management center and pipelines for app engine how does that concept how do we expand that concept to handle more of the use cases about about devops moving things up and down understanding um uh my my pipelines where things come through again tests are important updates are important as well I don't want to in I don't want to activate a plugin in prod that I have not activated in a subr at least we want to be really clear about that because that's a really important step uh so a lot of thought and investment going in that and we'll and hopefully see some of that come to fruition in the next few releases all right um okay I think we've answered all the questions that are in the Q&A um and with that I think we have a couple minutes so I think we're going to wrap up just a few early give you a chance to get some coffee before your next meeting oh let's y has a few more resources so I'll let y can wrap up yeah so um if you want to followly increase your your knowledge on ATF we have um pre on demand courses on now learning so ITF Essentials then also I I think also we have a couple of blogs and articles on servical community we're always trying to update this as much as possible give you more content and um with this you can also ramp up your knowledge on ATF and also um we have a couple of other academies analytics Academy mobile Academy next experience and Academy so you can go go look at all these resources and help yourself ramp up your knowledge of um all these different areas okay sorry well I'll wrap up that this recording will get posted in the community um it may be a week and a half from now but we'll get the session up there we'll get a PDF of the slides for you um and yink has also been working on some other great content out there to Showcase some of of the demos and things and the features we' talked about so a little bit greater P smaller pieces for you um and there was an ask about getting this link out um I'm going to grab that really quickly the link for the training okay uh oh sorry the link for the getting started of everything I'm putting that in the chat now if I can figure out how to cut and paste um there is the the center of expertise um so the I think if we go to the the next slide with the community post that has the link to the training the training get all of the service now training and now learning gets updated uh when needed what every release or two so be careful about bookmarking a specific course because that course may get updated um this one I think does need a little bit of an update and the updates for Washington um generally going to come out in April they come out normally about 30 to 60 days after the ga release that's when the courses get updated um but this course is available now um the I'm not sure if we have an update planned or not actually for Washington but it's always always best to just go into now learning and search for automated test framework then you'll always make sure you're getting the latest one rather than bookmarking this one and having an updated one that that you miss and that's true for ATF for analytics for workflows uh for workspaces whatever whatever the case may be just go to now learning and search um and uh I believe we also have some Labs at K24 which we didn't put our plugin our general plugin for that but if you're at K24 um there'll be some sessions on ATF um and and we're happy to talk to you about there live as well all right you you have anything else to add oh I think that'll be over thank you for attending this session and we hope to see you in the next academic session later in April all right thank you for joining we'll talk to you later goodbye by