all right well go ahead and get started thanks everybody for your patience so hello and welcome to our webinar an introduction to automated testing and service now my name is Scott Jacox and I will be your moderator for today's session on today's webinar we're Club will cover five topics first we'll give you a quick overview Kovac next we'll give you an overview of the ServiceNow automated test framework or ATF as we'll call it through much of the presentation we'll discuss the technical requirements common use cases best practices and finally we'll wrap up with a live demo not just introducing myself again my name is Scott J Cox I lead our service management practice at Co Vesta ik and that includes our ServiceNow partnership and practice joining me today is John gubec IO he'll provide an overview in demo of ATF today John's been working with ServiceNow since 2008 and has completed over 35 projects spanning everything from your typical core IT service management and customer service management implementations to complex custom integrations as we progress through today's session if you have any questions please feel free to type your question into the question field located on your control panel we'll do our best to answer questions at the end of the session but if for some reason we're unable to get to all of the questions we'll follow up via email I'm also a recording of today's webinar will be emailed out to you for playback or sharing with other members of your organization who were unable to attend today's session so with that let's go ahead and get started so quick capacity go review we're a full-service consulting firm we're a silver ServiceNow partner headquartered in Kirkland Washington we employ over 200 senior level of consultants averaging over 15 years of professional consulting and IT operations experience and have been delivering service management services in solutions since 2001 our expertise is organized into three different practice areas our IT ops services practice focuses on developing and standardizing and operating I key operations functions including cloud and enterprise infrastructure security operations applications administration and operations build and run project delivery services focuses on providing project leadership to deliver complex IT projects Kove Essex experience in project management ensures compliance quality and timing of projects and deliverables aligned with our clients lifecycle practices finally our service management consulting practice focuses on assessing defining and implementing optimized service management solutions with ServiceNow being our only service management solution partnership so with that John I'll turn over to you great Thank You Scott and thanks for everyone for joining us today for our webinar an introduction to ServiceNow is automated test framework so we're going to start our introduction with an overview of the ATF tool so I've got mentioned I've been working with ServiceNow since 2008 I started out as a system administrator and then moved on to consulting with the software and this super cool functionality has finally come as of the Istanbul release and I know since being a sis administrator it's been something that I've been hoping for so I was really excited when I first heard of it being released with Istanbul and and I've been working with it for a while now and really found to be very cool functionality that ServiceNow provides us so the automated test framework provides us a means for testing forms within ServiceNow so for example we can set up tests to test the incident management form one of our problem management forms as of the Jakarta release we now have the ability to test the service catalog so we can test for instance ordering a new laptop or we can a lot of people have a record producer for their end users to submit an incident and so now we can set up a test for that record producer out in the Service Catalog to make sure that our end-users can submit incidents in to us so we have the ability of the test on the client side of ServiceNow and now we also have the ability to test on with the server side so we can use the automated test framework to test business rules as of the Jakarta release we now have the ability to test rest inbound so a lot of clients and customers have created integrations using the REST API and often times when we're doing upgrades or getting quarterly patches or adding new functionality to those integrations we need a means of regression testing them and making sure they still work for our important integration points so as of jakarta we can now test those inbound rest functionalities so the cool thing about excuse me atf is that it provides how the box so it's built into ServiceNow so the advantage of this is is of course ServiceNow is customer support supports it for us and we will be able to use it for the main thing for our upgrades so as new versions of ServiceNow come out we know that service now in theory has tested that ATF functionality since its native and built into the tool and it's ready for us to perform our testing or internal testing and custom testing for our environment for those new versions also with the Jakarta release we now have the ability to schedule our testing so we can schedule testing to occur nightly so perhaps we've got development working on different stories and sprints and so we can have a nightly testing of that of that development and then you come in first thing in the morning and there's a handy-dandy email in your inbox telling you if any sort of test might have failed and possibly been disrupted from development and keeping in line with a ServiceNow platform of course we have the ability to customize the automated testing framework so as an ATF administrator you have the ability to set up any sort of customs step configurations that you might need for maybe so you've created some custom applications or you have some unique configurations within your environment that may not be available for you with the out-of-the-box test steps so using they've provided us a special object a utilities object and we can create our own test some steps to kind of cover any sort of gaps that we might encounter with the out-of-the-box steps okay so now that we've covered an overview let's go ahead and dig into the technical requirements so the automated test framework was first released with the Istanbul release and it's been greatly enhanced with the Jakarta release so as of the Jakarta release we now have the ability again to test those rest in bounds and so if you're on a Jakarta release of ATF your system administrators will have access to a special plugin that can activate that rest inbound functionality additionally with the Jakarta release we have the ability to use the noose scheduling of test Suites so you can turn on that plug-in for the scheduling functionality and then with the Jakarta release we now have the super cool ability to test our Service Catalog so again testing our catalog items and record producers that are being consumed by our by our end users so to utilize if you're going to be doing any sort of testing with the client so if you're going to be testing out your forms you'll need to use a supported web browser for that so unfortunately most of the current versions of Chrome Firefox edge ie and Safari are supported by the automating test framework and the service now provides us what they call client testers and these client testers will be running inside those particular web versions and they'll be checking in to the ServiceNow instance to see if they have any sort of tests assigned to them so most of the most the web browsers that you're using currently to interact with ServiceNow we can use those with those client testers okay so let's discuss some of the main test use cases for the automated test framework so the number one use case for the automated test framework is for testing out major releases for of ServiceNow such as this Tim Bell and the newer Jakarta so ServiceNow is getting a lot of feedback that one of the major impediments for customers upgrading of course is all the regression testing that they would have to do for the major release so because of all that testing and that burden on the system administrators and the developers of the ServiceNow instance of course a lot of people were delaying the upgrades and and waiting you know until practice some of them waiting it so they're actually forced the upgrade until the mandatory upgrade so with this automating test framework that hopefully reduces that burden of the testing for the major releases and it'll get you up to the most recent version and also get the functionality that your business may need in the new releases so a lot of times you know the businesses business partners your company may need new functionality in the tool and sometimes one of the roadblocks has been well you know your internal system administrators and developers they don't currently have the bandwidth you know to kind of get you there as soon as you need so this automated test framework will hopefully reduce that that drawback and get you have you getting up to the newer releases to support your business needs of course it's also going to help out with those mandatory quarterly patches so from my experience it seems like man I had just finished doing the manual point-and-click testing of a quarterly patch and lo and behold I get an email telling me that another quarterly patch is going to be employed to my environments so again with once we've set up these automated tests that'll help us with with our quarterly patches as well the automated test work can also help us with our mission-critical applications so a lot of us using ServiceNow for incident management and of course this is a mission critical application for for our ServiceNow environment and so the automated testing we can set up and we'll have some of this examples of this in our demonstration well set you can set up tests and test Suites for your incident management so if for whatever reason if there's an emergency change coming through for your incident management process you can quickly execute you know all of your automated tests against your incident management and hopefully keep it up and running for high availability you can also use the automated test framework for any sort of new development or enhancements that you're doing in your environment so if you're a system administrator a developer for your company's ServiceNow you're adding in some new features you can perform the regression testing so I know as a system administrator a developer a lot of the work the backend work that I have to do is okay add in the new development or enhancement that the business has requested but now I need to make sure that I haven't disrupted any of the existing configurations within ServiceNow so the ATF will help me perform that regression testing similarly if we have a defect that's been reported to us and we need to get that defects fixed I know one of the drawbacks sometimes is I'll fix one thing I'll be fixing I'll be focused on fixing this urgent defect and then inadvertently I'll add in some dude and their defects to go along with it so with the ATF you know I can work on my defect and then run the tests I have for that particular application and make sure not introducing anything new for those of you who might be using sdlc or agile development for your service now environment you can so a lot of times I'm on client projects and I have to create testing plans and testing strategies for the stores that I'm working on well now with the automated test framework I can give you know I can create up a TF for that particular story and then fellow developers can you know run that ATF and check it and make sure that my code solid so if you're in a team development environment you can set up basically take some of that burden away from your fellow developers and create your own ATF tests for for your a July ok so now that we've discussed some of the use cases for ATF let's talk about some of the advantages and disadvantages so from my perspective one of the advantages is again this is built into ServiceNow so we have ServiceNow as customer support for it and we have fairly high-level confidence that when the next release comes out that it's going to it's going to work in that ServiceNow release and so just what we've seen so far from Istanbul to Jakarta we got a lot of great additions with the Jakarta release and so I'm sure on the roadmap for future releases of ServiceNow they're going to continue to expand out the functionality of the ATF the other advantage to using ATF is because it's native to the tool and its built-in it has those unique testing functionalities for ServiceNow so for example we can impersonate a user we can open up forms within ServiceNow we can set the the fields that are within ServiceNow you can also do validation of the ServiceNow forms to make sure things are mandatory read-only invisible to the user we can also check to make sure that UI actions are available and then now with Jakarta we can do the testing unique for the Service Catalog so submitting those catalog items and record producers oh and also with Jakarta now we have the damn bound dress so you get out of the box the unique test steps for testing your ServiceNow environment and kind of saves you the hassle of having to build those up maybe in a third-party tool of course the other advantage of the automated test framework is it's an enormous time-saver so in the past since the early days you know I've got this huge multi-tap spreadsheet and here are all my test steps for inside management here are all my test steps for problem and change management and I have to go line by line reading through this excel document the following the particular test steps typing in Manly my test results is attaching screenshots of what I see so this is as most of you know this is enormous amount of time and again that impediment that a lot of clients have faced with adding new functionality to their ServiceNow environment as well as taking the major upgrades because it's automated we kind of take out that human error of our testing so as you know for at least for me it's kind of tedious doing all this manual testing and so it's pretty easy for me to accidentally miss a step and and not to not do that test in the consistent manner so the nice thing of course with our automated test framework is those test steps are going to set those same fields consistently it's going to do it in the proper order and so you can run these other use cases you can run these tests again for performance so you know that it's going to be it's going to be executing these tests and in a consistent manner in reducing kind of that human error that that gets introduced through you know manually testing so there are some disadvantages for ATF right now so as of now as of the Jakarta release we don't quite yet have the ability to test the UI pages in the service portal but I've heard that that's kind of on the roadmap for ATF so looking forward to that additional functionality in future releases another disadvantage is of course because you've got to transfer all your manual tests and the Excel spreadsheets over into into the automated testing framework there is an initial amount of time to setup those tests those test steps and test them out and make sure they're working but fortunately this is an area you know that Kovac would be able to help you out with is kind of taking off that load and burden of that initial configuration so you can reap the rewards of the ATF over time so it's one of those things where you have to put in the time and effort in the beginning but you're going to reap the rewards of those tests over time over the lifecycle of your ServiceNow environment currently we don't have there isn't support provided for the mobile and tablet interfaces for testing so you do kind of have to rely on some companies have already used third-party tools that can perform those so some customers using selenium and sauce labs and so the other third-party tools so that's a little bit of a strength for some of those third-party tools is they have the ability to you know spin up those like sauce labs spin up and spin up an instance of VM and and test more a little bit more thoroughly some of those mobile and tablet interfaces but I hopefully you know this is functionality that will get down the line okay so now that we've discussed some of the advantages and disadvantages let's go ahead and discuss the components of the ATF so the main component of the ATF of course are the tests so these tests are composed of the test steps so you'll have tests most likely for your incident management process your problem process change management and then all your different catalog items out there so you'll have test setup for those different application areas within ServiceNow so those tests are composed of what service now calls test steps so test steps form those individual manual actions that's kind of how I like to think of it so I'm impersonating a user so I'm acting like I'm this an ideal user I'm opening up a form I'm setting field values so these test steps essentially automate the automate those manual tests once we have a grouping of tests together we can set up what they call test Suites so test Suites are like are this is kind of like our in the end testing so in the demo that's coming up I have an example of incident management test suite so it's kind of walking through the tests of you know creating a p1 setting the category and subcategory making an assignment of that incident adding a work note an additional comment and then resolving out that incident so the test Suites provide you the ability to group together these tests in a sequence order so you can kind of do your in the end testing of an application service now also provides near the what they call test templates so these are kind of cool because what you can do is you can build up templates for your most common type of test sequencing and then when you're setting up a new test you can apply these test templates and they'll automatically add in all the individual test steps and the proper sequence and so this facilitates you know the development of the tests also kind of keeping them consistent and standardized so so that's what we can utilize those test templates for and then as of the Jakarta release we have the ability to do the scheduled these test Suites so again similar to scheduling within ServiceNow you can set these up daily periodically or on-demand and then you can execute these these test Suites may be off hours so maybe you're having your upgrade occur you know in your dev environment off hours and then you can just say okay well we anticipate the upgrade will complete within this amount of time so that he can schedule your your test Suites to run after the upgrade and lo and behold you come in in the morning you can kind of see okay this is where we need to start looking at and troubleshooting some of the issues that may have occurred with the upgrade so again that's newer functionality with with what the Jakarta release okay so go ahead and discuss a bit kind of the steps for setting up the test process so the very first thing you're going to do and we'll see this in the demo is you'll be assigning a name to the test so like you're testing an incident p1 creation and then you can put in a description of what that test is going to do so again this will help out other developers or maybe QA people on your team to understand what that test is going to do once you've got the test created then you're go ahead and add in the test steps so in these individual test steps again are kind of mimicking the manual steps that you would have to do to test that particular functionality so you own personate a user like an ideal user you'll open up a form for the incident you'll set some values or validate things so these are the individual test steps that you're going to configure again if you have some test templates and ServiceNow provides us one out-of-the-box you can apply that test template to your test and automatically add in those tests the nice thing about that is is then you just have to configure the individual values for those individual parameters for those those tests so just telling it which user to impersonate and which table to set the form on and then which form values to set so once we've got the tests in the test step set up the fun part is we get to execute that test so ServiceNow uses what they call client test runners so if you're in if you're running any sort of UI tests you'll spin up a client test runner in a browser and then that client test runner will basically pull check into ServiceNow to see if it has any tests so as of the decart to release you now have the ability to set up what they call scheduled test runners and those special test runners basically just constantly pull ServiceNow to see if there's any scheduled tests that they need to execute once we've executed the test then we can observe the progress and we'll see this coming up in the demo you can see the tests it's pretty cool you can see the tests getting executed interactively and then finally they'll give us the test results for for that particular test and again we'll take a look at that coming up here shortly ok so let's discuss some of the best practices for the automated test framework so the number one most important the best practice in my opinion is of course to always perform these automated testing in your development or test instance and so because you're being you're going to be creating record you don't want to be testing your p1 your p1 creation in your production environment you don't want pager alerts going out and email notifications going out and everyone getting alerted that there's a p1 and then because the automated tests work does a rollback err kind of cleans up after itself you know one minute they're seeing a p1 and then next they go to look forward and it's not there anymore because it's been rolled back and cleaned up by the test so so of course you'll be using these you'll be using the automated test framework and your development and test environment the nice thing is the service now has made these properties for activating the automated test framework and it kind of gives you a little warning it's also made the property private so you don't accidentally transfer the activation of the ATF through update sets through to your production environment so they've done ServiceNow has done a good job of kind of calling that property out to system manage traitors so they don't convert early get it over to prod another best practice I recommend is configuring some groups for managing your ATF so there tool there are two main roles for the ATF there are administrators so they have the ability to configure the custom test steps and also set some of the ATF properties and then you can kind of delegate the testing out to other hello developers and system admins maybe you got some admins for your HR and other applications within ServiceNow so you can kind of delegate that testing out to test designers and they have the ability to set up tests for their application areas I've also found it's kind of useful to have a group for the ATF test runners so you can kind of manage your scheduled test runners and any sort of manual test runners that might be logging in and then ServiceNow recommends that you perform impersonations for your UI tests so I found it's useful to create test users for the different application areas so you can set up a test user an ATF test user for incident management change management and your end users and then you apply the appropriate roles to users so that they can perform those tests actions it's also recommended that you run the test runners in their own browser windows so this gives them a little bit more access and so a little bit more access to the computer so that they can perform the test so there is a knowledgebase article out on HIGH that describes some some steps for reducing some of the timeouts so after the assemble release there is a feedback from from the user base that you knows tests for timing out and so now you can actually for some of the validation test steps you can increase the time out and also if you so happen to be using Chrome in a Windows environment you have you have the ability to turn off some of that CPU throttling so again this will give those most likely this is going to be your scheduled test runners but maybe some manual ones you're running on your desktop so that they don't hopefully don't encounter that timeout in their compute you know timing out because during that they don't have enough access to the CPU okay so those are some of the best practices so let's go ahead and switch over to the demonstration here okay so for today's demonstration we're going to be in a jacquard to release and I'm using the newer UI 16 so what you're seeing here is there's a card to release of ServiceNow with the newer a UI 16 interface and so to find the automated test framework so if you if you're a new client that has Istanbul or Jakarta then out of the box you will have this the automated test framework activated now if you've upgraded to Istanbul Jakarta one of your system administrator x' can go basically go to plugins and they can automate the different ATF ATF plugins that you may need so once those plugins have been activated in your environment you're going to have the automated test framework show up in your navigation menu so for our demo I'm just going to start off with describing some of the application areas for the ATF then we're going to have a demonstration of execution of a test I will also have a demonstration of for a lebackes test failure so you can see what it looks like unfortunately when a test fails and then we'll wrap up the live demonstration with with a test test suite okay so let's go ahead and start with basically a description of some of the options within the automated test framework so the main one is are the tests that the ATF are going to run so I just have an example in here of the p1 incident creation so we'll come back to that in a little bit the other area is the test suite so again these are the Indian the end-to-end testing that you can perform so the test suites or the grouping of these those individual tests and we'll take a look at an example one that I've created for incident management you also have the ability to look at the different test results so you can look at past results these are retained out of the box for 30 days but you as a system administrator have the ability to preserve those for long individual test results for longer or bump up the clean up basically of of these test results we also have test results for the test Suites and and then newer with Jakarta we have the ability to set up schedules so I just have an example of setting up a schedule for incident management and similar to other scheduling within ServiceNow you can set up the different type of interval and then you associate which tests which you want those scheduled tests run again these since these are scheduled you'll need what they call this special the new with Jakarta the new scheduled test runners so we have we have the manual client test runners and then we have specifically just for our scheduled tests we have the scheduled client test runners you also as an administrator can see what manual test runners you have running out there and then you also have the ability to see and this one's kind of helpful to check to make sure that your scheduled test runners are checking in and kind of still online and ready for any sort of scheduled tests you can also check to see what tests are waiting to be run so these are the ones that are kind of queued up so you can check to see which tests are waiting for a client runner to start processing them and similarly you can check and see which test Suites are running or waiting to be run okay as an ATF administrator you have access to the ATF properties so again the most important property here is the activation of the automated test framework and again this should be done in your in your development or test environment and and and ServiceNow has made it nicely that this is a again that this is a private property so it doesn't accidentally get transferred over to production in your ATF update sets I also found it useful to turn on the the debugging properties for ATF so this provides us just a little bit more information as the tests are running in case that your tests are encountering some failures you can you can get a little bit more detail on that now this is cool new with Jakarta we now have the ability to determine when the screenshots are captured so initially in this damn bull release the screenshots were captured for every single step and so this added a little bit more overhead and of course taking up a little bit more space for your test results so so cool cool new function now with Jakarta is we have the ability to determine when those screenshots are going to be captured so for my most of my stuff versus administration and development I'm mostly interested in just finding out if my test steps have failed and so I I said it to capture screenshots when the test steps have failed I found it useful when when I'm debugging a failed test step to take a look at the screenshot and kind of see okay you know what is that field not as that field not real only was the value not set so it kind of helps to facilitate the troubleshooting of of a failed test okay you also have the ability to set some of the timeouts for your client test runners so out-of-the-box 120 seconds so every two minutes or so it's it's it's checking to make sure that that runners still out there and available for use and then again because similar to our mid sir these client test servers are checking back into the main instance and so that you can set the heartbeat interval and so every every minute or so they should check into it into your instance to say hey I'm ready to go are there any tests for me so those are kind of like our drones checking in with us and then you can set you know how aggressively you want to consider those test renders offline so out of the box if we haven't heard for them for a day you can go ahead and kind of clean those off the lists with the Jakarta release you have the abilities that you set up some of the configurations for the new emails that get sent out so let me see I think I've got an example down here we can take a look at so these properties basically are setting up some of the coloring which is kind of nice that they give us that ability to kind of fine-tune that email so this is an example of one of the emails that you'll get from a scheduled test suite so it gives you the name of the test suite and it's kind of nice to give us links to it so so we can quickly get into that particular test suite and then it gives us a breakdown of how that scheduled test suite went so it tells us you know what the failures were what the errors were if any sort of if any test steps were skipped and then how many things overall the suite pasts and how many steps had passed and then again you can also drill into those individual test Suites to see the results so these are these are the the need new email notifications that you get when a test suite executes okay so I think that covers all of our properties so the next area is the test templates so again these are this is an example one out of the box and I like these because it was helpful for me is kind of a building block so they give us out-of-the-box they give us an example of a default new form so what they've already done is they've given us you know the impersonation opening up a new form setting some values you know validating those values checking to make sure fields are read-only mandatory checking to make sure that our submit and resolve buttons are on the form and then actually clicking the submit and submitting those forms so these are the test templates that you can set up to kind of facility facilitate and standardize the development of tests for your environment and then this is where you can change if you need to hang on to your test results a little bit longer maybe more than the out-of-the-box 30 days then your ATF administrator can kind of bump out bump out those time those times and then for the advanced users you can actually this is where you would go to setup your your individual test up configurations so I found it was useful to kind of examine the out-of-the-box test steps and again they're using there's there's some methods and functions within these test steps that you can kind of repurpose if you need to build a custom one for yours and then they group these based on the different category areas of of testing and so you can kind of look at the examples of these ones if you need to if has an administrator if you need to build some of your custom ones okay so that's kind of an overview of the automated test frameworks options so let's go ahead and start with taking a look at an individual test so I've created an example of of a test testing for p1 incident creation okay so I've given it a name I've given it a description and then this is where we put in the individual test steps for this particular step so the very first thing that ServiceNow recommends is that we impersonate a user so in this case I'm impersonating a user that has the the ITIL role the next thing I'm going to do is I'm opening up a new form for the incident on the incident table so I open up that form and then once we've got that form loaded up let me show you here it's going to load up our individual test steps here so the nice thing about building these test steps as the the UI interface is similar to what we you know assists administrators and developers we've seen for setting conditionals for our our business rules and so and templates within ServiceNow and so you know once you select the table you'll get access to the fields that are on that table and then you can set the particular values so in this case I'm doing an example of email is not working so of course this would be a p1 incident and so we set a short description for the incident we set the the collar and again this this this is a lookup to our user table so we can select the color that we want to associate to that incident so this we're creating a p1 incident we're going to go ahead and set that impact and urgency the hi will set the assignment group over to the service desk since its email is not working we'll go ahead and set the business service over to email and we'll set an initial category for software and then this p1 came in through a phone so we set the contact type so this allows us to basically set up the individual values that we want the automated test to perform for for this incident form so let me go back to you our test okay so once we've set those values on the form the next thing we need to do is we need to validate that that that that those values were set properly so ServiceNow uses what they call a value a field values validation and and this is where if your if your test if your validation happens to be timing out you can kind of increase this a little bit so it gets a little bit more opportunity to validate this information so what it's going to do is it's going to check to see okay well we said impact and urgency too high so we want to make sure that the priority was set to critical we want to make sure that the collar was indeed set to you able to tur we want to make sure that the assignment group was set to you the service desk the business service was set to email and the category was that software in the contact type so basically this is the flipside of setting the values we're validating to make sure that those fields for success Lisa once we've checked to make sure those field values okay then what we'll do is we'll actually check to make sure that the fields on the forms are all right so here we're checking to make sure that these fields are visible on the form so these are the the fields that that should be visible on the form or category a short description contact type impact and urgency if we had any read-only fields we could check to make sure that those are set to read only properly and then importantly we need to check to make sure that our mandatory fields are set so one of the nice things when you're setting up a test step is if there are any mandatory fields the ServiceNow automatically put those mandatory fields into the set values for you so you don't accidentally miss those guys but this will this will double check to make sure that those fields are mandatory on the floor all right so again because this is this is out of the box and native a service now we have a special test step for checking the UI action visibility so this is where we can check to see that the submit button and the resolve button are on the incident form and then our final step of course is to submit that form over to you ServiceNow okay so that kind of describes what this test is going to do so let's go ahead and execute this test okay so they they've been nice ServiceNow has been nice and kind of colors this button so it kind of sticks out so you can go ahead and click the run test to run this particular test then what it's going to do is it's going to check to see if there's any client test runners running so for today's demonstration I'm just going to kick these test runners off as we need them so you go ahead and click the run test button so what that does is it kicks off the client test runner so now the client test Runner is connected and it's it's checking in for the tests and so now what it's doing is it's executing those test steps for the incident creation so it's setting those field values that we saw and then it's performing the validation now it's also checking to make sure that those UI action buttons were visible lickety-split how do me a penny have a time to describe it all and it's it's actually completed that testing for us so we'll go ahead and close this tight client test runner and then once that test has run you'll get basically a summary of the processing of that test so there was a server-side test step that ran so that was the impersonation and then the rest of our tests were considered UI test steps so for any any test that has UI test step that's when the client Runner will be needed to perform those particular UI tests so now we can drill into the results for that particular test excuse me so it gives us the test results for that particular test the good news is is overall was successful so it gives us the status for that and then it gives us when it started and ended and this is the cool thing with ATF this particular test tip of the p1 instant creation took the system in an automated fashion 18 seconds so I can guarantee you and when I was working through an Excel spreadsheet and doing the manual point-and-click and capturing the information and validating is there's no way in heck it would take me 18 seconds to perform this so again that's this is the power of the automated test framework is that it's able to perform these tests and take away some of that you know take away that manual human interaction and increase our crease they reduce the amount of time basically it takes us to test our test our functionality so for the test results we have the individual test steps I found it useful to sequence those those tests step results in the order that they were executed so basically this is telling us that you know the system successfully important impersonated the ITIL user was able to successfully open up the incident form it was able to successfully set these values and then it performed the the the field validation and was able to match those things and then it checked to make sure that they were read only manatorian visible so those assertions were successful and then again because it's negative deserves now it's able to check those UI actions and so it was able to confirm that to submit and resolve buttons were there and then finally it was able to confirm that it was able to save that particular incident so if if you needed some additional details the ServiceNow provides us the test log so the test log is basically giving us you know the Java log or the console log so if you need to if you need to dig in a little bit more to what some of the logging might have been while the tests were executing you can look at the test logs and then if you if you really need to you can also look at the individual transactions and so the ATF is making a call back to the instance to perform these these automated tests and so you can kind of see the different api's and transactions that are occurring for those particular test steps okay so that is an example of a successful test for an incident p1 creation so let's take a look at what it looks like unfortunately if we encounter a failure so out of a box the service now provides us some examples of tests and these are super helpful for learning out learning the automated test framework and so your system administrator has the ability to load what they call load demo data for the automated test framework and you can load in that demo data and I'll give you the example how the Box the tests so that you can kind of learn from those so that so those are those are helpful so this is an example that sort of snot provides out of the box of what a failure would look like and so it's going to create create a new Service Catalog task and what it's going to do is it's going to set the description to about all the tasks that need to be discussed and then what it's going to do in the validation is it's going to say it's going to check for something else so it's going to check for a description of you know this is a different message than the one I said in the previous and the previous this have values okay so let's go ahead and run this example of test failure so again we'll click the run test button and then for then we're going to go ahead and start off a client test render to execute this test so now that client test render is checking in he's performing the first step of that test so he's opening up the log tasks form he's setting some values and then unfortunately you know he's encountered that air so he basically stops his testing so we'll close this and and then this is hopefully you know this is the bummer screen we see if we're system admin or developer like ah shucks you know our test has failed so now we need to drill in and kind of take a look at add our failure so we'll go into our results here so because I set that system property says hey take a screenshot hit whenever a test fails you'll notice for this particular test result up of two up at the top we have a timestamp the screenshot and so you can click into to view that particular screenshot and then here if we zoom in a little bit we pull this over we can see where the description was set to you know all about the tasks that need to be discussed but when we were doing the validation we were checking for a different value so the screenshots I found or super helpful for for debugging when a test fails and ServiceNow is as helpful what they do is they roll up the failure information for for that particular test step that is failed so you can look in the output here and you can see that that the description here says you know it was looking for this as a different message than the one that was just said so that's that's that kind of roles they kind of roll up the failure information and then you can drill in even further to that individual test tab and see kind of the detailed outputs of what what the failure was associated to that particular step so again all this kind of facilitates the troubleshooting and resolution of the field tests okay so we've looked at an individual test we've looked at a failed test so for the last part of our demonstration or we're going to look at the indian point testing of the test suite so I've set up an example of testing a test suite for incident management and so what this is going to do and I found again ordering this in the execution order helps to kind of keep track of the test suite so what's going to do is it's going to perform that test that we did in there earlier of making sure that our people an incident can be created then it's going to check to make sure that the categorization of an incident works properly then it's going to check to make sure that we can assign that incident and then we're going to make sure that an ITIL user can add a work note for that incident and then we'll also test to make sure that night till user can add an additional comment so this is the communication that I'll go back out to collar that's important we've got to make sure that that works and then finally for you know the end end testing of the incident management we need to make sure we can resolve that incident so this last test step well we'll test the resolution of an incident so let's go ahead and run this test suite so again we'll go ahead and kick off a test runner for this test suite okay so now it's working with the first test in the suite so it's checking to make sure that our p1 incident incident creation can occur okay so it's completed that particular test within the test suite so the next thing is we're going to check we'll make sure we can categorize the incident so we're checking the category and the subcategory is being set properly so it's confirming that that categorization occurred all right now we need to make sure that we can assign an incident so we sign that incident over to the service desk and we assign it to our old friend David Lu as the assigned to pay some homage out to David the original developer of the whole tool for us okay now we're going to test an incident work note so this is again making sure that an eye tail user can add a work note to to this incident I think if I scroll down we should be able to see setting up that additional comment yeah so so now this is setting the additional comment to make sure that that communication could go out to an end user and then finally what we're going to make sure is that we can resolve an incident so this this one setting up the incident for resolution it's checking to make sure those conditions that we said are correct it's saving that incident and now it's setting because we're resolving the incident we have to set the resolution notes and the resolution code and then it clicks the the resolution button and it's completed the resolution of that incident so let me close this okay so now we have the the results for that test suite so let's go take a look at that so similar to the other test results we saw we've got an eye it's a little bit easier to kind of see these from A to Z so overall the test suite was successful amazingly again you know testing all this functionality took the automated test framework about a minute to perform unfortunately all of our tests were successful but if there were some failures errors we get those individual accounts if it were a scheduled one it would tell us when this ran and then we see the individual results for those particular tests in this case they were successful and then similar to what we saw a little bit earlier you can drill into those individual test results for each one of those tests that guy composed this week so if you happen to have a failure you can drill in like we saw earlier to those individual test suite results okay so it looks like okay cool got some time for some Q&A so let me go ahead and switch back over to the Scott here all right thank you John if you have questions please submit them using the question field on your control panel we'll do our best to answer as many questions as possible but should we run out of time we will follow up with answers via email so let's start with our first question here John is is there a direct link between a TF test plans and stories and then in parentheses it says I've had to add that relationship yeah so you're correct out of the box there is not a direct link between the SDLC stories and so it's a common common related list that you'll need to add in so that's a tied those two things together and that's that question is also similar to I've seen questions out on some of the other presentations by ServiceNow now in the community about is there a tie-in between the test management Suites those are where you set up tests and you have manual test steps and manual you know all set up basically the test management area so it's another customization that you'll need to add in to also link any sort of manual tests that you have to the newer automated tests and the test framework so you're correct that's something you'll need to add in all right great thanks John second question what what would you recommend is a starting point for EDF so I recommend much like we saw on today's demonstration starting off with your incident management process so normally this is a critical process for a lot of organizations and so what you can do again is your system administrator can activate those those demonstration tests and test steps and so utilizing the out-of-the-box test template you can kind of get started setting up your first kind of testing so as I was learning it I started out with the incident management much like we saw in the demonstration so I used the out-of-the-box test template to get started and made some configurations through the incident management form and testing and kind of got my feet wet if you will with incident management so that's that's kind of where I recommend you know looking at the out-of-the-box test steps and test examples utilizing now the Bach test template and kind of kind of get your feet wet with with incident management okay thanks next one have you noticed any performance issues when setting test runner intervals to intervals so I've left them out of the box so far so so I've left I've left them right now to the you know the 120 and such but if you're if definitely if you're getting timeouts again if you if you have the ability use Chrome you know you can use those special parameters so that chrome gets doesn't encounter any of the CPU throttling and then butt in and now you can also with those individual test steps you can kind of bump up the timeouts so so you can bump those up as needed most of my testing so far the Dow the box ones have worked okay but if you're deaf you know if you're encountering those timeouts there also is out on the high there's a special knowledgebase article that kind of addresses and they're the ones hi was the one who kind of gave me that information about a chrome of the CPU throttling there's also a best practice article out in the community and they also provide some additional things that you can do to kind of kind of avoid some of the timeout issues okay thanks can APF verify that the VIP flag is displayed for VIP users so that's a that's a special little macro thing that will show up so I haven't I haven't with the field validation thing right now again they mostly give us the validations of what UI policies can perform so those are if the fields are mandatory you know required or a read-only so I don't think I haven't seen a way yet to kind of test to make sure that that little icon shows up next to the field so so that little UI macro icon so I don't think we have that yeah maybe I bet you when they give us the functionality for UI scripts and UI pages that hopefully we'll be able to test those those the more UI interface type things in future releases okay currently using test management in production well the automated tests work with test management yeah so so that's that's usually one of the main migration paths and so we've already you know we're using the test management and so we just add in a couple new related lists so that we can link the existing tests and test cases that we have in the test management thing over to over to an automated test so the nice thing about those things is they're already kind of structured out for us in the test management suite so it kind of facilitates us building building out the automated test so I was working off of Excel spreadsheets and and Word documents that we had for for our testing processes and I was just working through those documents and transferring them into the automated test framework so if you've already got those tests set up in in the the service no test management area it should facilitate pretty easily you know your template if you will of what you need to set up for an ATF and then as I mentioned earlier you can you can add a tie-in for those test management test cases so if they link over to their corresponding ATF so that when you kick off you know a new test a new test and the test management suite at all it already basically have a link over to the ATF that your QA or you know your system minister you will need to run so those those things can be tightened fairly nicely together with in service down all right and John we've only got about a little over a minute left so let me just grab one last question can can you test for dependent fields for example category and subcategory yeah so so you have to do it with two so for instance the example of the incident categorization the first test has to set the main category so I was setting it to database and then you'll have another test step after that one that sets the the subcategory and it was kind of nice as I was setting that up it kind of gave me a warning when I selected that to field it recognized in the dictionary hey this field is dependent on another one therefore you need to use a different step to set this and so it kind of guided me and let me know that for that particular dependency field I needed to use a separate independent test for it okay thanks John I feel go to the last slide so we're out of time we did have additional questions come in so like I said at the top of the webinar we will send out those questions and answers via email if you would like any additional information please feel free to send us an email at service now at cove st comm that concludes today's session thank you for attending and remember to keep an eye out for upcoming webinars from cove st thank you