since joining us today oh you all do have the ability to unmute yourself please be more conscientious about that ability you can also use the chat window to type your questions and all is monitoring that do you Paul you want to go ahead and get started yeah we'll get started otherwise I'm gonna probably run way away over time today so I appreciate everyone that's joined today I understand that this is probably not the best time zone for most people as I'm from Australia and my local time is 9:02 a.m. so fairly early in the day for me probably the officer end of the day for most people that would be attending knowledge 2020 so I'm just out of interests if you if I could get the chat just to let me know what country you're from so I can get a feel of the kind of audience that I have today we're all from just type it in the chat Texas awesome Oh God Australia probably guess one of my colleagues Canada excellent all right I'll get this underway so today I'm gonna be talking about all about automated testing and my journey with automated testing on the ServiceNow platform and how community has helped me kind of guide me down that path over time I've got mostly North America today North America and Australia all right cool so I'm from a Australian company called kinetic IT kinetic IT is a managed service provider company so we come in and we run service desks and like desktops and enterprise services generally and when we come into a customer we usually either implement or support service now as it's just such a good platform nothing really beats it so that's how I got into ServiceNow myself I started on service desk using ITSM products used to use HP service desk and then I moved into service manager and then service now and when I got into service now I never looked back so since then the last seven years I've pretty much been exclusively working on the ServiceNow platform and my interests have really been in automated testing and Enterprise Service Management solutions achievements I've been a lead developer on multiple ServiceNow implementations using agile methodologies usually delivering an ITSM solution and I have had the opportunity to recently do a customer service management solution as well customer service management is now pretty much as big as ITSM it's huge and it's getting bigger it's a really cool product and in 2020 I was awarded with a community MVP award which was really cool current projects I've just finished another implementation of ITSM for a local government agency and look forward to the next one and I've now been with kinetic IT for 10 years and kinetic IT is a sales services and technology partner I'm sorry agenda for today talk about my journey we've automated testing throughout my ServiceNow career talk a little bit about the automated testing framework some of the use cases that ATF addresses ATF being the automated testing framework some of the challenges that come with ATF and then I'm gonna give a demo on some of the functionality that I built on top of ATF to make it easier to write test cases so just before I start if anyone is not familiar with ATF if they don't know what it does or what it is can you put so on the chat because if I don't have to cover it means I can cover more of the cool stuff but if you're unfamiliar with it if anyone is unfamiliar with it I would like to give you a little background primer on it so unless someone shouts out saying that they don't know ATF is and they want an introduction I'll probably skip over that slide so I can spend more time doing a demonstration like to try and personalize this to the audience as much as I can so the three things that I would like everyone to learn today I guess is a bit of a broader history of automated testing on the now platform ATF didn't always exist being on the tool for seven years for most of that time ATF did not exist what some of the use cases are for using ATF and how to get the most out of ATF so my journey with automated testing so back in 2013 was where my ServiceNow journey begun I went on a implementation boot camp I got to fly over to Melbourne to do that with my colleagues that was a lot of fun no burns on the east coast of Australia probably the most kind of hipster city and Australia great coffee great food all that kind of thing and I joined a ServiceNow development team just doing day-to-day enhancements to the tool there was a project going on at the same time to implement a self-service system for the customer and as I was fairly new to ServiceNow at that stage I got put on the testing team so I helped the testers test his self-service portal rather than developing so they gave me an interesting insight into what it actually is like to be a tester and finding defects and a lot of the challenges that come with testing a lot of the repetition a lot of the changing requirements it was quite an eye-opening experience to see what it's like to be a tester and that really started my interest in automated testing when in 20:14 my technical lead asked me to start looking into automated testing so I used that as a business case to attend knowledge for team I've read up a business case sent it to my manager outlining that I wanted to go to this big conference in San Francisco to learn all about testing so luckily by the business accepted that business case and I went to knowledge 14 where I walked around the main pavilion floor talking to all the big partners asking all of them and any anyone I could find are you doing automated testing how he if you are what are you doing lots of people had their own kind little custom solutions but there wasn't really anything that was consistent across the board and it seemed like no one was really talking to each other there wasn't really quite yet the community around automated testing that there is now so I just found it amazing I had such a great time and I learned so much I decided to build my own little testing utility after that in 2014 there was a little testing utility written by a ServiceNow employee I'm not sure if he still works for ServiceNow but it was just a basic kind of code you know testing a little application that was up on community on the old community so I I tried to find the update set I'm proud of this presentation but I just could not find it and I was still fair in my kind of developer infancy on the ServiceNow platform so I'm really needed framework to work on so I started to build a GUI around the automated testing application that was on the community and just tinkering with it building more and more on top of it and then when application scoping was introduced I rebuilt the entire thing from the ground up by that stage I'd learnt enough about ServiceNow in the API halves to feel comfortable with creating my own application I I wanted to share what I had created with the wider community and contribute to a creating a community of automated testing so I brought up a presentation and sent it in for submission and it got accepted and I spoke at knowledge 16 about automated testing and demonstrated their app that I had made theirs me up on the pedestal there at the end of my presentation there seems to be so much interest in what I had shown and I already had it in an application scope so I decided that he might be cool to put it on the store so I managed to convince my company for us to sign up and be technology partners and started building this application for the store so there was many many hours after work that I spent on this because I had a day job as well I was really passionate about it so it wasn't hard it was something that I really wanted to do and refine my cleaning skills so during that time there was a community call for a request to use cases for automated testing which was kind of exciting I suppose but I didn't know where where it was going what that was all about and then a little bit later in the year a former colleague shared a video of the automated testing framework with me so it was interesting to learn that it wasn't just me that had this good idea ServiceNow were there as well so I'm at that point we just wanted to get the applications certified and on the store we put it up there for free as can't really compete with automated test framework it's just too powerful doesn't have the limitations of being in an application scope like my app did and then so the video this we got a certified we got to up on the store and then in 2017 ATF was released to the world and that's really what changed everything from that point so we all know what ATF is we don't have anyone in the chat that doesn't know what it is so I'm just gonna skip over that slide I'm going to talk to some of the use cases around ATF but while I'm talking about the use case I'll just love in the chat if anyone could share that if they're using ATF at the moment and maybe a few words as to what you're doing with ATF right now while I cover these use cases off so for me before primary use cases that automated test framework addresses is number one it automates and manual and otherwise manual testing process number two which the next three kind of lead off number one once you have if you can automate your testing you can therefore speed up upgrades because when you apply your upgrade you can do your testing quicker a free is that it can be used like it is in software in general to reduce defects it's not really geared up for test-driven development although you could use that for script includes it's more for regression but once you've got a automated regression bed you can run it and if your regression fails then you know you've got a defect so you can reduce the effects because you can test more often and number four it's quite handy to measure deviations from out of the box as service now comes with tests for all they're out of the box so you can run the out of the box test against your instance and see how far you've deviated from out of the box so for me those are for the four primary use cases for automated testing framework if anyone else has any other use cases that they would like to share feel free to put that in the chat so it sounds like most the audience here have maybe played around with a TF or they're keen to start using it but we don't have any anyone that's a expert looks that's been using it for a while we've just got a few people that have tattled with it all right cool so these are the the main use cases for a TF how do we get the most out of a TF so in my experience the best way to get the most out of a TF is to use it for the purposes of regression testing so that is just having a current state of your system that you accept is a correct current state and then mapping that current state into a set of tests that you can then run to validate that the expected state is as expected I suppose so an example in a use case number two is doing upgrades it doesn't necessarily have to be an upgrade it could be any substantial system change which an upgrade generally is so build ATF tests to represent the current state you say I accept that the way the system is functioning now is correct you write those test cases you execute and refine them until they're all passing and then you migrate those tests up to production as you normally would with any development change and then performing an upgrade you would then perform a environment refresh you would clone production down to your sub production environments your tests would come with that run them again to make sure that your regression tests are still all valid and then upgrade service now then finally you re execute those tests that represent the expected state and that will bring the failures straight up to the top rather than needing to wait for testers to manually go in and find those failures and bringing the defects to the top instantly it allows you to fix them very very quickly and bring down your upgrade time quite significantly because you just turned what could be like two weeks of testing into two days that's an example and fix those defects right away and really decrease that window of time for that upgrade so good right and sound almost sounds too good to be true is it really that easy what's the catch so there are some challenges just as there are challenges with everything in life a three key challenges I've identified with my experience using a TF is the technical knowledge that's required to use a TF you don't need to be a developer expert but you do need to be familiar with the terminologies what a UI action is what a mandatory field is what a read-only field is you need to be able to do some basic ServiceNow queries using the query builder so you do need to know a little bit and it's generally I would say you probably would need at least a good six months on the platform with like a regular usage of the platform and in my experience generally it ends up falling to the developer to write those tests because they already have the knowledge of the platform and that's less than ideal because developers are very busy and there's other things that they want to do not only that it takes a while to set up a regression bed it does depend on how much you want to target and how exaust if you want it to be but it does take time to set up at the start just like any automated testing solution time is money and the third challenge is opportunity cost which ties in with challenge one the developer he likes building cool stuff and the customer likes it when they build cool stuff to solve business problems so there's the opportunity cost that the developers spending time to write tests which could have otherwise been spent configuring ServiceNow or building functionality to solve business problems so those challenges do make it quite difficult to justify to a customer to set up a TF in the first place and I really wanted to solve these challenges to get more customers using a TF and this is still this is still a challenge I haven't completely solved all of this but I wanted to uplift ACF to make it a little bit better so how do we solve these challenges in ServiceNow when the platform doesn't quite do what the business wants them to do what do most developers do we build on the platform and extend it to solve the business needs so that's what I did in this case I extended at ACF by doing my own application on top of it I can tell already that I'm already five minutes behind where I wanted to be I wanted to start with demo five minutes ago so I will be going over today so I built an application that I called regress short for regression ATF recorder which automates the generation of test cases so essentially what it does is it just records the current state of the form when you submit it and converts that into test steps at the moment if supports things like field state are user populated fields changed fields and UI action visibility as generally I found when I was writing tests they were almost always the same thing it was like open a record populate some fields check which fields and mandatory read only visible etc check that certain fields changed correctly like if you change impact an urgency check that the priority is set accordingly and that those fields could be changed and which you are actions were visible so it was the same thing over and over so forth surely we can automate this most API is already there to determine these things except for your action visibility but I got there in the end so I'm essentially just translated form State at the point of submission into test steps also added a few extra features like determining similar records for retest so if you open an existing record in the test it will try it it won't try open that record exactly but it will try find a similar record for retest or if your you navigate to a record that was previously created in another test step output it will automatically link it together and some basic support for Service Catalog forms including service portal and that's a one of the things that customers really care about when an upgrade is they don't want their service portal broken and they don't want the service catalog broken so being able to do this for service portal service catalog makes it look easier to justify spending the time and it really doesn't take much time anymore so it's a fun part demonstration of my little utility all righty so just login to my status now developer instance oh just make everything a little bit bigger I just bought a 4k monitor on the weekend so just gonna type in regress here in the application navigator make this full screen so this is my little application here the important bit the atf recorder so i've got a few different modes of recording got actions only which only records the actions that you take for when you want to do some pre work and then test an output at the end you don't really care what fields are mandatory read-only etc you just want to record the actions there's simple mode I'm not seeing anything except for the slide I don't know if you're actually sharing oh okay demo or not but things fix her letting me know there yeah cuz I only shared the PowerPoint thank you I just bear with me for a second it's not the first I did that and it won't be the last so here can everyone see it now no try on sharing and turning again okay here we go this should work now there we go here we go all right cool so just looking at the application navigator at the moment on the side here the modes I was talking about actions only simple just covers kind of a light version of regression where it checks for I guess more of the positive cases are things visible are things mandatory and then exhaustive checks absolutely everything every field that's visible every field that's not visible every button that isn't showing everybody is showing etc so I'm just gonna help in the chat how many so we've got one person has actually tried building automated so not automated tests our tests in the automated test framework using the example of a catalog item for example we've got this one here I'll just say try it in service portal so if we imagine what we want to do in our test is we just want to go through ordering a provision a database item and then once it's ordered we want to validate that all the correct fields are mandatory read-only visible all that kind of thing so normally I'd come in here fill out a field like who's the primary contact the cost center what is the business purpose for this a special projects an order I don't check out it's done and then we might want to check in the backend of service now open it up and just do some validation on the form check with your actions are visible check which which fields and variables are mandatory etc so if I wanted to do that in the automated test framework tests new we'll call this vision a race through the GUI go test steps ad test step we want service catalog and service portal so the first thing we did we want to open a catalog item wait for the form to load service portal and we had provision a database submit so do you typically not set up a user like they're creating used on or in person that's a very good point to start off a test you'd probably it's advisable not to use an admin account it's better to use an end user so you can use that create a user to set up a throwaway user that then gets impersonated or you if you've already got like a test set of users you could do that in this case I'm just gonna use impersonate quickly just because it's quicker to set up and just for the purposes of this demonstration gonna hit submit it's not really good practice use impersonate anymore from New York onwards that suggested to use create a user step instead just because it's quicker just going to change the order of this one zero all right what's the next thing that I did I set variable values I'd like load I do have a plug-in open that shows me all the field names it's called SNC utils it's really helpful as a admin if you're wondering what why I've got all the field names there on the side what's curious there's no way it's a really really good utility so I populated business purpose we have input special projects cost center I'll just put in an arbitrary value here cost Center and primary contacts all right so that's three fields can look at them set variable values noted that they all the variables had a particular state they were all they're all visible for this will kind of make it simple they were all mandatory that's how he is check the variable state going to submit it order it and then we want to open the request item so you kind of need a little bit of knowledge on how to write queries in ServiceNow I called curiosity if you don't mind when you when do you choose submit as versus using a UI action in this case it's just ordering ordering a catalogue item I can cover that in a bit later I'll just quickly get this one free sorry I didn't mean to be really that's that's cool questions are always welcome record query so one of the really cool things about ATF is you can make your queries based on the outputs of other test steps so the order catalog item has an output that gives you the request number but I want to get the request item sorry I want the request item which has a request which is the output of this so we'll just leave it at that for now you'll see it's it doesn't take too long but it still does take time and if you've got a big system with lots of catalog items for example this is just like a very basic one just to go through the notion of impersonating or opening it up setting values checking values ordering it and then opening the requested item I'm just going to run this test waiting for too to run so I've got the execution frame here that's just opening the item populating the fields hopefully I've done this all correctly order now order waiting for just around should do the server side component off the test unless it's already done that cool it's already done that successfully and then it's deleted the record so it doesn't do the up your system what's the records so if I wanted to do further tests on the request item for example if I wanted to do a UI action on this week go to reform we can say your action visibility and type in let's just say check the update now there's two update button says update button on existing records and one for multiple update form so that's that's also that that feature part of that you till you were talking about us and I haven't haven't quite got to the utility bit this is just kind of sensor describing because you had something that described the the visibility or action when you went and clicked in there so oh and that's that's out of the books I'm that no cuz when I know yep sorry so oh that's weird okay I never see them when you were just looking before you had said there was two UI actions right and you said yeah and and you had something that explained more of what they were in my system all I ever see is just the UI action I'm saved so I see like save five times or whatever and you had oh okay yeah is that part of that MSN utils thing that you're talking about or I don't study something it actually describes instead of just saying save and when I see you know I see save yeah I can't actually remember I might have might have done so just because yeah that's one of the it is quite annoying when there's like one two three four five six saves and you don't know which save is which which is also annoying because I have the ability to do HF tests but not I have not a full admin in the system so oh okay that might be wise and I am an admin yeah it's probably not just because you don't have access to that little bit of maybe things text anyway so you can see but it's not the hardest thing in the world but it does take a little bit of time so I thought wouldn't it be nice if I didn't have to do that at all so I created this ATF recorder so I just created this test manually to give everyone a sense of how long it takes and the kind of technical knowledge that you need to know you need to have a little bit and as sorry what was your your name Oh Joe so as Joe was just saying it does kind of require a little bit of technical knowledge to be able to build it up but not if you can just record your actions so I've just favorited the ATF recorder here this will just record using the default settings just to be sure actually I will stop recording and I'll use the simple of recording so we'll go and go in our provision a database item again try and service portal this is where the magic happens all right so I've got a message pops up this just alerting me that I'm recording this and this is going to go into an ATF test primary contacts able cost center customer support let's go with Katie ello order the item that's just telling me that the form snapshot has been added to the test check out the item let's get the reference number for this come back into service now go searching for all rightie em again I'll just a little reminder telling us that we're recording I'm just gonna save the record now to make a snapshot of it not actually going to change any fields at this time I could if I wanted actually let's let's just add a CI to this now we're gonna save the record so one of the limitations I did have building this is I wasn't always able to reliably determine which UI action was selected so I just prompts the user with a little reminder saying just saying is this the button that you clicked so in this case save is the one that I clicked okay this only happens if you with particular UI actions that doesn't happen with all of them it's just a technical limitation around the able to determine which button was clicked because in writing this I try to use as much out-of-the-box API as possible and there just wasn't any API there to reliably trace the you action every time so I've just added a configuration item they're gonna scroll down to catalog tasks I'm free we might see whoo yeah that's very true as well you can configure those variables to add the graph autocomplete attributes as well I'm not sure if in Joe's case it was a CL not being admin or that I added them in there can't remember personally and so I want to run free like an end-to-end I've ordered an item I want to change something on the request item and I want to go through and close all the related tasks make sure that I can so this tests essentially testing that the workflow works as expected and that the form state the UI actions that are visible all that kind of stuff is as expected close the task and I might do a final save on the request item just to capture the state on that one all right so in that took like probably a minute to do and now I've got a automated test here and all the test steps have been generated automatically including impersonation so got impersonate open catalog item set variable values KTLA people treat us and customer support variable state validation now this this test did not exist before I did not write this ahead of time this is not this is not staged this was generated just then as you can see I just put in Kelo able to de customer support you can see the configuration item there that was the one that I entered and you can see the timestamp there nine nine 3606 my local time that's the exact time I did this so yeah this is not staged so this was all generated based on what I just did so the ordering of the item the record query to open the request item it's recognized that the record that I opened um was related to the one that I ordered I opened it it checks for every UI action that was visible now this would take forever if you wanted to exhaustively work out yourself every UI action that was visible this determines it automatically knows that update delete update delete github export save follow-on life they didn't you which those are like related this your actions it's recognized that they were all visible can you imagine how long that would take to do manually I'm standing the field value of the configuration item and then it's just being a simple test it's more testing for the negative so it doesn't test that everything is visible it just tests what isn't visible so you wouldn't have even noticed that additional comments wasn't visible that's a very hard test for someone to write manually because they don't know what's not there that how the recorder does which was read only variable state validation just checking that the variables are all mandatory as well that I click Save and it also recognizes that the task that I opened was related to the request item that I already opened opens that record again it does the UI action check does the field state validation clicks close task and then it goes to the other task check see your action Phil's day and again opening Lee request item through to the end so that's 27 steps probably would have taken me half an hour to an hour to do that manually and I just did it in about a minute automatically and just to proof as well that it works I'm gonna run the test and show that it passes it's doing the first bit that I wrote the test for manually populating those variables all right now order confirmation you I posted a pic question phonetics committee on comics the eternal community so it's perfect one of the nose it's possible to be that Oh I haven't tried to do that myself if there's parts of it that you could do you could from a unit testing perspective you could probably if you have abstracted some of your male script art until script include you could use the server-side scripts and Jazmyn to do unit testing on those scripts in Orlando ServiceNow did just add a test step for outbound mail for checking the outbound mail has been sent so one of the features I want to add to this would be to at the end of the tests just query the email log for every record that was touched and every email notification that was triggered and automatically generate the outbound emails there but I think that's more of a check of did it send not so much did it have the right content to the content look right cuz I know emails is a very time-consuming thing to test but I think at this stage it still requires a manual glance of the emails to make sure that you don't have like an error message where your mouth script is as an example so I think in that time it's just gone through and passed so this solves a lot of the problems I was having with a the person needing to be technical now they don't need to be technical you can just sit down with your testers and say hey when you're doing your testing hit start record and then when you're done hit stop and then you've got all your tests already generated so I'll show the same test I prepared earlier with a actions this is a actions only one so it just shows like my impersonated I opened I set values I ordered it I opened a record I clicked an action it doesn't worry about the field state and your action visibility because you don't always care about that depending on what you're testing then the exhaustive version which does everything as in what's visible what's not read only what's mandatory looking at the UI actions not only does visible but it also does not visible same with the field state validation so it's a lot more exhaustive version of exactly the same yes is this whatever run into problems of when you're doing especially you're doing both what is there and what isn't there is sometimes things like dates or when when you did it relatives or something else can affect yeah same change management I mean change management has a lot of very abilities in our case especially where depending on what you're signing group is your and your risk and your other things there that get done dynamically generated based on the CI or some other things that or even just how soon you did it versus how far in the past you did it yeah dates are always a tricky one because like if you set the planned start and plant into like the day that you're running through the test then the next day it's in the past and it becomes invalid so it is quite tricky like this this tool doesn't mean you'll never have to modify a test step or create test steps again it just speeds up the time Tommy awesome I was just a framework for the test yeah so what it does do to try I guess it only captures the fields that you change yourself it doesn't capture every single value on the field because then you do run into that date issue we feel it's like open created and that kind of thing so it's just about being mindful when you're running through this to set values it you know your aren't gonna screw up the test if you trying all right at a later date so I do have those modes of testing as well because sometimes exhaustive is simply too much and it just picks up too much it just depends I guess what you're looking for if you want an exhaustive regression right you might you might do this just before an upgrade just so you can immediately highlight so I know for example the last upgrade I did some change your actions became deprecated and some new ones were created and using this tool you can immediately see when it comes to the UI visibility that some things have have dropped off some things becomes incest for example so you've got those those free modes of testing there that pretty much covers it there's some property as well where you select the capture level which tables automatically get recorded which fields it uses to base similar record queries off and just all the templates that I used to generate the different sort of types of tests whether it's a new record or an existing record or you're recording an action and whatnot so that's so the cool thing this is all available on share for free for everyone so it's just on chair called regrets I've got a few versions the latest version 0.99 four has just introduced the catalog item recording and it has instructions on how to set it up there are some things that need to be done to get it working in service portal a few simple steps otherwise it's fairly standard there so that's on share and yeah just go for it download it put it in your Silk Road instance play around with it it's not perfect but it does a pretty good job and most of the time when you replay your test it will pass but keep in mind that you will always need some level of test creation it just really sets up the framework and makes it a lot easier to create the tests quickly many asses can you use a second quarter for custom scope applications yeah as as far as I know applications and custom scopes shouldn't work however I can't say I've exhaustively tested my testing has just been before this app has mainly been on out of the box ServiceNow instances so quite simple and have lower levels of customization but feel free to download it and try it and let me know I don't see why it wouldn't work alright I'll just pop back into the presentation bear with me Shea the screen again come on zoom disappeared on me screen demo so just highlighting the key challenges it selves is you don't have to be technical to write the tests you just have to hit record do your actions and then hit stop the test steps are created for you so greatly minimizes the set-up time required to use ATF and number three is because developers don't need to write the tests they're freed up the testers can just record the things that they're done and it creates the regression tests for you as you go so um I'll go to the chat see if we have any other questions does anyone have any further questions they would like to ask the community feel free to unmute yourself if you have a question you're just too shy just type away okay that's a good point I'm sorry I've got a link in the presentation however that probably won't help you terribly because it's recorded on YouTube so I'll put these links in the community posts which will have the recording for this session also give some links to some of the coding blogs I've written and I did write a little blog about my journey creating this tool so I'll put all those up on the community and I'll just put the link in the chat right now as well what's it doing there well hoping this thing Blackstreet is it just showing a white screen I just pasted in that's the link there so I encourage people to download it and let me know what bugs are oh it's still pretty pretty new application I've only just added that the service portal recording it's very basic and what it does but it does does enough I think so just if you find any bugs so if you have any feature requests or anything um feel free to post on the share page give it a good review as well if you liked what you what you saw very nice thank you for sharing our problem sorry that from around 24 minutes later and I actually cut down the presentation on what it originally was you know we'll just make sure that those links are posted on that community link and since I don't think they came through on the chat so please pause make sure that you just get along to that for me privately not sure if it went sorry I didn't go to everyone let me do that again there we go wonderful thank you all really appreciate it look see on the community and have a great knowledge