[Music] welcome to Serna podcast live episode 7 the one where we talk about understanding kpo and ATF I really appreciate you guys taking the time to join us for these events for actually coming out and participating speaking of participation if you have any questions throughout this podcast we want to hear from you right away as soon as you have a question go ahead and throw that into the questions module on your GoToWebinar screen if you open that up and just type in the question we may be able to answer it fairly close to two when you ask the question or we may end up getting to it at the end of the presentation depending on how how we're able to manage that but either way as soon as something pops into your head make sure you throw that in and we will do our best to get that answered for you I am joined today by the incomparable Sal Costa Sal why don't you go ahead and introduce yourself let us know a little bit of your ServiceNow background and and give us kind of a little bit of a high-level overview of capi oh sure thanks to him so yeah my name is Sal Costa I'm the principal architect of the products team here at Serna solutions I've been working with service now for probably about 10 years now since well before they were city names and and I'm the I'm also the guy that that wrote the book Gunn jelly and I'm pretty active in the community I published a lot of the ServiceNow tools and I'm pretty active in the the slack and reddit communities so people may know me Cappy o itself is is an automated testing platform it actually predates ATF by by by quite a bit here and it was really something that was was born out of my my feeling that there was a lack of of an automated testing tool in the ServiceNow space I have always regarded ServiceNow is as real engineering and I've always treated it that way men and every other every other aspect of engineering and the rest of the world they've got an automated testing solution you know built for that more general and some are specific but I thought there needed to be one for service now so I went ahead and built it and really didn't have any idea that it would become this big but I'm glad that it has and that it's that it's helping people great fantastic and so just so you guys know excuse me pardon me normally we do have our webcams on if you guys are wondering where we are and why you can't see our faces we had a little bit of a technical difficulty but we actually have a lot of information that we will be doing on screen that will definitely give you something to keep your your monitors on for as opposed to just watching some Talking Heads going back and forth at this point so definitely you know do your best to stay with us if you guys have any questions as I mentioned at the top throw those in there and let's just go ahead and to get started so as I brought up in the first in the poll for anyone that was here for the first couple minutes so I'm trying to get to the next one there yeah I think that we're still talking to a lot of people that haven't pulled the trigger on automated testing yet we still are seeing a lot of people that are using ServiceNow that are still testing manually either because they haven't seen the return on investment or the the value of going to an automated system or there's just too much of an obstacle for them more than they were thinking and Sal can you go ahead and kind of explain I mean you you brought this to my attention as we were talking about this can you kind of explain the relevance here yeah so this has yeah this is a this as a comic from xkcd calm who seems to have a direct line to the mind of Engineers but it's just sort of just sort of poking fun at the idea of approaching a task and automating it and and in a sort of is pointing out the the reality that being a dream often gives way to a lot more development work and the the whole automation task in general kind of becomes a sinkhole for for that work and and you know you end up doing more work towards the end than you were in the beginning so yeah this one's a pretty common one that we see out there and obviously definitely definitely very apt and true yeah and I think that you know this is a perfect example of why some people have looked into automated testing and not gone through with it or just are scared to get started I think it's because there is just that sense that there is still you know when we pull this other part up there's still a lot of investment both financial investment and time investment in making sure that that automation is initially set up correctly maintained correctly and these things you know generally they still take more effort you think you're just building a robot that will do everything for you but you're not and so there there is sort of that that blocking issue for for a number of people also the fact that tests will always require people that tests are always going to it's never going to be a hit the play button and it'll just run for the rest of time as long as you're building something that humans are interacting with you're going to need humans to verify that that thing works let alone maintain the robot and make sure that it's working correctly as well and more often we're hearing people and a lot of questions that we got before this webinar we gave you the chance to kind of give us some you know what your questions were on the registration page a lot of people said they just don't know how to get started in the first place and so what we want to do is we want to talk about how does ATF approach some of these problems how does Capp yo approach some of these problems obviously you know we are the developer and vendor of kpo so we're going to have more information there and we're going to obviously try to show you what some of those benefits are but we also want to show you you know help me make decisions on on both ends so first of all let's talk about how are these two systems used so so obviously I'll throw this to you you know what you know what would you say ATF is doing well and and why do you think that people should be using that yeah so yeah but you know at a very high level I think I think ATF would be great for system style testing I think I've heard some people refer to it as as unit testing in some context too and that's kind of maybe that's maybe a philosophical discussion as to the difference between those but in essence it's it's validating you know piece of functionality at a time might be you know something on a form client script or UI policy might something on the back end there it it's it seems to be built in you know in a way that it makes that makes that very easy and obviously the system the system that they've built you know that kind of that kind of mirrors a lot of what they do in servers now is definitely set up so that you can walk up and use it I'll talk a little bit more about I think the UI later there but you know the approachability of it definitely makes it so that you can walk up and start to write these tests immediately and they come bundled with tests which i think is a which i think is a good way to get started and in definitely definitely gets the ball rolling a little bit but you know that's that's kind of the limit of what I've seen APF be able to do there now I do want to interject real quick so for those of you that are non-technical non developers trying to absorb this information I'm here with you I am your non-technical liaison to this information so I'm going to be doing my best while Salas is speaking out on his developer level mostly to developers to try and also fill this out a little bit more so I don't want you guys to feel that you're intentionally being left out we are going to kind of expand this into the business applications of what this information means and if some of this seems a little bit head-spinning in terms of I know me personally I've had issues trying to understand the difference between all of the different levels of testing and all the different varieties we're gonna break this down a little bit more as well and I meant to bring that up at the top but if you have questions if something's going over your head we don't want that to be the case just throw into the questions what are you talking about what does that mean and we will definitely try to dig a little bit deeper there but Sal go ahead and kind of I'll throw it back to you I apologize can you go and talk a little more about copier at this point yeah so so in addition to the the system testing might be in validating the the functionality it's sort of the you know within the forum type level we go beyond that a little bit more we've got a layer below that so the lower level testing unit testing what I would consider pure unit testing does exist there too we've got a we've got a method for that it's a little bit mature you know so there's a there's a you know there's there's definitely a process that goes to these well you know talk about a little bit more about that but there is the capability to do unit testing and then there's also the capability on the other side to do that and acceptance testing their that that's what most people do now if you're doing anything manually you're probably going in and testing a form and to end and and it's got all of the capabilities to enable that fully there's always another well as you'll see there's always something that that is kind of unique and special and we you know we made sure with capya that you're given all the tools to to do that to do that as well now we actually have a question about are the bundled tests quote self-contained and they reference that in ATF QuickStart that it depends on existing demo data in the instance is that something that applies here I don't I don't actually bundle test out of box and we'll see we'll see why towards the end here I you know it's been my experience even though we've built it multiple times I probably set out and built like four or five an incident problem change type bundles they were not useful for customers because of that reason like you want to get in there and test it with your data it doesn't and doesn't work if it's if it's out of box data because you've added fields you've added you know you've added new things to it so for me I'd rather start clean or start with start with a starter script that we can generate and move from there rather than to have to customize and undo lots of things with an out of box test so yeah the the we don't we don't bundle it in the same way although the you know the the tests themselves would be self-contained depending on the type that you that you make which is select from this list here okay and this is something we're going to show off a little bit more directly correct mm-hmm now I have another question are you able to do GXP testing and validation I am not familiar with GXP testing or validation if it goes by another term I may have heard that I haven't heard that one specifically though okay if you could give us a little bit more details if we're able to get to that on this on this podcast we'll definitely try to approach that otherwise we may end up reaching out to you to the person that they asked that question so essentially what you're saying here with the information here is you know if I understand system testing is I think some people call it functional testing you're saying some people unit testing there is a little bit of a disparity in the language and terminology in the testing world especially between you know the ServiceNow specific environment and you know maybe more general versions of software engineering but that ATF you know functional testing does what you would imagine a tests to do in the first place which is make sure that that your functionality works right so why is it important on a business level would you say to have those other layers those other categories of testing as well yeah with the the I mean the main point with any testing is is to validate the code behind it as working so you know in in practice in practice here there are definitely cases where we're testing things at a smaller level for instance at a unit level is easier to do then whether you know then when it's deployed within a system or when it's it's part of a greater process like an acceptance test so so you know providing the ability to test these at multiple levels just just really gives the the business or the end-user more flexibility in how to how to increase that coverage there so you know just as a as a as a for instance here if I've got a if I've got a script included with a function that that is uh that it's kind of buried between you know under UI actions and informs being set up in a correct way that might be hard to get to might be hard to test that individual function so if I can pull that out and run that function at a unit level and test it that way I've kind of cut out I've cut out some of the sisty and I've come up some of the UI there I'm able to validate that piece of code and so and I and I've increased the coverage there whereas if I were trying to do that through a system or acceptance test it might be hard or impossible to do so it's you know it's having all of these at your disposal which which mean it gives you the greatest flexibility to test in the greater amount of code it's not we're not just creating tests just to just to say we have them point validate the functionality now I think that another way to kind of explain the business applications for this and why this is important is this hypnotizing little gift here on the right and the way that I understood it and the way that you know you and I spoke about it that you test would be in this situation of what we're looking at here would be the the servos the motors the sensors all the individual components of the gate if though all of those individual pieces work that would be similar to a unit test a system test would be more does the gate work does the gate open up does it sense that a person is walking through however acceptance test is going to be do all these things work the way we expect them to when you put them all together and in this situation obviously now we've got doors that are just constantly sensing each other and opening up and closing and creating a crazy loop which seems to be apt analogy to what it may be like for certain developers would you say that this is you know something that a lot of people are running into in their business and that's why we wanted to include all of those different varieties into that automation process yeah there are there definitely you know plenty of plenty of cases where where that happens here you need to you know in this case to it's true of this case as it is in service now that one of the most important things is the acceptance test so even if we didn't even if we couldn't get the other levels of testing in there it's it is still it is still important to make sure that we've you know got this ability to replicate you know the exact business application in the real world but testing those other components making sure that that you know those various features of the of the products here and then the individual components work individually is going to you know that's really going to ensure that if we take this and Transplant this and another location that we're gonna in it that the quality will be the same too so it you know that getting that greater amount of coverage really is the goal here and yeah you've got to use a you've got to use a combination of these different tests to be able to enable that okay and we had a question I'm gonna read it verbatim so do I understand this correctly do the scripts that run when Cathy o generates tests for me to get started will they all pass initially that yeah that depends so they you know the if talking about the scripts that get generated through through our tool through our case creator tool which would show sometimes it doesn't because there there are there are discrepancies between you know what's configured in the system and how they end up running on the and it's kind of its it kind of goes to this this example here or this analogy here that yeah and if we were to just test you know UI policy completely separate from the system it might pass but when I load that on the form because of the way it interacts with other policies and other client scripts in the default values and the way the dictionary is created it might be different so you know at this point case crater is not able to understand things that happen on the client side and there's definitely things that it doesn't recognize but the the you know the the goal for me is with those starter scripts to - you know decrease the amount of work it takes to get to a running script sometimes they are perfect sometimes they do come out and they just work 100% out of the box here but I would be I would be investing my time and getting those starter scripts created and then going through and and and making sure I put in the work to make sure that their hundred percent passing it there that they're set up correctly so yeah it's not I'm not I would never expect it to run a hundred percent passing right from creation okay and if you for the person that asked that question if you need any follow-up definitely feel free to punch that in and we'll see if we can answer that any differently for you so right now let's hop into you know what is covered what do these two different systems do so yeah so with with ATF they they they they cover the the core system I would say pretty well they've got they've got abilities and commands to go in there and do things like forms and validate things on the navigation menu they've even got some some commands or test steps in there to validate portal and catalog and there were there's some additional ones too which are a little bit lower level that you let you do things like run you know run scripts and I think there's there's even things in there to execute some some rest type commands to validate the API endpoints so they've got they've got a core they've got a good core amount covered in there in capya though in a bit because the you know because my goal really was that you know the the full end-to-end acceptance test or at least that that's the most difficult thing I had to go my had to go much further the first thing I guess to know is that you know that we are selenium based here and so that you know these these commands are the types of actions that we can do are pretty much whatever a user could do in the browser you know clicking filling a keyboard events looking you know pulling things off the screen dragging and dropping now all that sort of stuff is is part of what you know selenium enables us to do so with those core commands we're able to do everything that ATF can do and we can go you know further beyond that to anything having to do with any type of form or feel the UI action is doable anything to do with list the core components of ServiceNow is definitely doable but there's a lot of one-off areas and ServiceNow or anyone that's been using it for a while knows clearly there's a difference between knowledge base and a standard form or service catalog and a standard form and so all of those areas get some gets of treatment as well too we build out functions for those as well as you know the reporting tools things like related lists and the chat and pop-up windows and modal windows surveys risk assessment calculator all of these things give functions as well because these are part of your process - if I want to go through a whole change request process and I can't do the risk assessment calculator I'm kind of stuck there aren't I and so you know we've we had to and you know we had to enable all of those pieces and you know frankly after having done this for you know about three years now and getting although the feedback from customers that we've gotten there aren't too many yeah there aren't too many new commands that we add occasionally there's a new piece of functionality and service now or there's a unique way that we find a client has customized the instance and you know we'll add a function in there but you know as far as the core the core pieces that you know affect 99% of all and end processes we've got them pretty much covered here so what's the benefit then for someone if Kathy was built on selenium what's the benefit of going with that as opposed to just kind of building your own tests and selenium if that's something you're able to do yeah you could sort you could certainly build out the same thing in in selenium you could go through and do that however the the focus or the type of development that you're gonna find you're working on mostly is in maintaining those commands and doing that low-level work the way we've built these we've we've exposed it so that people can just kind of go in at a almost like a business level and assemble the cases using commands that match the type of action that they would do in ServiceNow so it's much like with ATF ATF is kind of done the same thing they know what people do and they called it that way you know obviously we've we've done a bunch more but you know writing it at that level allows you to focus on building a test case rather than building and maintaining a framework and I can I can I can give you a personal testimonial from the people that maintain our framework that it's it's very hard to do when ServiceNow comes out with new versions and also patches come out we've got to maintain the framework we've got to maintain it across all you know three versions of ServiceNow as well as four major browsers and also by the way the browsers and selenium are changing too so there are a lot of dimensions to keep track of here whenever a new version of the browser comes out there may be a new version of the driver we got to make sure that those work and those are bundled in there correctly so it's it's a big process there's a lot of regression testing that takes place in there and you know just just say frankly there's there's just a lot of hard work and sometimes some misery that goes into maintaining those functions because it just is so tedious and there's just you know a lot of different things we got to make sure we cover yeah and when you I mean when you talk about that that maintenance I think that's something that we've in the marketing side of frequently called the future proof testing right that you're able to build certain tests and capi oh and you know ServiceNow is is not only are they amping up how often they're doing their version upgrades but they're also going to be supporting the previous versions less at this point I think they're it's going to be you know they're gonna be requiring you to be up to date more often so that's you know something to think about if you're maintaining your own system how much time goes into keeping those tests from breaking every single time you have that version upgrade er or or patch and then if you guys are curious about a little bit more of a filled out side-by-side we do have a handout available in the if you look at the handouts module and the GoToWebinar panel we did put together a little bit more of a side-by-side for Kapo and ATF in terms of you know what what is available there and what are the differences one of the things I know you mentioned is we have a certain amount of commands and ATF has a certain amount of commands I think we're up to about 500 commands that people are able to use so I I break into commercials every once in a while guys that's part of my job I apologize if if that offended anybody but but so now so when you're talking about all of this what's the difference between how when you're filling out the form and capi oh how does that interact with ServiceNow differently than then if you're interacting with a form in ATF yeah so from what I've seen ATF because of the way it runs it seems to be using what I've moved from what I was able to reverse-engineer almost like a JavaScript bridge between the command and the page to execute the command into me from what I saw this was this was no different than going into the page and running a script that uses g-form to set values in the form and you know from if you're just looking at it if you're watching it execute a it looks the same it looks like what a user would do but it's it's it's not it's not actually doing those things on the spring so I'm actually doing that in the browser it's just kind of I don't say faking it's just going a little bit different path to execute it a lot of the same things get triggered that normally would be when you do it as a user manually but it's not exactly the same in capi oh we because we use selenium and it's a completely different approach selenium you know automates the browser by sending events to it so from from the browser's perspective it doesn't know whether or user actually clicked it or what it whether it was done through selenium that's it's really kind it's indistinguishable from from the the browser and the application standpoint that to me is a little bit truer test right that you know you're gonna you're gonna find that there are there are differences between you know using javascript to set a value and actually setting the value you know there's different things that it reacts differently what you're saying there is is in in one situation if a test is javascript-based when it tests the code it's just kind of testing the hypothetical of what that code should do as opposed to if you're running it selenium based it actually simulates the action so it's much closer to an actual human being actually performing that activity is that correct yeah and I'll give you an example things like setting and setting a reference field I can set a reference field through a script on the page and and you know sometimes it will have no problem but I've been we've definitely seen this in practice when I fill out a reference field and I get that pop-up and I need to find that value in the list sometimes it's not there because the you know the the the search is broken or the you know maybe the system is not working correctly I think in a couple of times it it was you know there was additional scripts that were interfering that's something that a user would see that this you know running a JavaScript command wouldn't see so there's a practical difference like that that there's just a difference in how those execute there is a little bit more of a technical reason you know the way that the browser and JavaScript process events you know in and how they were attached can differ between the two and so that you know there's definitely things that might end up not being executed if it was done through you know through you know in selenium or native event and then and then automating it through JavaScript so that's where I also say you know the I wouldn't say that it's not a true test you know to run to run some of those things in ATF but you're definitely going to miss some components or some elements there sure it's it's just more a wider margin of error essentially mm-hmm we had a question so can Cappy Oh run the same test against two ServiceNow releases if I have Subba prod instances at a later SN release it absolutely it absolutely can you know the the the commands are the commands are in or kind of they're independent from the ServiceNow versions they've got an implementation in each version so yeah if you want to run the same test against two completely separate instances with two completely separate versions of ServiceNow as long as the step is the same for both of the instances that test will run so if you're trying to fill out a field that's not on obviously that's gonna fail but if they're in the configuration of the system is is is identical then it would you know it will it will it should execute and pass or have the same result on both both instances okay I think that makes sense so let's talk about about actually running these tests one thing I want to do is oh you know sorry I was thinking about different slide yeah so essentially right now looking at test execution I think that's that's something that a lot of people don't quite think about and I think a lot of people assume that they are able to do something that they can actually do in terms of the way these tests are actually triggered and the way these tests are actually run so can you again kind of break down what you've seen in terms of you know what how the atf user would go about doing that versus what capya would be doing mm-hmm yeah so it's so an ATF they they have bundles right within the application runners and those runners are what is what is used to execute the test it opens up in your in your browser in your session and basically waits for a test to run and it executes sit within an iframe using that the bridge that I mentioned a minute ago there and you know returns the results on on to onto the test that executed the the kind of the drawback or the thing you know I you know it's kind of obvious right there is first of all you've got to be there to start the the runner or have it open on your machine the other thing that was that was kind of a it's a drawback there you know is that it takes over your session so you've really you really can't be doing anything else in that session while that's running so it's got a it's got to be dedicated I've seen some people try to come up with some clever running workarounds for that but at the end of the day it's still going to be you know that it's still going to have that browser up and running or multiple browsers up and running I guess and and there's not really much else that you can that you can do with it there it runs when it runs if your session times out the test does not run some other things that I've seen is if there is a browser pop-up it seems to stop the test as well javascript is not able to to or it's not able to get around those so there are some there are some drawbacks the one the one thing I will give it is that it's incredibly easy to start the runner you just click on it and run it and if you're if you're there while you're running your tests and then everything's great however in practice that's not really how people run automated tests they want to be able to you know they want to be able to start it and schedule it and not really have to have to watch it or attend it so with capi oh we have we have a separate dedicated server obviously it's selenium so it's got a it's got to be separate but we've bundled that into its own server and these are these are these run they're completely disconnected they usually run on a separate machine it's it's very much like a mid server it's not a mid server but it's like a mid server in the way it's deployed and how people typically run it in their environment and you need at least one of these it's got to be there and it's gotta be and it's got to be active and responsive you can't have multiple of these so if you want to run these tests multiple tests parallel at a time here you're able to do that you you know you've seen up to up to four typically people have you know one or two in their production environment but yeah they're able to they're able to run those completely separately and you know the benefits of that obviously are you know you can have it scheduled and run at a completely off time most of the time the the and the core or the the you know the most common configuration we see is that people run these tests full tests every day after hours so that you know they're able to make sure all of the day's changes are in and not that it would but it just doesn't impact any of the other users using the system so that's the the most common one we've got a lot of different ways that the tests can actually be triggered there's integrations with other systems there's events and everything that that allow us to trigger those and you know the the the handling of the results is also automated within our system too so it's a lot more robust system a little bit more setup there on our side certainly I mean you need a dedicated server but you just get lots more flexibility you're able to run it like a like I dunno true engineering you know development pipeline okay so like from a from a business breakdown then the business benefits here what you're saying is if you with one version you're essentially running one computer for the duration of the tasks correct like you have to have one dedicated person one dedicated computer running that test and to end yeah so yeah then the yeah they pretty much what it looks like there yeah you've got it's got a dedicated machine and that it's that person's machine most likely and you know it's kind of tied up and you know that may take to get through everything you know like a full a full set of the test it might take a day it may take 8 hours maybe maybe 4 to 8 hours to run it fully end-to-end which is which is pretty good you know it's a pretty quick test a fully regression test an entire system but there you know there's there's obviously yeah as I mentioned there's all those drawbacks there it's just not you know it's so I good use of the person's time and they did there's there's certainly other things that they would rather be doing there then then watching that execution there and of course if anything breaks they've got to be able to respond to it if the session times out they need to be there to resolve that if there is the pop up on the screen they've got to resolve that if anything else breaks for whatever reason they've got to be beaten up be there to resolve that make sure that that runner stays active capya obviously we can scale it out we can you know we can have any number of machines there and you know that's that's you know that's able to execute it in a target time that you get you so if you need this to execute in you know in a couple hours you can get it down to that point you did you can just scale out to meet that that bandwidth there if you needed to go quicker you can you can you can certainly do that and you know you don't need to have someone there watching it you have it scheduled and then you just look at the results whenever they come in sends you a notification sends your notification and and whatever so that's the the end goal there really is that you know the if there's still some interaction with atf that you need still it's still something that they you know you kind of got a sittin babysit a little bit here maybe the kids are a little bit older and they've kind of babysit there but with with kpo it's you know it's fully under tended and because of the the scalability of it here now you can talk about enabling things like continuous building and I want to run this I want to run this whenever commit and update set which I might do several times a day I know I want a and there's there's all sorts of things with that enables the the results there is that you're you know you're getting you're getting the test results cracker you find out when there's a defect you know typically within within 24 hours sometimes a lot quicker so just to get to a couple of the questions that we've gotten in how does multiple browsers affect each of those different systems so if you are trying to get results for Chrome and edge and and Safari in all those different situations how does that how does that work on each side yeah you can you can configure the test to use whichever browsers you want so in the in the test setup you'll say alright I want this to use chrome and ie and Firefox and you can even tell it the specific versions that you want to use as long as those versions are available and one of the servers it will execute typically when you know when you do that you know when you add an additional browser there it was you know it will double the time it takes for the test to run so if it normally takes fifteen minutes it with with two browsers it will take double that time because it only server only processes one job at a time it's the way it's architected it's set up for that for there's a lot of technical reasons why we went that way but it can't do it there's no problem with running them or anything like that it does it does increase the time so if you're if you've got one server and you're thinking about doing two browsers I probably add an additional server so that it it stays roughly at the same time and and to test those things and then I can automate you can automate each of those browsers sequentially right I'm sorry the what was the question you can all be them so clearly you're able to build the multiple browsers into one plan or suite correct Oh correct yeah absolutely you can have multiple browsers in that suite okay what about in terms of I know that ATF has a batch mode how is that different than you know for for running a test unattended as opposed to what Kathy OH does yeah I mean eight the ATF has the it definitely has the ability to run these things in a sequence so there's there's no problem with you know running a set of tests together at a time in ATF but the same you know the same problems are gonna be there you know it's it's still gonna be limited to what you know what that browser is is able to do if it the browser is not responding and their machine's not there then you're kind of out of luck in that case okay and just so everyone knows if I'm ignoring your question it's possible that I know that we're going to be touching on that information later in this presentation or you know we may just need to get to it at the end I will be running through a few of these so we're getting a lot of questions coming through I'm trying to to work them into conversation right now as much as possible but why don't we right now actually take a look at at each of these two different systems right now so we don't need to spend a lot of time on this slide cuz I know we want to jump into the demo but can you kind of give us the breakdown style of what you see here yeah so at its heart the in the ATF interface there's a GUI driven great thing about that like I said is just you just you know anyone can pretty much walk up and use it and you click the test and it kind of guide you through all of the steps to implement what I would call a command or what they call a test step they've got the Pickers for the values so you can't you can't really put a wrong value in there you know you're going to be restricted in what you can choose there and so that's you know that's pretty nice in terms of setting that you know setting the system up there too and obviously the system knows how its configured so kind of prompts you in the right way for those things which is which is pretty nice capi Oh on the other hand we're we're all script based so on the on the core you know the core of our forms is a test script it is JavaScript based so you know your your devs and devs and admins really already have the technical abilities to to use a system like this but we feel that we felt that the script gives you the most flexibility and it's you know it's it's JavaScript so you do have all of the abilities and advantages of the language itself so if you need to do things like create a loop or manipulate strings or work with dates or do conditions and stuff that's all available within the within the copy of script is as well a descriptive field as well but we did we know we did kind of identify pretty quickly that you know getting started with the script or letting everyone in an organization contribute to test writing just a script filled alone was not going to be the the final solution for everyone there so instead of creating or customizing that form we came up with different ways to build or generate the script so of course you've got the ability to write it from scratch and like that for a developer I think that's that's the best solution there you know you've get you are you know you get the ability to write it there and you know if you're proficient with coding that you know that won't be a problem and you also get the benefits of the code reuse so if you're if you're writing to test cases and you've got one that is 99% similar to the one you just tested but just sort of some different values you can easily copy and paste it or we you know even count abilities to ingest data to configure it drive those tests for those people that are not scripters or for those that kind of new to the system and want to just you know get their feet wet with learning the commands we do have a recorder that's a browser extension that lets you point and click and generate those scripts in addition to those we've also got a case creator which I mentioned at the outset there that lets you generate what I would call like the starter script for many of the cases we've also got a behavior driven development interface for those companies that have opted into BDD to you know talk about that form and years that is an input method as well so we're trying to you know trying to give a whole spectrum so that you know lots of different people can can sort of do the same idea walk up and use it but we want the end result to be script based so that it's you know fluffy and flexible and has all the benefits of of code reuse and everything at the at the heart okay okay so can you go ahead and hop in and kind of show us around both both systems yeah so let's let's let's look at let's look at an 80 F test case here first let me zoom in a little bit so that people can see this a little better so Etienne ITF to the other great thing it's just you know it's almost always just enabled in the instance there and when I want to create a test for this one yeah it's pretty straightforward I come in I define you know what I want my test name to be but you know just for example here let's let's do a create basic incident and and I just save it here after this you know reloads here we'll get the what they call the test steps and I can start dropping in the individual commands to do this automation so if I want to for instance I want to start by going to a new form here it's going to drive me through all of the you know all of the wizards and dialogues to do that which again is a as a as a beginning user that's that's really nice here and the one drawback here that you can see is you know we are you know we're limited to ServiceNow forms and you know occasionally here in the depending on the system these can kind of get in the way of themselves here so I've got a do have to wait for this to load here this one's taking unusually a long time here so when I get into the step and I'm prompted via variables for you know the to pass in the parameters for this step here so in this case I'm wanting to do incident so I'll expand this table view search for incident okay here it is and then I've submitted so I've created this camp this is this is a yes that's that's how I go about adding these steps here so I'd continue in this fashion here defining and setting you know setting these setting these steps up here obviously I'd want to look through this and identify and what are what are the functions available beforehand so I know exactly what I'm capable of doing here but I'm going to continue in this fashion here let's set some values and this was nice here too when it when it comes up it does prompt me for the required field so it knows short description is is required so it forces me to fill this one in here okay I'll just prompt these values here we'll use my good friend able to here submit that so gone through I've set the field values now I want to save the form here and you know in this case here it has a submit form here sometimes you know we want to do a specific UI action and as well here so I may come in here and look through the through what UI action I want to do I like to do the save it's just something that's kind of been baked into my process forever so I can stay in that form here but four months incident here I wanted you want to do a save function here okay how it's giving me that and a bunch of other options here so I think I think it will be that one here let's just go ahead and risk it I'll pick that save there I think that's the correct one but I'm not sure and then finally here let's do a quick validation so that after it saves we know that the you know that the field has has taken here I've got a business rule which which should which should put in some values for for the or a pencil values in the short description let's validate whoops I'm sorry I picked the wrong one here let's go back validate field value we just want to say that our short description contains the word test it should should be part of it so this is again this is this is very easy right if I'm if I'm if I was know if I'd never done this before I you know I'd be pretty I'd be pretty thrilled with the way that you know that this was set up here because I can't do I can't do anything wrong you know it prompts me I have to put in the correct values here for advanced users here you know this you're gonna you're gonna immediately run into an issue here and I've done for I've done four steps here it's taken me it's taken me about you know maybe a little over a minute to do these things here when I what I'm doing you know what I'm doing steps for when I'm building out a test case you know it's pretty routine for me to have 10 to 20 steps in a in a test case so you know what I really want to be sitting here waiting for this dialogue here to load up and and to and to kind of its I wouldn't say it's a hindrance but it definitely makes it a little bit more difficult to to work without you know a large-scale test here this is not going to lend itself to that the other thing too is that you know if I come in here and I want to change values for this because these because the values are buried in variables that's not an easy thing to find out or do or up there you know update those in bulk so we've got you know there are some issues here the the pro very easy to use can just walk up and do it I've got templates so I can drop a lot of these commands in but you know at the end of the day I'm gonna have to go through and configure these and that's where you know some of the some of the drawbacks of the forms and servers now sort of present themselves so real quickly here let me let me do the same test here take some of the same actions here and let's let's write it in in and copy oh here and this is this is our copy a mini tool when I write tests I usually start in here because it allows me to play them back and it's also got all of the the autocomplete and everything that I need in here but opening a new form so an ata if I went through I selected the action in in in capi oh here I'm gonna need to do a couple of things I'm going to need to expand the menu here so I need to from working with this as an end user I would need to expand it I can't click on something I can't see so I need to expand it and then after that I'm gonna want to select the module there too so I need to tell it to click the create new sometimes you can skip this ad can go directly to the URL there is a you know that's another path that I follow but generally for if I'm trying to simulate what a user would do I want to do what they would actually do so that's in the involves going through the menu when I'm on that form and by the way these commands have weights built in them so the you know when service downloads a page here there is a there is a moment after the page is ready and that you know they'll continue to do things like you know run UI policies and client scripts and stuff like that that affects the page right everyone here is probably everyone on the call is probably been annoyed that you know a page looks like it's fully loaded you try to interact with it and then you know the page continues to do things there that you know make it unresponsive when that happens if the automation is trying to run it will fail so we need to wait the proper amount of time but in this case here let's we had set these values right this is what it looks like to set a value in capi oh here I can do this and you know changing these these are just strings obvious I can I can replace these with variables in JavaScript as well or I'm through whatever I need to do they're saving so in this case I wanted the UI action to do that in the the context menu so I click Save there through the content menu I wait for the page to load there's my smart my smart wait and then I do the validation here so I you know I kind of cheated here because we've got some time constraints I did this but this would probably take me a minute to write this now the benefit here is that when I you know if I've got that same test that I need to do a moment later that is slightly different I just need to populate a different caller now I can just quickly reuse this code and I've changed this and I've got me and a 90% of the case already done there so it just takes me a few seconds but I can even bundle these up and create my own custom functions to make it even easier so there's there are some there are some huge benefits of using the script there and then there's you know there's there's still the there's still the the recorder that's out there too and you know for for a lot of cases where I've got to do things that are kind of repetitive or you know I need to do a lot of validations on the form I still use the recorder to myself even though I can even though I can script it out sometimes I just just don't want to type that much here so and just a preface here so the way you were showing us before was sort of a what some people call a pro code option for someone that's you know a skill developer can go in and actually get the most customizable the most unique results possible through going in and entering the code but what you're showing here now with the recorder is something a little bit less technical something for for more of an average user or maybe a business analyst to go in and create those tests correct they definitely edit yeah I mean what anyone that can can use ServiceNow will be able to use the recorder I kind of I kind of backed away from calling a pro code or low code or no code because there is always code there you know there's always code behind this which is flexible and by the way you know I've I've seen non developers get into it get into kapu and be able to build those tests it's it's more you know if you can translate those commands you can you can build it so I've seen non developers come and I specifically had someone telling me they were not a developer and they didn't want to do it but they figured it out and they got actually they got pretty good at it towards the end so yeah I've backed away from it I call it advanced it is an advanced way but I wouldn't say that it's in you know something that you know a you know a junior developer couldn't grasp or anything like that it's definitely doable so let's let's replicate that same test in the recorder here to just capture these events I'll just turn on the recording by clicking that the command there get that button there but let's go about the how I normally would would fill this out here and go through these commands expanded it I've created click new here the page is loaded so I'm gonna go ahead and fill out my fields here it's a table tutor as I do these things these commands are getting captured actually captures captures a lot it captures everything that that's even said on the form here so there's gonna be some things I need to clean up here and then let's go ahead and set the short description and save the record so here we can see that that this has definitely been the quickest of all the methods it's it's it's 1x right and it's as its as fast as I can do it and or the browser will allow me to to execute these actions and then if I want to validate quickly that that display value can go ahead and drop that in here and turn off the recording now I just want to remove any of the steps that I think you know may have been captured it would capture it accidentally here so the collar for the location field will remove those there so it's been it's now it seems like it's pretty much set here I could toggle into the code view which gives me pretty much the same view as what I've just what I created before here mmm is these commands here and I've you know I can take this and copy this script and bring it into the app or I can run it in mini just to make sure that I've got these steps right so this this way again very quick to get started obviously no technical skills needed here but you know when I want to go when I want to go and if I wanted to copy these or do that you know do a similar test case I've got to rerun the steps manually here with the script if I just wanted a toggle or change that I can just tweet that script and in the app or on mini so three different ways to do it and I think there's pros and cons to each of them but the you know the obviously with copyright but you know I felt that that was the the most important thing was that it ended up in script just to have that maximum level of flexibility there now I'll let you know Sal we are getting a few questions now and we had some questions before about how kpo works with ATF are you able to imports ATF tests into capi Oh is there any way that those two devices actually speak to each other or work together I can import ATF cases you know they we have an equivalent command for every test step that they have so importing them was just a matter of translating those between them and yeah that that's that's really the only interaction we have with ATF we don't run ATF command or ATF tests directly ATF is not a runner that we support either there it kind of starts and ends with that import ability and and and that's it here although you'll see in this next section here that part of the reason why I think people end up not doing that is because the the tests themselves may have you know they may not they may not be exactly what they want now with case creator that's something that we have where we're able to just take those configurations that already exist I know you've said that it can kind of pull thing and pull some options in from ATF there is that is that's true as well it can yeah so case creator here and let me and let me let me quickly show what that case creator looks like here for this it yeah there's a there's a couple of different ways to to pull in or create cases here though the most important one here I think is the is the system test they're being able to being able to generate that system test completely here just from the configs around if I pop in the table name and tell it yeah can you quickly give I'm pretty excited we're a lot of us are excited about case creator can you give people a sort of a high-level overview of what this specific feature does yeah so based on a template here these are things that we've designed it will go out and direct pull directly from the configurations in ServiceNow and try to generate the case for you so depending on the depending on the type of template here and but in you know as I mention here we do it the system test we can create unit test we can you know can look at workflow context or import ATF here these are the core templates this will sort of drive what type of cases get created and then depending on what you selected here we can you know we can optionally configure some additional values here to tell it exactly how to create those cases so in system test for instance here I want to pull the config for each of the each of the each of these tables as it relates to the Quartier table incident here and I want to just go ahead and in generate these while it's running here it will go out and look at the configs it will look at the scripts and all of the metadata and try to come up with and generate a case here on the fly for us we can see it came back it pulled 46 of those config here and so now it's this is this is showing me everything that was configured in the system and if I go into one of these into one of these these cases here we'll see that the results this was one to validate a form layout so you know you can imagine if we were to do this in ATF here with all of these different all of these different configs here this this would certainly take a while and even in copy o to do this manually would take a long time these is very redundant work although it's not terribly complex here you know we can most people that are on this will will definitely understand this right away here but you know and they know or doing it's still tedious to create something like this so nothing you said you grabbed out 46 cases and you just built those pretty much instantly yeah what's within a few seconds here that's something like that will be created so you know form layout we can you can usually we can usually get pretty close on those things same same deal with UI policies right UI policies are basically config driven and so generating a test case to validate these is it's pretty straightforward as well where it gets a little bit more interesting and kind of theoretical or the the things like business rules things that are script driven because the scripts are infinitely complex here right we have no way to know what's going to be in script so I you know and I had to create a system that will sort of look at the intent of the script in pick out actionable actionable commands and create the script from that and in this case here I'll tell you I'll tell everyone here my goal was not to it was not to get it perfect because I know there's there's going to be there's going to be a level of customization that I'm just not able to figure out a reverse engineer from a you know a system like that but if I can get this being a 50 60 % of the way there for you if I can cut that work down in half then I think it's I think it's a pretty good pretty good compromise here so this is one that was an advanced business rule that was just setting up a field here it's just a dummy business rule and this is what the result looks like for that directly from the script creating a test case for me mm-hmm and what if these steps change if if these steps change here so well there's I guess there's there's a couple options here one you can you can come in here and manually change these steps right so if I know that you know I don't want to do save anymore I want to do update I can come in here and you know I can change that manually there we do link the test case itself to that configuration record so the the source record here which was a business rule has been linked to this test case and the way we set it up we can actually we can actually track when this script changes or runs on a schedule and it just just looks to see if that if that script changes you know we want to update the we want to update that or send a notification to that test case that this that this case should be updated as well so here's the here's the rule for instance here this is the the business rule and here is my set up and everything and if I go to my my script field here the second to load here's that here's the you know here's the script that it pulled that out of and it's set active to false let's just say we you know actually we wanted to set this to true and update this now they're never there would normally be a scheduler that runs through and tracks this change and alerts me on this business rule but the end result for me is you know I could come in here and I could manually change it or if I really wanted to I could just hit this regenerate test case button and or link there and that will that will go out look at the new configuration and pull back that that newly generated script for me and we can see that that's reflected here so pre you know and the surface pre it seems pretty simple right there's a there's a lot of logic in there to be able to figure out the you know what to pull out and how to parse those values but the end of the day that's you know then you know my goal really is to make this as simple as I can for the developers this like the system tests gonna be a lot of tests and they are gonna change you know they are gonna change weekly or monthly so you want to you want to be able to keep track of these and make sure that your test cases don't get stale now guys I understand we're at the top of the hour right now what I want to do is I want to just kind of finish up we have one basically one more section here that we can get through which shouldn't take us very long and I understand that a lot of you been asking questions if you're able to hang on with us that's great we will address every single one of these questions that we can and if you need to drop off we understand that too we will be sending out this recording to everybody you can also find it on our YouTube page or if you follow us on LinkedIn or any of the socials Twitter as the other one that we really use very often we do posts some of our activity on there from time to time so if you have a chance to follow us there that would be another way for you to see some of this type of activity that's happening but you know as we said at the very top of this one of the things that people have a problem with is just understanding the logistics just getting getting started with with ATF and getting started with Cappy oh so can you address really what are what are the differences for someone that wants to get started on either one yeah well I mean with ATF you've got everything that you've got everything that you're you're you're gonna get to get started here you know I think I think the approach with ATF for me if I you know if no other tool existed would be to just identify what what type of suit you know what system test I can create you know there's gonna be some limitations there I can't I can't do everything that that I want to in ATF there's gonna be some configurations that are just not able to be testing so I'd want to identify and just make sure that I understand the scope of what can be tested in ATF as far as the system test goes and then and then just you know kind of just using that walk up and you know walk up and use it approach just go through and start creating those and you know I probably would have a list of things that I want to do and just kind of go down that that checklist and you know just hope that I'm able to get you know 50% to 60% of my things automated in there I know there's gonna be gaps and I'm gonna have to augment it with some manual sting but that's that's really the approach that I would take with ATF in Cappy oh you know because we're it's it's obviously a much a much larger solution there you know there really isn't I'll walk up and use I would never I would never tell anyone to just walk up and start using it and create cases when we you know when when we've you know what do we use the system and when you know from our experience we've we've kind of identified the best practices of how to try to integrate this into your system and to use it and there's really there's really a bunch of different levels that we we would need to go down from the the top process level all the way down to the workflow level and we've you know we've identified those through you know well to be honest from from you know from just from doing it and using it and seeing what works and what doesn't work we're able to you know we've been able to find out you know the best the best strategies for each of those levels and we've built the methodology around that what's called 360 360 testing or 360 methodology we you know we do that and we when we onboard people we try to you know try to show them that that process there of course it's gonna adapt to whatever people are doing so you know there if people have an opinion about what they want to do we can adapt to it there's no problem there but for those that don't really know where to get started we and we certainly have recommendations for that and the tool supports the you know the tool supports that process I you know try to make it open and integratable into anything and so we know we have our recommendation and you know we can we can adapt it and you know just to give you just to give you the the flow of what our our methodology looks like it starts with the testing process which is in my opinion no different from any other business process you know you wouldn't just go in and turn on incident and deploy that and have people start using it out of box you want to understand it you want to you want to understand the philosophy behind it that's why we have you know you know I mean business process workshops and stuff like that so I think testing is the same there are there are testing processes which we know which we've identified and you know we wanna we want to give we want to give the whole team the view from that from 30,000 feet so this is this over here on the left is kind of a sample of what of what that looks like here with everything built-in and depending on your maturity and what you want to do it may be a subset of this but this is what it looks like you know I'll put together here and so this is what you know this is what you know is you know would be high-level and this would also all dictate manner how this how testing integrates with other business processes how does this affect change or release or as you know sdlc and these are things that we can talk about at a higher level independent of that and kind of went kind of underneath it are the testing strategies for each of the individual levels so an acceptance test is not the same as a system test or as a unit test they all need their own unique strategy the who what where why and when of that testing you know what are the inputs what are the outputs when should we consider it done how are defects remediated those are all things that need to be discussed upfront the tool can be augmented to support those and and you know really it does support a wide variety of those underneath that now that we have the philosophy how do we actually you know go and organize a test plan clearly you know if you saw from case creator I have a recommendation it usually you know just write description of it is enough to bore people but there's you know there's a lot of detail just in the test plan keeping those organized is really important because if you and if you can't find it if you don't know what's where just builds up this anxiety about test automated tests and once you get that really decreases the the confidence in the test so having those nice and organized and having a plan is really important below that how do I design the actual cases what do those look like I've definitely seen this when we on board people we do we do show them our you know our I think there's about 30 or so patterns that are really repeated in a lot of cases and so we show people those and you know understand how to use those and then finally at the lowest level here and then I'm getting this is a this is this is going to be more technical for more proof than most people want to get into but there's a particular workflow for how we build these things from the tools that we use you know Cappy Oh Minnie which we've built or you know your text editor and and you know the whole workflow I've got a recommendation and how I do it I even show people you know how do I lay out the windows and you know switch back and forth so that I know that you're working the most efficient way each of these has you know each of these has a purpose and you know there you can you know you can't you can definitely you can definitely go into this ad hoc and start to build tests but I will tell you that almost everyone that I've seen whether it was manual or automated whether it was ATF or another tool if you don't have a plan it's it's most likely gonna fail you need that you need to know what you're doing you need to you need to have some you need to have something that you're working off of even if it's just an Excel spreadsheet with you know things that you want to automate have some sort of plan and you know here our methodology is you know everything and you know you can you can kind of pick out what you need from this but definitely don't just go into a blind here have some have some sort of process and these are all the levels that that we've identified great so what I want to do right now style if you have a few more minutes I do want to try to grab some of these stray questions that we weren't able to to answer right now we had a few that a lot that came in in the last 10 or 15 minutes so if you guys are still on with us you know thank you again for giving us your time otherwise if you've dropped off and hopefully you're listening to this on a recording we're going to get to your questions right now so one question that we had is how do you parameterize the data the data in the scripts yep so so our system within our system we have what we called the shared context it's just it's it's an it's a variable that's accessible between all cases and and that allows us to to plug in variables into our scripts the data plugins and there that exists at two levels that exists at the the case level that allows us to specify data for a specific case we can also expose it at the test the test plan level those that data that gets input becomes a variable that we can we can just use natively within the script so yeah that that can come in from a couple of different ways with the test case level the one of the one of the cool features of that is that if you specify multiple sets of data the test will repeat for each of those tests or each of those sets of data so if I have a lot of different permutations that I want to do but the step stay the same I can parameterize that and use variables and use that data driver to just replicate or rerun that test multiple times so yeah that's that's theirs and that you know that that data at the test plan level that can also come in a variety of ways you can hard code it in we've got kind of like a little excel grid type layout there it can run from a script to so if we want to pull this in in real time when we execute the test if I want to pull values from the system or generate fake data whatever you know whatever I may want to do that can come in and a variety different ways there too but they're all just variables and just you know it kind of goes back to the heart of in the fact that it's in JavaScript there gives us that's that flexibility and how are the case generator or the case creator cases defined are they defined dynamically or is that something that is controlled in a configuration the cap yo manages sort of sort of both their you know it is dynamic based on the configuration of the system however it definitely follows more or less a pattern so each of the types you know every every type of configuration and service now what we found was there you know there was a best practice for testing that type and the pattern is the pattern is derived from that and it adapts slightly so it knows you know if a test if a business rule can only run on insert versus update it knows how to toggle in and you know it will change that the test slightly based on that fact and you know the conditions the other conditions can drive that as well but more or less it's it's that pattern it you know wants to find the conditions for the test and then it wants to find the actions for the test and it assembles the you know the correct the correct ones based on the type the type that it is UI policy versus business rules Oh on mm-hmm now we had someone that said they were trying to build some API tests in selenium but they weren't able to do so and they were curious if kpo had the same difficulty or if it was able to test api's within that platform yeah if by API is you mean like the restful endpoints then then yes so we and we're not using selenium for either so part of the benefit of the server is that I mentioned there's like some flexibility there we can expose other types of functionality in our tests so we can go beyond just the functional test in selenium and we we do expose the you know restful type interfaces so that if we want to test a phone on a test like the ServiceNow endpoints or a custom endpoint that I created or maybe I want to integrate with another third-party system we can use those those low-level commands in there to do to do those actions so yeah rest the restful interface was one of them there to the we can do other things too you know they send an email that's not a functionality of selenium at its core we've extended the framework to do that we can run things like PowerShell and then another client of ours just recently had it need to validate things in Active Directory so certainly not a functionality of selenium but we were able to build those commands so that they can use those side-by-side with the functional tests because sometimes it goes beyond servers down the browser and we wanted to be able to you know we don't want to limit anyone certainly we need to selectively bring those things in but we want to you know in those cases we were able to extend it to to give people that that ability okay great well guys I'm gonna probably call it right now it's a it's about you know just close to 15 minutes past the hour we appreciate everyone that stuck around with us if you did not get your question answered but you have a burning desire to scratch that itch let me know you guys can email us directly at what you see on the screen here and let us know that you have that question and we'll be happy to follow up with you or like I said you can follow us on our social media LinkedIn and Twitter and contact us that way and we'll be more than happy to interact with you and get you all of those answers to those questions but Sal thank you so much for being generous with your time talking about ETF and Kathy Oh today thank you to everyone who joined us today we appreciate it as always we will be sending out this recording in early next week to your email so definitely make sure to check that and hopefully you guys have any other questions reach out to us I will be happy to talk to you about anything ServiceNow related and talked about what else we're doing it at Serna solutions so thank you very much thank you - appreciate it alright everyone have a great day