today we are going to be talking about mastering the service now upgrade hopefully being able to help you go from perhaps if you're part of the n minus-1 contingency and you would like to get to an early adopter these are going to be some of the steps you might want to take but at the very least trying to take the headache out of the upgrade process trying to make sure that we can identify any common pitfalls any obstacles that people normally see and what we can do to really kind of beef up the your your upgrade game I want to get a few things out of the way first by way of housekeeping if you're not already familiar with Serna solutions I've posted on here just a few of the areas of expertise that we have on the ServiceNow platform a lot of the reason why people work with Serna are the reasons why you call an expert to do anything really because you want it to still work the same way it works now as it does years from now and part of the way we've been able to do that is through hiring the right team hiring people that have the experience to build something in the long term and not just for what you're looking to do right now and sort of work as that expert guide through that process and make sure that the way things work now are the way you're going to want them to work years from now so you don't have to redo that you don't have to put in a lot of extra time and effort to fix what you did from years ago and that really does segue well into this whole upgrade build something the right way in a way that prevents problems online and that makes sense I also wanted to point out a couple of current event pieces of news for you ServiceNow has published a number of applications and solutions based around the recent co vyd situation and so I will go ahead and put a link in the chat right now they did ask us to help put together a guide for their emergency outreach deployment and we did go ahead and create something there there's a number of other apps that they've created I've listed here so if you follow the link on the chat that will bring you to our guide as well as links to those other applications if that is of importance to you but I want to make sure that we pointed that out and put that in your hands if that's useful for you some other points if you have any questions during the podcast make sure you ask them right away you don't have to wait until the end it's very helpful for us often to hear your questions as we're talking about the subject matter and that will also help us steer the content what's most useful for you so if you're hearing something and just have some questions or you want to hear it in a different way and hear it from a different angle make sure to let us know that and we'll be happy to try and integrate that into our conversation if possible um another question that's usually asked you will be emailed the recording of this event as well as the content files however if you want to see other episodes that we've done or just be like checking in in the near future and see what else we've done you can go to certain solutions com to find other podcasts that we've done or you can go to our YouTube channel which is another great resource for content that we've put out as well as our linkedin which you can go ahead again all of these links are in the chat right now if you go there and and follow us that way that's a great way to keep up but right now I'm going to go ahead and introduce around the circle say give me your name what is your location and give me a little bit of your background with ServiceNow sure Tim oh I'll go first my name is Jeff Marlowe I am located in Chicago Illinois I've been working on the platform as a developer for roughly 11 years so in those 11 years I've worked in most of the modules as you can expect worked on many many upgrades so bring a lot of great experience as well great Alejandra I am Alejandra chickie I am based out of San Antonio Texas and I have a little bit over five years experience with ServiceNow mostly focused around ITSM and HR but I do have experience in other applications in other areas answering a solution specifically I am the VA administration lead so I get a lot of different types of requests it's a lot of different kinds of exposure and then I also have experience with automated testing and how it relates to upgrades all right great Brett awesome my name is Brett Ishmael I'm located in Fort Collins Colorado I have about five years of experience as a developer on the ServiceNow platform couple more as a customer before that I've done upgrades ongoing support implementations for various customers of different sizes and I've done working mostly the ITSM suite as well as HR and kind of but poking around and everything for a while now so yeah great yeah and this was a great team I think for us to put together around this conversation I know that you know many of you guys have worked on on you know updates through the years definitely seen kind of the ups and downs of the up you know ServiceNow upgrade process so I'm really excited to have this group here and thank you guys for joining us first thing I want to talk on a little I don't want to drill this into the ground cuz I think it's a concept that most people understand but I think it deserves some attention and a little bit of detail can we talk a little bit about n minus one which is you know right now the requirement the service now has for its customers to be either at the latest version of its product or the version just behind that and one of the things I wanted to ask right up front is kind of as an ice breaker what do you guys see as how this is going to affect ServiceNow customers yeah so I think the the fact right we're on as of this this this podcast the market launch releases is Orlando right so n minus one would require that customers be on New York I think the force mandate of doing that I think is requiring customers to to upgrade on a more regular basis you know I think the the cadence is roughly two releases per year so that means that the bare minimum clients are required to to upgrade at least yearly I think that's that's forcing clients to you know truly put their test cases together and and upgrade where maybe previously they they had delayed maybe a couple of releases yeah but it's beneficial right I mean I think that the idea that that service now obviously they it's it's so much more than it used to be it's it's not just for IT applications anymore it really is across the enterprise and so the idea that they are supporting such a large product in multiple versions you know that's that's most likely their reasoning but there are benefits to the customers as well right I mean can can you speak to that a little bit more yeah it's critical I think we're you know in the 11 plus years I've personally been working on the platform and us as a team we've all experienced this it started off as a very small and a niche product whereas now it's it's it's a very robust platform really and so the features and functions of of the platform are really developing at a rate that require consistent consistent upgrades and consistent releases to really support the support the the new functionality it's it's really essential you know we've all been on on releases where there there bugs in there there patches that have been introduced with their problems right the only way to to have a consistent fix or implement consistent fixes as through release management consistent release management mm-hmm and I'll put this to the rest of the group why do you think customers wait why do you think that for the most part they they don't jump on the the newest version as soon as it's out I think depending on the you know size of the environment and the level of customization that they have it can be somewhat of a lengthy process so it can be a little bit painful to upgrade for every variation or maybe even once a year if it's an upgrade that takes a client you know two months to complete because you also have to take into consideration the development freeze that happens during that time so no new enhancements can be introduced by the development team while they're upgrading yeah I've seen other issues like acquisitions large projects that are underway as well so timing can be an issue you have a large project that's lasting for a while you don't really want to enter into a code freeze mid project and yeah my most recent client in acquisition is something that was making them hesitant as well especially if you have several different ServiceNow instances that are going to be playing with one another you know it doesn't make sense to wait sometimes two to merge them or something along those lines before upgrading after that and I think another another big thing to write as customizations and I think that's when the technical that you've incurred throughout your life as order threat your ServiceNow instance is is realized especially is when when upgrades occur right you you start having to pay pay your dues right if you've customized a modular application extensively upgrades to really where you where you have to to vet that and work through that technical debt sure I think that makes sense so I that that leads us to kind of you know what is the process here you know we've you know we work with a lot of customers who are trying to either get this done themselves or looking to bring in a partner to help them with their with their upgrades because it can require a varying amount of help and like you guys have said a couple of times the customization area ServiceNow is trying to get everyone out of the box as much as possible but I can think we can all say with certainty there's not one customer who's 100% out of the box that's just not the way she's not the way that works what about in terms of that preparation process I think this is usually kind of an overlooked part of the upgrade but Brett can you kind of talk us walk us through actually you know what I forgot to bring up one other point everyone who is in the audience right now attending we do have this guide I don't want to bring that up before I ask this one question oh well you guys can see this now and this is in the link and this will be emailed to you as well or if you're really quick you can go to this URL go to our website to find this but we have put together an expert guide for you in terms of helping you get through the upgrade a little bit more diligently it goes you know beyond some of the other documentation that you might find from ServiceNow but this is definitely a resource for you as you know either as as you're following along with us and have some questions this way I want to take this back to your team that's definitely something I want to make sure that everyone had had access to and was made available to you but let's talk about getting ready the the preparation process Brett what what usually is involved in that process sure thing Thanks so several things to kind of kick it off right firstly you're probably going to want to identify who's working on the upgrade itself from a technical perspective so maybe your developers as well as you know some of your key stakeholders people who have some skin in the game people who know your what's on the platform and what's in use so speaking what's in use it's good to know what you're using so what modules and applications do you have so when you look at it release notes from a service now that's kind of how they have it broken down as well you know what module how has it affected what within the module is being affected so therefore it kind of starts to pave the way for you to tell what you need to look at and you need to verify what you have is working the same way that you would expect it to or some of the new features or do you want the new features so which of those features do you choose to include or not to include during your upgrade so that all being said you go to your release notes at that point you analyze those you start breaking it down into a development chunk that you can look at with your developers and say what things need to be fixed and/or remediated potentially before you would even kick off some test process right so then from there you're basically going to go through and you're going to clone down your production environment over a sub production environment so that you have a good test route something that is as close to production as possible and then you go ahead and you upgrade that guy you take all your notes that you've made with your release notes compared to what you're currently using and you start chipping away at it you go down the list and say what functionality is working now whether you use specific test cases if this might be an early phase of that but you're gonna want to go through and just do baseline verification are these things working the way that I would think kind of a look feel and while you're doing that your key stakeholders are going to be putting together something like a test plan which of course is going to include things along the lines of code freeze so that way you're not having people developing new functionality while you're also trying to test out to verify that things are indeed working the way that they're supposed to be so once you kind of have that laid out that's when you you start to really piece together your full-blown test plan is the end user going to be able to test this all the way from A to Z successfully what does that look like who's involved in that so your testing team are going to be the pea that are running through that though through those test cases and you know or something like ATF so automated testing framework or in our case Cappy Oh which is our preferred method that can kind of start to tell us what things we need to go over mediate and or fix as a technical team and yeah then you kind of just kind of rinse and repeat from there you know you test and you fix and you go back now you mentioned before about the the release notes and I obviously you know we've in the in the chart that we've included here it has a link to the Orlando checklist which which we've also mentioned and that's the number one check item is make sure that you've gone through the release notes how big of a process is that what's usually entailed there good question so that can take varying amounts of effort right if you the more advanced your client is that the more applications they use that kind of grows and scope pretty quickly especially if they have a lot of complex integrations or customizations as Jeff said earlier you're inheriting technical debt so that really comes into play then so obviously as a rule the less you're touching but the less time is probably going to take to fully test so basically you're going to go through as I was saying before module by module and you're going to compare the release notes against your current functionality to see what differences there are things to look for understand and to be able to fully vet and test and for me it helps to do that so I can start to break it down section by section which also corresponds to your key stakeholders hopefully to be able to understand what needs to be updated and/or fixed before you even get to the point of of your tests now looking at the the teams that we you sort of touched on near the beginning in terms of how to how to put this team together I think for the most part this is pretty cut-and-dry right I think that you know who's going to be on your technical team you know who's going to be on your project management generally the managers and owners are going to be part of the stakeholders how do you determine who to put on to your testing team in my experience the key stakeholders will probably know your most of the time going to be going with your your subject matter expert experts or people that are currently working in the system who really know and as we're all working in this we start to get a feel and we understand who these people are if it's incident management while you're probably going to have some sort of team lead who fields a lot of the incidents or knows how it works on a deeper level I think it's important to stress the difference as well between a customer who actively uses this tool versus the developer who builds in it the developer kind of has confirmation bias in a way when they start to develop fixes or things they know what they expect they know how it should be used so they're in a way when they test it they're looking for that which is why you need people who are in the tool actually using it as an end-user might be using it to be part of your testing team that's why that's separated out there and it should be tested for those that both angles really but fundamentally the customer it's going to be in their hands so you're gonna want to have the smees and the people that are used to seeing it every single day so as well as people who can go through and approve requests without needing to call on somebody else to say hey can you kick this off for me can you nudge it along type of thing so people that have responsibility for the product people who know about it and the people who can go through and test it fully and and clicking all the buttons is what you're saying like the boots on the ground people that need to this application to work the way you want to see it at work what I mean to the rest of the group I mean how how long does this process the planning process usually take because it's it's my understanding that this is probably the easiest well one of the two of the easiest parts of this process to overlook so how long do people investing in into that time so so that the planning itself can take it's iterative right so if we're doing as a service now recommends typically two releases a year you generally can can figure out the the planning or flush out the planning over time right but it can take you can take anywhere from you know week or two to months but calling back to Brett's point about release notes right reviewing the release notes that's something that needs to be done very on a regular basis right as as developers it is our is our obligation to stay on top of new features bug fixes patches things like that so that when it comes to you know market launch for Orlando we're familiar with some of the preceding fixes and release notes so that it's not you know drinking through a fire hose type scenario and my experience yeah can be anywhere from from a few weeks to two months to plan but it's it depends on the complexity of the environment greatly and you know we've mentioned that not everyone really can be an early adopter right it really does depend on the maturity of your of your service now practice but when and we've also mentioned how it's I think it's pretty common knowledge that it's a good idea to wait until maybe a patch or two before you actually starts to consider the upgrade which that's all part of the the guide that we added in here as well but when when is the right time to start planning do you plan as soon as the release notes come out or do you plan after the patch is start to come through when when on their timeline should they begin this process as early as possible usually you can only suggest on the fly you're going to need to adjust on the fly as more releases come out really some patches come out you're going to need to look at and evaluate those to decide so for instance if you're saying okay New York patch two is what I was going to go to or orlando patch one by the time you're implementing it chances are there's going to be another patch out that might actually address a major issue that you were running into so you need to be vigilant and make sure that you're keeping an eye on those notes and those patches specifically because those ServiceNow text they're creating problem cases and the resolving a lot of these issues for you you know and you can actually work with them through hi to help you with these types of things as well so it is really vitally important to plan as early as possible and to stay on your feet and be willing to adjust quickly as well to help resolve any potential issues that you'll run into during your upgrade process I think the only the only thing that's rough with that answer is when we tell people as soon as possible the idea is that no one's going to do it as soon as possible right I think that that's it's just this good habit that people should do it kind of gets swept under the rug and so really if that's if you are starting to see yourself get panicked during the planning phase you know definitely it's it's worth considering starting that process earlier than you're currently doing it and maybe developing the right team to put in there but something else I wanted to touch on I think that probably the second pitfall that people run into with the update process is they don't quite consider how long the testing process is going to be or what's going to be involved with testing how much planning goes into this the tests process and we hear a lot of the time that people who work in the testing department say look we're the last team to be congratulated for a release going well and were the first team to get scolded for a release going poorly so I kind of wanted to throw that out right now I mean when you know I'll talk to Alejandra about this what is you think what do you think is the reason why testing is so often overlooked or I think part of it is that people have a different definition of testing you know like a developer it's it with the example that was given earlier between just developers and like a business user the way we test is wildly different and I think that could have something to do with it so maybe as developers we're assuming that a stakeholder is going to test out all of their processes and that they know what those are but that might not be the case or they might run into issues that are actually enhancements and not related to the upgrade so it's it can get a little bit blurred with testing mmm so I think it's important to have a test plan and like have all those details laid out at the beginning yeah one I you know I want to kind of jump back a little bit sorry for the clumpy nuts with the visual but I mean the testing team is really the largest team that you need to be working with and that's it's probably it has the most variety but also the least definition in terms of what's going to work best for when it comes to testing I mean what's talked a little bit more about you know what are what are some of the processes that the customers should be doing in terms of planning out you know planning out the way testing should be carried through during this process I think identifying your key stakeholders is very important and making sure that they understand the area that they're responsible for oftentimes I've seen they usually delegate testing to other users that are very familiar with the processes like I think Brett mentioned earlier the ones that are in the platform that are you know actually going through these processes on a daily basis tell ya I that's it's definitely an important step so why don't we do this now I think that um something that's worth pointing out here is the difference between automated testing and manual testing right I think that for a lot of customers specifically ServiceNow customers automated testing is something that has taken a backseat I think we it's something that's becoming more and more common in the business world and the business IT world but it's still not something that has really full adoption from ServiceNow customers right now and so they have their tool they have you know the ATF automated test framework we also have a tool called capo that we had actually develops we were one of the first automated testing solutions on a market for ServiceNow um what is that I guess one question I have is what what do you think that the obstacle is for professors and this is to the whole group for customers in terms of making their upgrade process easier using the automated testing tools available to them I think and anyone else can can add to this as well but I think part of it is that a lot of customers don't think about testing until there is something to test like an upgrade for example so taking the time to develop all of these test plans and all of these test scripts in preparation for the upgrade can kind of seem overwhelming or too time-consuming so I could see customers shying away from it in that perspective and then you know they could probably have the good intention of revisiting it you know for a future upgrade but it probably gets put on the back burner and doesn't get revisited and so you know the next upgrade at which point is probably too late to develop all those test cases so you know if if the regular testing effort is we'll say a and the Ottoman the effort to implement automated testing is a plus B don't want to go through they plus B even though it means they're only going to need to do you know D or smaller every single additional time okay yeah I think that can be a little bit difficult to see the long term especially if you're just trying to stay compliant if you're just trying to get that upgrade done before you're you know out of compliance so don't you don't lose service no support or something like that I could see them kind of you know putting it on the back burner and and just holding off until the next upgrade if they're like on a time constraint sure yeah I mean resource allocation is a big deal there I think right because they're looking and they're saying well if I'm going to use this person for X number of hours already then why don't I just have them test it and it is again looking forward to to understand that you're gonna be saving that later but it's hard when you're just trying to on a spreadsheet say how can I justify a plus B as you said earlier so I think there's that as well as potential technical limitations as well atf can be a little confusing at first especially if you're not very technical it on top of testing your upgrade now you're looking at learning something new too so i think there's a little bit of fear there for people to embrace something new that you don't you don't necessarily know that it's a guaranteed result to save you time I would argue that you know statistically it is and that people benefit from it across the board but I think those two challenges are almost too much for most people to take on at least right now until it gets simpler sure why don't we do this I think it might be helpful for anyone out there that maybe has not implemented automated testing yet to get a kind of a first-hand visual of what that looks like Alejandra would it be okay if I make you the presenter right now yeah of course um okay so let me share my screen here can you all see my screen okay sure can okay cool so earlier we touched on the importance of test plans it's it's definitely true for manual testing where you have a document to to kind of outline what that test process looks like all the different roles and details about what's being tested so that it capi oh is pretty similar in that sense in that it starts with the test plan so your first step in Caffey Oh to creating your test scripts is to create plan and this is usually broken down by the application that you're testing so you'll have a plan for incident a plan for change a plan for problem like you can see in my example here I have the incident plan and then the next level to the hierarchy is your suite which is basically your different processes for each plan so in my example here for incident I have a basic incident process I have a first call resolution and then I have a child's parent incident process so I'm going to go ahead and click into the basic internet so as I mentioned sweet is your second level of the hierarchy and then cases are your third and final level so these are the more detailed records within your automated testing which is basically your steps of what's being tested yeah and all you know your steps for that process so I'm gonna go ahead and execute this and then come back real quickly this this might seem kind of technical and we I know that usually this this type of event draws in both technical and non-technical users and there is going to be an element to this that we're going to highlight for some of the less technical people in terms of how this directly applies to the work that they do as well so I just wanted to didn't mean to do Riel you there might want to do to bring that up real quick is that as a heads up to the attendees yeah definitely so one of the really cool things about Cappy Oh is it's you know it can there's different tools to accommodate different technical levels so you can have a developer go in here and know the ins and outs and create scripts just off of scripting and an experience or you can have a business analyst go in here and use one of our UI tools to create some of the testing according to the process that they're familiar with so yeah and I believe we might walk through that a bit later it's time allows yeah can you mean so what we're seeing here I mean with with the automated testing process so the idea here is that you're you're still building the tests are still basically doing the manual the very first time you do this you're still going through the manual process of creating a test right I mean but doing is you're doing it in an automated way what about in terms of [Music] I'm just trying I'm trying to think about this well I guess you know what what what are the differences for customers in terms of you know what what kind of changes do they generally see or what kind of time savings do they they see in this process so it's really cool because kpo can cut down your time to test by a lot so in a reason in a recent you know client experience we had a client that was you know it took them like roughly 200 hours to test a process so or not a process excuse me to test their ServiceNow instance for an upgrade so they had five people doing testing for a week full time and with kpo after you know they had all their test scripts built out and everything they were able to reduce this to just a couple of hours another that that's that's not necessarily you know the way that a result that every single company is going to see I think we've seen some companies you know turn forty hours into an hour or some companies turn 200 like you said it really can vary but it sounds like at the very least it's a usually a pretty drastic change what do you think meaning well before I get into that why don't you can you actually walk us through the process of creating creating a plan and kind of showing that off a little bit and this is really where I wanted to make that note that for the non-technical crowd this is actually going to be sort of a non developers path for for doing this so if you're able to do that that'd be awesome yeah of course and then just to add a little bit more to the value out and the the time-saving that you get it also makes retesting things a lot less painful a lot less of a you know time-consuming tasks so when you have defects fixed for an upgrade it's as easy as just executing that suite or executing that process to test it versus having to you know have someone manually test it and this is also true for patches so it makes testing patches you know pretty easy to do and not time-consuming at all okay so what I'm showing here is the list view of plans and you can see we have a button up here available to us called test designer so I'm going to go ahead and click into that and this is our Capua robot here and he is loading up the UI so this is the UI that time you'd mentioned that makes it easier for any technical you know any users of any technical level to go in here and create these test plans so I'll go ahead and just to be consistent with my example I'll create an incident plan you do have two different types of testing you have acceptance and system testing otherwise known as unit testing I'm going to do acceptance which is the end-to-end process testing for the purpose of this test designer demo unit testing is also available and it's very handy and for testing your configurations so here you can see you get a screen to fill out your environment if it's not already available in the system you can create it along with the test account I'm going to go ahead and pick a environment that's already been created with my test admin and then you can select the browser that you want your automated test to be executed on I prefer Chrome but they do have Firefox in IE and then just as a quick note you do get a navigation on the on the right that basically just walks you through what you can expect out of the test designer so we're about to complete that and triggers and notifications another wonderful thing about capi o is that you can set it up to run on its own on a schedule which is very useful if you have a planned upgrade and you want it to run you know say three or four hours once your upgrades completed I'm setting this one for daily at 8:00 okay and then the sweet setup is really neat because you can select a reference record you know say you're testing your parent-child incident process and you have a record that you want to base your test scripts off of you can select that here I'm just selecting this demo incident and then some additional configurations I prefer the screenshot it's a very very handy tool to use for troubleshooting or for whenever you have a failed case this is basically provides you a screenshot of where the failure happened so you can see whether it's like a hidden field or maybe a tab that didn't get selected okay so this is your review of the test designer and I'm going to go ahead and select create okay so this created my incident plan according to what I selected and it created the suite named it after the incident that was selected but you can definitely rename your suite here from here you can use case creator to create your cases so this is another really handy tool and Cappy oh that basically creates some you know some pre-configured script for you so that you can use it as a base for your test cases so you can see here with the case creator I have four cases I were created can definitely rename these according to each step in the process but you can see here we have some script creative for us based just on that on that test record so this really empowers you know users or business unit users that don't have a lot of technical experience it really empowers them to create a lot of the test scripts without needing help from a developer so yeah essentially it looks like you you were able to create both all a plan a suite and four test cases really just through navigating ServiceNow a configuration level uh-huh and I was just based on based on the existing script that already existed within the ServiceNow or any other customizations or changes that were already in there what about in terms of you know that we have we've mentioned it already it's worth worth pointing out what is the difference between for ATF you know when you're using ATF for an upgrade what are some of the the differences that people should expect there so ATF is different and that it's more like gui-based so it's it's like more based on UI versus 'm like the actual scripts that you see in the cases it CF is tricky though because if you're if you're upgrading your environment and you know say you have like five incident processes that fail when you run your ATF scripts it's a little bit difficult to tell whether the failures are related to changes in the process caused by the upgrade or changes to ATF caused by the upgrade so that can be one of the challenges yes so like ATF is is is part of the system that's being upgraded and so when you're running the tests and and and changes are being made to it it's hard to tell you said where those are coming from okay something that I thought was that I know a lot of our customers are really interested in with McAfee oh is the idea that not only do you not need to be a developer to create those cases but create a whole plan like you just did you don't have to be you don't have to know best practices for testing because that's already kind of baked into into the solution there so that's exciting I know this is something that you know we wanted to point out just as far as how automated testing applies to someone going through a ServiceNow upgrade there's obviously a lot more to capi oh and if anyone has questions about capya or wants us to fill in some more information that's coming over to you obviously feel free to put that out in the questions or you know reach out to us directly there so yeah I think that's that's great any anything else that you guys want to add jeffer or Bret specific to Cappy Oh or automated testing in general what what people need to know well I think it might be important to point out what Cappy Oh does kind of versus ATF I think we've kind of talked about this before a little bit but um Cappy Oh is basically like you have a robot that's going through and doing exactly these steps and it's able to do so because Kathy OH is hosted on a server on our you know on a separate server instead of where as ATF is built into the system and is more like a a possibility this was done yeah if this was done here what would happen versus Kathy I was saying this is being done here so give me the output of what happens when we run that and I think that's an important distinction kind of to your point earlier Tim on the the differences between you know maybe a potential pitfall of something like ATF versus capo and why it makes sense to use something like Kathy oh it's a like-for-like not a if this was happening it's it's it is happening and here's the output right yeah what you're saying is is because because ATF is built on service now and I want to specify that that ATF is a very useful tool that it absolutely doesn't serve its purpose and and there are a lot of use cases where it's a great way to get some of the automated testing done but when it comes to others other use cases that a lot of our customers have come to us for the reason why they're looking for something more there are a variety of reasons one of them is that it is it is running on the system itself using a system to test itself usually means what it's like you said it's it's basically pretending what if this happened and then giving you the result whereas Kathy is built on selenium and it's hosted on an external server so it literally runs those tests on the browser is that that's what you mean mm-hmm yeah okay make sense and then along with that just to highlight another important difference is capo is equipped with so many tools to empower the users so not only do you get you know the test designer in case creator and capo recorder that help you know users from all levels you know give them the ability to be able to create these test cases and test plans you also have cafe au mini as a you know troubleshooting and debugging tool which is incredibly helpful because it opens up in automate a browser that's like controlled by automation and you can troubleshoot any you know anything that you run into that's upgrade related with Kathy o mini and you don't have to necessarily run an entire test plan or test suite so it's just very neat very neat tool to use for troubleshooting some of those issues yeah awesome so I mean kind of bringing this back to the upgrade process then I mean how often do you see how often do you see an upgrade release go poorly I mean just just in your own experience what what are the types of things that when it goes poorly due to poor testing what are the sort of problems that most customers see yeah I guess probably more often than than not I see I think upgrading maybe not the outcome the eventual outcome but I think testing is a big hurdle and in triaging Divac defects so defining test cases identifying testers is is a challenge we identified early on right knowing who to assign to a team of testers knowing how to write test cases knowing what test cases - - right but but also you know making sure that what when when test cases are being executed that it's clear what in fact is related to the upgrade what was broken as a result versus what what was never understood to be working right so if if I'm testing a process that I'm a tester I'm an assigned resource I was assigned to test change management that I didn't fully understand I'm much more likely to not appropriately test that and test all the edge cases and test all the scenarios and identify something that that may not in fact be be a break may never have worked in the first place so a lot of time is spent and spinning chasing potential bugs that that don't benefit the client and that's where writing appropriate test cases and reusing test cases were possible is really really effective so you have more time to remediate the real bugs and less time chasing down fake bugs yeah and I think that ties into something like knowledge as well right like if somebody's not aware that functionality might change because of an upgrade or you know something is operating differently it's important to be able to capture that somewhere so that they can go the reference point and tell that something is actually operating as its supposed to as well that's a benefit fault that I've seen people run into is that they don't understand maybe you know the UI changes for instance they have a client that we have to go list out and say hey here's every single UI action or button that you click it might operate differently so we have to go through and verify that these things are working correctly I think between automated testing and manual testing these can help kind of track those things but at some point the customer whoever's using it is going to have to be able to go somewhere and be able to tell how it's supposed to work so I think that's an also an important part you know document everything as much as you can I think that and that's also going to fall into that ASAP hole right it's like you know document everything is sort of that good hygiene that that it's the best practices that that you expose but maybe not everyone does and I think that's where you know having something you know some form of automated testing is definitely going to mitigate that for the customer but guys we're actually a little over time right now I do want to leave this open if anyone has any additional questions I want to throw those at us but we were getting a few questions throughout that I think we covered all of them during the conversation but if we didn't feel free to reach out to me and let me know what we can answer for you guys if you have any questions you can always go to Serna solutions comm and submit a an email to us that way follow us on LinkedIn find me on LinkedIn my name is Tim Lee Ellie IgH if you want to send me a direct message with any questions I'm happy to help in any way possible and to the panel thank you guys so much for lending us your time and your expertise deaf Alejandra and Brett always it's a pleasure and everyone stay safe and have a great day thank you thanks everybody [Music]