absolutely welcome everyone to beers with Engineers session 25 so very excited to have everybody here um so glad you decided to come spend an hour hour plus um chatting with us today and hanging out um it's always wonderful to have you all here as always we have to leave out with the Safe Harbor slide um some of the stuff that we talk about on this show is um forward-looking may not necessarily be fully GA yet may not be released in the form that we're showing so therefore they are forward-looking statements and don't make any purchasing decisions 100% based on this awesome so let's we always keep it formal here informal here um with the agenda um we this is a community so as always we recommend everybody you know Chim and bring your thoughts questions you know that's what we're all here for so we'll run through why we're here who we are and then we'll get into the tech deep Vibe for the day so as I mentioned right the goal of this is really to kind of merge together that kind of like blending between Cloud native Technologies and the service now ecosystem um to truly understand kind of where customers are are in that Journey um and have a community for people to talk about uh this you know the similar pain points that they're dealing with as they're going down that Journey um so um again this is for you guys so uh as always feel free to chime in and ask questions you guys aren't shy so I love that so who I am uh I am the manager of the Enterprise applications team here at drw um I am uh an allaround nerd but specifically I like to nerd out on itom kubernetes um cdb components and Hardware asset management and software Asset Management um I am all about trying to be intelligent and how we apply technology in order to solve business problems um I love to train Brazilian jiu-jitsu hang out with my family and play a crapload of board games and I actually am bringing a beer today um this is a a German moked beer that a friend of mine brought back from Germany um it's uh apparently um they actually smoke the like all of the the grains before they actually make the beer out of it so it's supposed to have a really like very smoky flavor in the final product so I'll let you guys know how it turns out awesome hey everybody I'm will halum I'm a a member of the recently formed itom Center of Excellence here at service now uh been working in technology for over three decades and for the last uh 10 years or so focused a lot on it operations management and automating various aspects of that doing a lot with public and private Cloud things kubernetes and really just love being able to take something that's got a lot of toil and manual effort and and automate it and make it repeatable and and reliable and free up uh free up actual people to do much more interesting things in my spare time like to hang with my family as well uh play some pickup hockey and the occasional video game today I will be enjoying a fiddlehead IPA from fiddlehead Brewing Company in shelburn Vermont uh yes it is very Smokey it's it's very interesting I like it nice it looks super dark does it have kind of a stout kind of a flavor or yeah it's got kind of a stout but it's it's like kind of thinner than your normal Stout okay yeah um yeah so it's it it feels like it was like a like a thinner beer that was aged in like a smoked Cask or something that's kind of what I would equate to it's good very cool okay so now we'll get into our Tech Deep dive so um the topic for this month is going to be uh service now service reliability management so I'll have a few slides just kind of set the stage and then we'll dive into some uh test environments I've got where been playing around with it for a few weeks okay what is service reliability management so um I guess first off uh in no particular order on my bullet list here it it's it's kind of replacing uh an existing tool as well as kind of augmenting the functionality to be beyond what the existing tool ever did so it replaces a couple components uh one was called uh site reliability operations and one was called site reliability metrics and they were really focused around Sr teams and trying to enable them to gather key metrics about the applications that they support and track those and report on those on platform so this new capability service reliability management has that capability but it also adds a load of other stuff and what it allows is organizations that have a either a Federated or a hybrid model for supporting it where some of it comes from Central it uh and some of it comes from other teams which are you know kind of adjacent to it or part of it but more uh more autonomous and have a lot of times those tool those teams will have their own tool sets which are unique to the applications they support or the platforms that those applications are hosted on um they generally you know sometimes they've got their own um you know on call rotations they've got specific needs when it comes to things like event management where uh you know a common paino might be having to rely on the service not platform team to help them with their you know event all the rule sets that are in place for event management for example and so what this does is it provides out of the boox ability to Federate crowdsource delegate these key capabilities um among those are the ability to create teams within the platform and then manage the membership of those teams completely in a completely self-contained way um we do provide out of the box the ability to require those new team Creations to go through an approval flow so they can be kind of Auto approved or or you can put them through there's an example approval flow and then you can build on that to customize it so that it matches the appropriate rigor that your organization requires for that kind of activity uh provides similar capability in the area of creating an application or a tech service within the cmdb it allows teams that manage application Services tech services to create those net new within the cmdb and again it allows you to apply a an appropriate approval workflow to that if you so desire they can also onboard if you've got existing application services in your cmdb they can onboard those to their team for support going forward SRM provides uh a very slick um interface to the existing we've we've had on call schedule tables for quite some time as part of itsm and the user experience in in creating and managing those has not really been on par with things like pag your duty for example uh this closes that Gap and it gives you a a user experience for managing on call schedules and rotations which is very similar to the um to what you would historically see in a tool like P your Dy you can also so um we also provide the ability for a support team to Define their own event integration and I'm going to kind of dig in that's one of the use cases I'm going to dig into with um with the demo and this can really um facilitate rapid onboarding of new app event sources from these applications into event management because it puts the setup of that in the hands of the tool of the teams that are actually you know managing and configuring those those tools it also makes it very easy to kind of Route the alerts that get generated because when you do that it tags it saying hey this is coming from this team specific integration which then makes it kind of a lot easier to just kind of take those alerts and align them to that team without adding a lot of uh additional layers there uh so I mentioned that this is replacing site reliability op and metrics so it is bringing with it the ability to Define service level objectives service level indicators and error budgets for these application or tech services that are managed using SRM and it also integrates with the upcoming um Store app called alert automation so this is a new capability that's currently in Innovation lab and it is a UI ux Improvement on creating the various rule sets that are used by by event management so that's uh event rules uh field mapping rules and alert management rules and so that's been available for a while um in the Innovation lab the uh SRM piggybacks on that and lets you create team managed event mapping and alert management rules which uh I was just talking to a customer the other day who was mentioning that they've got a lot of those rules in place and they're current they're starting to face challenges when it comes to managing those because the platform team is in charge of them all but they don't necessarily know which ones were requested by which team which ones are still relevant to the applications that they're um that necessitated their Creation in the first place um and so this is going to help there because it basically puts the management of those rules in the hands of the team that's actually consuming the the content that those rules are producing and I've kind of you know mentioned a couple times a lot of this is just simplifying that user interface user experience it's consolidating it into the service Ops workspace so you don't have to jump around different table list views anymore just to create some basic event management uh behaviors and it also it's just got built-in ability it handles when you add somebody to the team it gives them the appropriate role so they can do all of this stuff so again you don't have to have you know custom ACLS created and managed by the platform team just to provide some kind of collaborative capability within within iton okay so here's here's the when for service reliability management uh it is available today in what's called a controlled go to market and what that means is you can see it on the store you can search for service reliability Management on the store this uh deck will actually have a QR code uh down the line uh in a slide or two which will take you right to that page uh it's also available in the regulated service now stores so Federal customers will also be able to see it uh controlled go to market just means that you have to work with your account team to kind of entitle your account to load it and that will be um that's possible for any customer that's running itom operator Pro or higher with itsm standard or higher because it does kind of bridge the gap it's a it's a service Ops offering where um you know it does a lot of itom things with event management but it also does a lot of things that are part of itm standard like the on call rotation stuff uh they are kind of recommending uh because it's okay so it's just kind of skipping ahead and then kind of going back up uh it's due to go ga in early August this year um so because of that the recommendation right now is if somebody wants to you know work with their account team to enable it um by all means do so but if at all possible you probably want to wait for the ga before you actually roll it to prod just to make sure that that process goes as seamlessly as possible um but we do have customers who are running this in production today and um for folks who are interested in that early adoption there's actually some uh free services in play which I'll talk about in a a couple slides that are possible to take advantage of for uh implementing your initial use case for SRM and so yeah so I already said at once I'll just reiterate it's du to go ga it is on track for August I heard the first week in August of this year and it's going to be just it's included for anybody who's got itom operator Pro with at least one subscription unit an itm standard or anything above those offerings so one quick question before we dive into this I I always like to think about unless I'm jumping ahead um I always like to think about the like the value right and as I'm thinking about you know from a customer perspective where this is GNA like bring in value there's so much stuff here that so many organizations are doing that is outside of central it that is not monitored in a quick easy standardized centralized fashion right so many organizations have oh there's this Grana instance over here and there's this thing over there and none of it gets kind of like rolled up into a centralized place where you know kind of people can manage that centrally and and at least understand what's going on from a centralized um system yeah so making this easy to consume easy to bring in easy to to pull in a bunch of these other external things and still manage it um and still give them kind of their autonomy and how they want to build and manage and operate it but yet still provide visibility for the entire or there's a huge benefit to that um so that's where I see this really kind of coming in and playing well oh yeah thinking back to when I was a customer um we had a platform team of one we had one guy um and actually this guy his name is Ed poor Ed is still the platform team of one at this this customer that I used to work at and it's a major bottleneck there's all kinds of stuff they could be doing with event management um you know taking advantage of the on call Scheduling but up until now it all has to go through Ed and Ed is busy enough just keeping the platform running doing the twice yearly upgrades supporting you know um they they do all kinds of patching you know just just for OS P just for a platform patching he's got to you know go through and roll that up through the um through the different the different instances so yeah putting this in the hands of of um multiple teams could be a massive you know it it's going to make stuff possible that's just constantly constantly in the back burner for a lot of a lot of organizations you know um and uh a question did come up when we were we're in an internal webinar around SRM and somebody did ask well what are customers doing to kind of manage if a customer has certain governance around their instance they may not want to just turn this on in prod and have a bunch of teams creating event rules in prod and stuff like that so um what we did hear was that one of those early adopters um has what they've done is they've actually turned it on in nonpr and they use nonpr which you know we do this a lot with a lot of capabilities right um but you turn it on a nonpr give the teams kind of full reign in there to create things and then they just track with update sets and scoped scoped apps the because it's creating it's it's just creating records it's creating in many cases the same records that standard event management was creating it's just it's managing the the permissions such that it can be collaborative individual teams can own their own stuff um but at the end of the day it's really it's just creating the same kind of record that we've been doing in the event management and in the on call rotation areas for a while so it it does kind of you know it gives you it lets you go to where your org is from a governance perspective and you you can you know have a very prescribed way to promote things to prod but you can still have that access for these organizations to just you know create create content verify it does what they want and and then you roll it up to prod yep so the wear um the QR code in the bottom corner um feel free to snap it if you're absolutely chomping at the bit but it will we will send out the slides afterwards as well um or you can just search on the store for service robut reliability management um and as I mentioned anybody with iton Pro or higher with at least one subscription unit and coupled with itsm standard or higher will get access to in when it goes GA it's actually going to be split into three kind of three tiers right now it's uh the the controlled go to market it's all together and I guess based on customer feedback they determined they would modularize it as follows uh there will be a separate service level objective management Store app which is the SOS sis and air budget management uh there will be a separate team management which just allow that facilitates the team creation and the on call rotation stuff and then there's kind of the the the Uber app for it gives you all three capabilities um it puts the it layers you know pulls in those first two and then it adds the team Service registration and the integration with uh event management the how so I mentioned that there's some the potential for free services in in play and this actually applies to a few itom areas I'm going to highlight this for SRM uh because it's kind of one of the newest things to to hit the street from the iton side of things so service now has had in place um for about a year and a half now something called The Innovation architect program and that's kind of the key the key phrase to mention to your account team if you if this is of interest um excuse me so the Innovation architect program consists of a dedicated team of implementation Specialists who are experts in various itom disciplines and among them is um service reliability management and they have extended an offer several times actually to the various account teams within uh within service now to extend out to their customers that they will take a use case for uh SRM in this case they will implement it work with the customer to implement it in production in exchange for at the end of that engagement the customer becoming a some kind of a reference and there's flexibility in the form of how that of of that reference um there's no kind of hard and fast requirement that um you know the customer logo goes on you know marketing materials or slides or anything it could be as simple as having just kind of a peer-to-peer conversation with some other customer who's um you know kind of in in the same vertical perhaps uh making a presentation at knowledge it's it's really kind of flexible so um I know myself I came from a customer that was particularly kind of conservative when it came to to um you know putting their name and their brand out there associated with any vendors um so that was kind of my first thought when I heard about this program was that you know that's not easy for some ORS to uh uh sign up to but um what I've heard is that there's there's a lot of flexibility in terms of what form that reference takes so if the idea of you know not having to like do a full partner vetting just to do a quick use case for one of these capabilities especially if you're an itom customer you get this as part of your if you're itom Pro customer with itsm you've got this and if there's an interest in um you know doing some kind of implementation check with your account team ask them about Innovation architect program I just heard today from that leader that they've um you know they've been soliciting for interest and a lot of times you know there's just it's it's a road thing the the customer isn't at the point in their Journey yet where they're ready to take advantage of this and that's you know that's just kind of the way things go um but they are you know interested in engaging with customers to stand up uh some of these use cases health log analytics is another another big one that they provide um help with but uh again just um if it seems interesting at all do work with your account team to get more information and see if it's a good fit for your organization a couple things to watch out for especially for so so my my observations are based on the control go to market version um so if your plan is to kind of wait for it to GA in August and then look at it these will probably not be um issues that you have to be on the lookout for but if you are interested in trying it out here's what I've encountered um hopefully they allow you to avoid uh a little bit of uh a little bit of pain whoops uh so the first thing is tag based app support so in my testing surprise surprise I'm kubernetes guy so I do a lot of tag based app Services um when I initially installed this it was not seeing my tag based app service so I checked with the product team and they said yeah it's uh that requires an update set they gave me the update set and I was Off to the Races the update set will not be required when it goes GA when it goes GA there will be you know it will recognize tag based uh app service in your cmdb uh there's currently a known issue with configuring the service level objective service level indicator stuff within the service Ops workspace which is kind of that new uh UI that's wrapped around all of this stuff um that's a known issue uh if you're on Washington the workaround is just to use the Legacy UI to manage those records if you run into basically it manifests as uh a form just not loading like I'll click the tab and it'll go to open the form and it'll just sit there it'll stay a white screen uh or it'll progress through the the guided setup interface that it provides but it won't Rec recogniz the SLO record that I created so when that happens I just pivoted over to list View and it worked fine when again when we go ga that is on track to being fixed and then the other thing is you know as I mentioned this includes Integrations with the uh alert automation Store app which is uh an app that makes it a lot easier to create the various rules that you manage to control how events transition to alerts and then how the alerts kick off you know remediation workflows and the like so that alert automation piece that's still Innovation lab um what we are it is it is fairly solid it's been an innovation lab for a while and it's the fact it's an innovation lab is a non-issue for folks that are um you know that install it in their nonpr anyway since um once you use that interface to Define your rules they're just rules that you can promote to prod using standard you know scoped app or or update sit methodology so it's just something to be aware of um at this writing I don't have the target for when alert automation is supposed to go um is supposed to go ga I believe they're look believe they're targeting August as well but um I haven't seen a confirmation on that yet all right so now pivot over into some of my instances and we'll walk through some kind of some uh use cases for SRM so my test environment that I'm working in is a kubernetes cluster that's got a bunch of example applications running inside of it and so let's see so first I'll kind of show you that uh I'm and I'm using Prometheus to monitor the cluster and the workloads that are running within it so if I just look at my name spaces first I've got each of my demo apps running in its own name space so I've got a compensation app got a doc Portal app a review site app and a warranties app and those are all I'm also you know surprise surprise I'm running cn2 in there so it's populating my cmdb with all of these pods and containers as well as the tags and that's what I'm what I've used to generate those those uh application services within my cmdb and so what I'm going to do is just walk through kind of an example experience with onboarding one of these apps that's managed by an Sr team and so that that starts with um creating a a leader or a manager for this new team that I'm going to create I'm GNA focus on the app that supports uh warranties so the first thing I'll do is I'll just go to the um I'll go to the user record for the person who's going to be in charge of that team and make sure they have the appropriate role set up because they're basically going to be delegated the authority to control the management of the membership and the on call rotation for this team that's being created okay so the leader of this team is g to be Lana so I'll just click into her user record to make sure she's got the right role assigned and when you first install the tool it actually does have a guided setup experience which does which will even do this for you it'll you if you want to kind of in bulk onboard a bunch of team leaders for example to then kind of go off and uh configure their indiv idual teams it does provide a way to do that okay so I'll look at the list of roles and just make sure that I've got okay um and so one of the nice things that I've observed with SRM is so I you know kind of walked through what I was going to demonstrate ahead of time and then I was cleaning up up deleting you know the teams that i' created and the and the app service that I'd created and I noticed that um there's logic working in the background so like I already ran through this once and I created a team that Lana was the leader of and I deleted the team and so she doesn't have the team leader role anymore to automatically clean that up so team's done a really nice job of kind of not leaving a bunch of stuff dangling if you know organizational changes occur it's it's pretty you know it's pretty smart about the way it cleans up after itself so I'll assign her this SRM manager role and then the event management uh roles are um required in order to kind of provide that uh to facilitate that integration with the alert automation tool okay so now has got the r she needs I'm going to go over to my other browser and impersonate her so we can see what happens from her kind of non-platform admin perspective I love how apil tutor finally has a picture right but I've noticed yeah yeah but I've noticed it seems to change on occasion yeah I can remember when AEL didn't have a picture so as I mentioned all of these experiences that come with SRM they're built as part of the service Ops workspace so this is just the generic serops workspace homepage and then each of the SRM functionalities just added an additional icon over to this navigation bar on the the right side so first I'll go over to the teams pain and click add a team and so the wizard does give you two options create net new or you can take an existing assignment group and kind of onboard them into SRM uh this is going to be the warranties app support Sr team Lana is the manager not going to give him a team email and now it just lets you kind of search for your users that are going to be part of that team and in her role as the manager she can always go and onboard and offboard users um or it's all working on standard tables behind the scenes so any kind of automated onboarding offboarding um workflows that are in place just Dove tail right in with all of this stuff just add a couple more okay so I've got a little popup that tells me here's your requested item for your team so I'll just click over to that I'm got this environment set up to Auto approve so it should tell me that the team has already been added once it loads yeah so right in the middle here it tells me it was approved and created and it gives me a handy link where I can go to complete the setup for that team hey real quick there was a question that came in the Q&A from Kim uh do you know if the old SR Service mapping API is still for building the SRM Services through the devops release process do not know let me make a note to get that answered for you yeah that was going to be one of my key things like this has to all be doable via an API yep yeah I'm guessing intuitively I'm anticipating the answer is going to be yes because we're all about apis and and um it wouldn't make sense to not continue with that especially in the area of Sr which is also kind of very API focused but I just want to do my due diligence and double check uh let me go to Q&A and just grab that verbatim get that answered for you Kim we our team little team setup dashboard here yep yep uh where's my there we go okay so we can see that you know just kind of walks you through the process we've got our team details and our team members and again there's tabs up here where those can be Revisited and maintained throughout the life cycle of the team uh the next kind of logical step in the process is to is to assign one or more services so uh let's see let me just make sure that I've got my warranty service I think I might have to reinstantiate that take a look yeah let's just pull that in this definitely looks like an opportunity for extending csdm this code to not only build out the app service stub but be able to automatically maintain the team config yeah hey that rap up guys go go go build that yeah that would be an awesome um use case for csm's code all right so now we've got our app service I'll just go back to our wizard experience just to kind of stay consistent okay so again you have the option you create a net new service or onboard an existing one I'm choose to onboard an existing one and it kind of fills in the managed by team option for me so actually I will kind revisit that service one more time just to set it to operational all right so I've got my team I've got service and so now I want to create an on call shift to respond to alerts incidents Etc that are bound back to that application service this will take me to the schedule tab I'll create a shift and this part of it pretty you know pretty intuitive it's a fairly standard kind of if you've Ed you know pag your duty is kind of the big one but there's a bunch of little Point solutions that help you manage on call stuff and integrate with um telepan services to do page outs and that kind of thing so anyone who's familiar with that will find this as this piece of it pretty uh pretty familiar as well uh let's see we'll have them work in 8 to5 [Music] a weakly rotation weekdays keep it simple and so voila I've got my different shifts populated on the calendar form factor and then we'll just add our team members and we'll we'll just use our standard escalation policy which is basically just rotate through the members but there's literally infinite customization available here for all kinds of different uh on call rotation functionalities so now I'm published and I can see in the calendar form factor here who's on call for each it's rotating on uh because I started it on Thursday it's rotating every Thursday but that's also configurable you can tell it which day you want to rotate all right so I've got my on call shift so the next thing to do is to start getting some events in from monitoring events from my managed service so we do that by going to the service itself and going to the Integrations Tab and we'll say add integration and so this um this experience for those of you who've played around with Integrations Launchpad before is very comparable to that um since I'm monitoring with Prometheus that's what I'll look for so SRM is right right now in the ga version is going to be focused on push-based event sources um the thought process there is that these teams are generally that's kind of where they live is with web Hooks and so um that's kind of the initial Focus so it's going to be as far as Integrations that are available out of the box it'll be anything that supports a push from the monitoring or APM tool as a back stop we've had support in place for a while for a generic web hook where it doesn't really matter what the tool is if you can configure it to send an HTTP post you can send the event into into the platform so in this case um we do have an outof the-box integration for pushes from prome so the setup is pretty straightforward on the platform side because all it does is it generates a unique kind of tag that you can include into your in your web hook so that it knows that it's associated with this particular um integration instance and then it gives you a URL to uh plug in to your monitoring tool config and so what I'll do is I'll copied that and then I will let's see I will add this to my Prometheus configuration I'm G to do that off camera because it includes some authentication but then I'll once I've done it I'll walk through an example showing the config file and what I what I had to add to it while he's doing that any any thoughts or questions so far okay as I'm looking at this I'm thinking about you know how how would I use this in production today um in most of these cases I likely would be kind of set it up first run figure out the API configs and then you know show the teams how to kind of build it out and automate it themselves VI the API um you know it'd be hugely useful for that yeah it's all about making that whole process way more self-service and just taking the platform team out of the critical path for it [Music] correct all right so while still solving a very use Cas what's that well still solving a very real world use case right that's important 100% saying um oh I was just saying so in the case of Prometheus you know we provide with the integration we provide some basic instructions on how to set it up in my case I'm using the Prometheus Helm chart deployed to a kubernetes cluster so I just had to um make some adjustments so let's see so that's rolling out now and so while that's deploying I'll just kind of walk through the changes that I made in an example file so the first modification I made was I just took that URL that was provided by the platform and I plugged it into a new set of parameters called a receiver within Prometheus that's basically a receiver's a Target to send your events to so I created uh created this one here just called service now great CNO demo Prometheus 2 and this is the URL that service now just kind of hands to me and then I put my credentials down here um and then what I did to kind of Route events that are specific to the warranties application to that web Hook is I did that up here in this routes section and so the routes section is a way to say hey send events that match this particular criteria to this receiver so I've got an entry here for the my prom to receiver and I said hey sent anything from the warrany kubernetes name space into prom 2 and and that's it and then I just redeployed my kubernetes or my Helm chart to the kubernetes cluster and that is completed so now what I'm going to do is I'm just going to start generating a little bit of uh a little bit of Controlled Chaos in my cluster so that we can actually see some events roll in from my warranty service uh so to do this I'm using a tool called chaos mesh which is a handy utility I just ran into the other day so essentially I'm just using it to whoops I didn't like me changing that clean up first okay so essentially I'm just deploying a little payload which is going to start killing containers running inside my warranties which should make the warranty is at pretty unhappy in theory so to kind of see how that process is progressing I'm GNA this is my Prometheus dashboard inside of my cluster so I'm gon to load that up and so this is going to show me all the active alerts and the alert I'm looking for is this one down here pod container restarts and right now it's registering from a different name space this was kind of an existing um app service that I had been testing with before we started our webinar so so now it's added some more and so now I can see uh like this one here is looking at the warranty Nam space and if I go over to the kubernetes alert manager for Prometheus I can see okay so I can see here this is what basically tells me what alerts are being sent to which uh receivers so I can see my promp two receiver here against namespace warranties and it's got some pod container restart events that are being sent in so now I'll activate my connector and so the other nice thing this does is it gives you um it will kind of give you a preview it'll say hey here are the events coming in from this source so it gives you kind of a nice way to kind of see oh okay I've already got events coming in and it just kind of gives you that progression from okay we've set it up we've done our configuration and now we can see the events coming in corresponding to my my new integration Source um so now if I look at the express list we should start to see some um so this is an example from another app service that I've got set up so you can kind of see what happens when you get those alerts start firing with in the platform um based on the integration that's coming in it uh assigns it to the corresponding group that owns that event integration and then if you want to see the on call piece at work if I click into it so SRM includes out of the boox flows to generate incidents um based on alerts by default it's a manual there's a it's a manual activity triggered by this promote to incident button but it can be set up so that it functions automatically and those escalation policies are also manageable by those same teams so in this case I just clicked promotes incident so now I've got an incident that corresponds to that alert group and then it's going to tell me in the work notes here's the escalation which is kicked in and so for those of you unfamiliar with this the rest of this is just kind of out of the boox on call escalation that you get with itsm but so what it'll do is it'll reach out via email or uh if you've got telepan configured it'll send a text and it's kind of a handshake process so um it does n automatically like if I look at the details it's not going to automatically assign the incident to a person by default it's going to want a handshake when it does the escalation so actually if I go over to my email so I've got since this is in test mode I've got all the outbound emails going going to me so the on call person would get an email like this or a text um and then it provides them with the ability to kind of acknowledge and accept the incident that's being assigned so let's see I think if I reply to that I will get an assignment um actually actually that may not work because my email it's gonna say it came from me and so and not Stephen I'm not Stephen yep but it makes sense I'm not gonna I'm not going to wait 15 minutes for it to let me try again uh but I assume that can follow this the standard notification processes that can use basically anything right email the teams yeah teams yep yeah yeah that's the nice thing about it is it's just a lot of what a lot of what is being used under the covers is has been in place for a long time it's just it's making this a lot more seamless and you don't know no longer have to roll your own ACLS to Federate and delegate it out to these teams it's all just nicely wrapped up for you which um I think that that was that was a major blocker for doing on call stuff when I was a customer was nobody had the time nobody in the platform team had the time to set it up or to kind of teach um operations resources here's how you navigate the different tables and stuff with this UI you can just kind of hand it off to somebody and it's pretty intuitive y provide some guard rails you know a little bit of structural hey go forth and do it in this manner but yeah you don't have to handhold all the way through yep exactly this is this is great I I do want to be cognitive time we're a little bit over yeah me too um and I've covered I've covered everything uh that I had prepared so I'll just open it up for any SRM related questions and then we can go into regular round table if people have non-related topics they want to bring up oh we got a question that popped up in the Q&A this will be key in two to five years where all about AI will fix 40% SM 10% and the last 50% is for the app teams to fix thanks Kim yeah appreciate it yeah this is definitely a lot easier I think than how it's been in the past and the UI is significantly better yeah