hello everyone thank you for joining our previously recorded solution tech talk on the agent client collector also known as the servicenow monitoring agent we ask that you do not make purchasing decisions based off of the content presented in this webinar and safe harbor does apply my name is richard brownstein i'm a solution architect out of our servicenow new york office and i've been with servicenow for five years and i specialize in the it operations management management or itx space which includes all of our um cmvb and and discovery service mapping and event management technologies as well as cloud management cloud governance and i'm going to today go into details on the agent client collector which is a new feature that came out in is coming out in our paris release so i've only got a few slides this is a technical presentation and i am a technical person and by the way if you have questions please use the q a um there will be many people viewing this so whatever questions you ask i'll make sure to repeat because i'm sure lots of you will be interested in some good questions on the subject what i'm going to go through is just a few slides overview on the agent client collector which is going to explain basically what it's about is monitoring servicenow for the first time ever in its history is now monitoring customer applications in the past we only integrated with other monitoring solutions like splunk or scom and we would take in their event or metric data and then present it and use that data within our servicenow's event management now our agent client collector is doing the monitoring directly it's our complete out-of-the-box solution and i will go into the technical details of how it works the modules involved these check definitions which actually do the real horsepower work the policies that use to deploy it the ones that are out of the box where you can apply them where you cannot apply them the kinds of checks we can apply on operating systems and on applications so that's we are now in a nutshell in the monitoring business where we can monitor uh microsoft windows servers or linux servers with an agent we're deploying an agent on these production servers and we will be monitoring different aspects of the servers and receiving events when thresholds have been breached and also collecting metrics on specific data that we are pulling and then pulling this all into your servicenow instance so it's very exciting for us now i also want to state while this this agent that we're deploying is used right now its first version is just for monitoring the same agent in a future version later this year you can expect it will also be used for automated discovery which will be very exciting for people struggling to find to get credentials for their microsoft windows systems because this agent runs as a non-administrator and will be able to use for full discovery on windows okay so this very busy slide um shows a lot of different things i'm going to go through them just one at a time so what's our our ai ops solution with our agent client collector all about now third-party monitoring tools if i can get your attention here on the left hand side these are many many different monitoring tools that are on the market and they work either as an agent or agentlessly and they pull data about systems and they send events when there's a threshold breached and they send this right now this is what we've been able to do for years in servicenow they push these um their events with rest api calls or we have these connectors that run on the mid servers this little green circle in the middle is our mid server and it would run a connector which would go into these tools and pull the data and pull it in and then we normalize the data so our ai app solution is all about normalizing events and metric data coming from many many different monitoring tools all making our servicenow event management a single pane of glass what we are adding to this is down here in the lower left the agent client collector and so this is now an agent that runs on linux or windows you can see the nice little penguin and windows logo icons right here and what happens now is this communicates to a mid server so the mid server doesn't have to go out and communicate to it it communicates the mid server it gets information about what policies in other words what do we need to monitor and it takes the monitoring data it will send events to the mid server which then gets sent to aiops or it'll even send just regular metric data to the mid server which then gets pushed this the metric data if you if anybody's familiar with it is using something we have called our operational intelligence um within servicenow which is where we're constantly monitoring the metric information going on on systems now um just another thing here on the bottom i'm not going to go into detail on this but you may have noticed this now loom sometime last year servicenow acquired a company called loom which has a log file monitoring and it's also doing some um an analytics on log files to determine where there are issues even when you haven't set up thresholds okay but we're focusing here on the servicenow agent client collector going to the mid server going into the servicenow instance and um so just you know our objectives we're trying to provide ways of ingesting events metrics and logs for multiple monitoring tools so this is now servicenow is the only solution you need you can deploy for monitoring of applications and operating systems and and do all the event management you don't need to depend on a monitoring tool um it has the ability to ingest events and metrics it also has the ability to do synthetic monitoring so we can monitor an application from an end user point of view a url and so that's very important because then you can know if an application's up or down the same way a user knows if it's up or down and so it does something called a proxy in order to do that and so why servicenow aggregation of all events in one place okay so um now this gets focuses into some of the details on the agent client collector so what happens is when you load the agent client collector um you need to by the way you need to be on your paris version so you'll be you'll upgrade eventually to paris and then you need to deploy the agent client collector application which will give you all of these components then on your servicenow instance um it's available from the servicenow store just so you know and um there does need to be an upgrade for the latest version for paris and when paris is released you'll be able to get that so i do want to stress that right now if you're on your parish instance you will not be able to deploy the agent client collector not for a few weeks when it'll be available from the store but when you are then what happens is it puts these plugins into your servicenow instance and then you'll have the ability to deploy agents which is a component that you download and install and you'll have a mid server now the mid server becomes a listener so the mid server has always been a workhorse to run discovery run orchestration flows etc now the mid server and it's always had this mid this listener capability but now you'll use this listener capability when rolling out an agent well basically able to operate as a listener for multiple agents so the way it'll work is you'll designate one maybe several mid servers a cluster of grid servers actually and you'll roll out any number of these agents on windows their linux machines wherever they can be very quickly rolled out they all need to communicate to these mid servers and so the mid servers are then constantly receiving data and they're managing everything from the servicenow instance so that so you'll be able to go into servicenow and say i want to deploy a policy and you'll say i want to deploy a policy on every machine that meets a certain criteria and um it will then push out go to the mid server and tell it to deploy the policy goes out to every machine that manages that criteria if it's running an agent and deploys those policies and a policy is basically a set of checks so it's the things to do so there's like for example a windows policy that says check the memory check the cpu check the network check the disk space and and these policies are basically just scripts that um i'll go into more detail on another slide so the agent is just running these scripts it runs the script on a regular basis you decide how often you decide the thresholds it collects the results and then if it exceeds a threshold it sends an event the service now and if it um or if it's set up for a metric policy it's constantly sending its metrics and you'll see the results so now we're able to use this for monitoring brings in events and metrics directly into italian health no it will also of course if you've mapped your business applications those metrics that will be bound to the cmdb will be able to be viewed through a service map i will show that and so again extended monitoring capabilities for servicenow okay so just as i was saying before we have checks we have policies and we have this thing called a check instance so when you get the agent client collector installed you'll see all these different check definitions that we have an example is right here it's this check cpu.rb that's exactly what a file looks like and it's going to take in a parameter that says the dash w which means what will happen if it's a warning you know in other words if the cpu is 80 give me a warning or dash c which is if the cpu is 90 send me a critical event and that's it that's a check so a check is one thing that's defined a policy is a set of these checks that are defined for a particular reason for example uh windows operating system checks or linux operating system checks or apache tomcat checks or um or end user http checks there could be any number of them and a policy might contain one check or it contains several checks and a policy is what gets rolled out when a policy is rolled out you'll have what's called a check instance this is an instantiation of one of these check definitions that's running on a particular agent so once a check is running on an agent it's an instance and you could modify the parameters for that particular check instance for that agent so you could have different agents running different kinds of checks for different reasons so ryan here has a question regarding synthetic monitoring capabilities is there a way to configure synthetic monitoring for a service portal catch a bad user experience in other words slow portal page load times so the answer is yes the monitoring agent allows you for monitoring purposes it will allow you to monitor the response time now actually i'll say yes no the http synthetic monitoring agent is just a redirect it basically just tells you if it's up or down it doesn't have a threshold for if the user experience is 30 seconds or slower that's something that they still need to add what it does do however there is a metric check on the http agent i have it configured i have a demonstration of that and what that will show is that um uh it's going to basically show what the um the numbers the values that are coming in over time and it will actually monitor through our operational intelligence and tell you if anything is exceeding what's considered normal behavior it'll look for anomalies so again i'm going to show that in more detail but great question ryan thank you okay so check instance as i said before it's just an instantiation on an agent and once you have an instance deployed you can then set what parameters you want for that particular agent to use okay um there's out-of-the-box checks now another thing to note here is this thing sensu so if you've not heard of this term before the check technology that we have is called sensetsu and ascensio is an open source plugin it's an open source community of checks and monitoring and that's what we're basically deploying we're using this open source technology called sensu nagios plugins and here's a url um that you can copy down this list of different sensu checks that you could add we have several out of the box that we distribute but you can always add additional checks and there's also instructions how to add them or even create your own you can actually build your own checks that you want using the same sensu technology and then here is the workflow so now i'm getting into the technical details so you've got the agent client collector installed how does it actually work so when you get started and you've set up your agents so now you've got these agents sitting out there you'll be able to go and say i want to publish a policy if so you have to activate a policy you'll go into it i'm going to show you this in a demo what happens is you actually to publish it you edit this thing into a sandbox you determine a filter which will determine on which agents this policy will get published and then you you set any parameters that you want set on that policy and it deploys then it gets saved and published once a policy is published and active that's when it gets pushed out through the mid server well it gets sent out to the agents or it could get removed you can deactivate it and the policy gets removed so um when you're working with this you see these things that publish and republish buttons and you have to know also that you'll see check instances that when you deploy a policy you'll see instances of the deployment as well as any parameter changes that you put in place so this is the process so um policy is is new it's in the workflow you have to make it active and publish it it can be deactivated once it's active you can set a filter to decide where it gets rolled out where it doesn't get rolled out and then you can even abort it with a revert all changes so these are the simple buttons that you have you can have thousands and thousands of agents rolled out and then all you have to do is go into one policy and say i want them rolled out on for example all windows servers or all servers that are running an apache tomcat or an is database or an oracle database and we start an is web server or an oracle database and then it meets the filter and then it rolls them out automatically to every agent but it follows the simple flow okay that's it for slides let's demo this stuff okay so i'm going to quit my um and i'm going to minimize this so i'm now in my servicenow paris instance in fact before i go into details on this i have a machine here that it's a machine in which i've logged in through my rds database this machine is running my agent client collector instance if i go into services you'll see um this is the service that gets run so i'm not going through the details of installing it but once it's installed you'll see this service called agent client collector the service is running one thing i do want to point out actually i don't have to go into it logged on as dot backslash servicenow so the agent client collector is logged in not as the administrator or as the local system it's logged in as a new user that will be created usually called servicenow a regular user it does not have administrative rights and that's something i very much love and admit security people also love when it's sitting there running there are checks rolled out to it i'm running the bear tail tool which is a log file monitoring tool it just lets you see what's in the log and you can see right now on the system it's got various checks and metrics that are running they're set to run every sometimes every 30 seconds every minute and i get a log entry here every time a check runs and what the status of this is i find this very useful for debugging i've actually got this in debug mode so what i'm running by the way on this is full operating system checks and if i turn off the follow the tail i can actually see some of the messages here i'm also running some tomcat metrics it's running a check to match let's see do i oh yeah windows os events so it's running a check on windows directory um and there's so there's several this is a windows machine so i'm running several windows checks i'm also running checks against a tom cat apache application i'm running an application which runs on apache tomcat and this is my application called customer service now i'm also monitoring this application as an end user through a proxy and i'm going to show you the step by step on how i did all this so once the check so checks are deployed you actually don't need to be logged into your servers i just wanted to show you this you have the i'm sorry once the agent is deployed so the agents are deployed and now i am logged into my servicenow paris instance i made sure to set my um application agent client collector monitoring it's now an application we've got several it's actually a framework that you deploy and but it's agent client collector monitoring that you want to go in when you're actually doing anything on the agent client collector if i go into my navigator i could type agent and i could see under the agent client collector all the things i need so policies check definitions the different plugins the agents um i can see my mid servers my mid servers are configured to run operational um operational intelligence as well as to be receivers for the agents and um first thing i'm going to do is just show my agents my screen down a bit so i can access this okay so i've now got five different agents that have been deployed um you can see here that agents have the host names that are running on them different ip addresses you might have hundreds even thousands of agents that can be deployed um they're communicating back through the mid server and that's how i know everything that's going on i want to take note of this one feature we have here called silent mode um agents take up resources and because they take resources and they're running on servers possibly production servers probably production servers we have a silent mode which means the agent's going to shut itself off within the agent configuration it's configured to say if the cpu utilization gets up to a certain value we're going to stop using resources and the agent will shut itself off so it does not consume any resources or as few as possible on that server and this silent mode will go to true and so this happens automatically based on configuration it's possible to turn the feature off but you shouldn't do that and this is for the protection of your applications you definitely don't want agents to be going in and you don't want agents to be running while your applications are struggling for resources for whatever reason now if i go into one of my agents i can see here that here's the latest alerts that have been sent and one of them is actually open it's a group alert on the agent monitoring status by the way you'll see the monitoring tool is going to be the itom agent that's what you will see when we show these but this checks let's me go and see here are the actual check instances that are running on my agent so we can see we're doing a q length check checks the queue length cpu system load the system disk system memory process so these are all windows um checks windows policies it's checking to see if a particular service it's checking for the tomcat service to see if it's running on windows if it stops running it will send us an event now some of these checks are events and some of them are metrics so we actually differentiate between the two let's go in and take a look so when i want to now make use of several of my policies i go to my agent client collector just like i was in there just a second ago and i'm going to go into policies and these are the different policies which are collections of metrics or events that i could roll out so if i go down here to let's just say windows os events i can click on this windows os events policies now it's already set to active you'd say i've already got it running so the way you go about deploying a policy first it takes a look at what this is doing so you go into this and says this policy runs windows checks on windows servers it checks processor queue like system cpu disk memory so this runs several different checks lots of stuff going on here right now it's published and i can see here there is the filter that determines to which windows servers it's going to publish so a policy is associated with a configuration item type so it's going to deploy on any cis in the cmdb that are windows server but that match this filter i'm going to edit this in the sandbox so now when i edit the sandbox this allows me to modify the filter so if i decided now by default it's going to be operational status is operational but i might decide i only want it to run on other windows servers that meet certain other criteria i certainly don't want to change the ci type because that just wouldn't work this is meant for windows machines um i can also go down into the check instances and make some settings so first i'm going to look at the filter to see what where things get rolled out now i want to point out this thing called a proxy i may have meant it before this is not run as a proxy a proxy is where you are monitoring something on a different machine with an agent on one machine so the agents that this is going to get rolled out to might actually be monitoring a different machine when we do our http checks when we're monitoring an application from an end user point of view we need to run this under a proxy and so that would be important in fact i will show you that http check next time and then we'll see how that is running on a proxy so we've decided the filter now let's take a look at some of the checks um system disks cpu load so i'm going to go into these they're active these are the check instances now remember a check instance is a check that's running on a particular agent i can modify that check in fact i can modify a whole collection of checks in order to roll this out this one currently happens to be active so here's some of the things i can modify the interval how often does it run on this check what the check is going to do here is it's going to run this this is just a batch file it's a command that's going to be executed as often as this interval and if it if it exceeds 60 seconds on response it's going to time out and that will send us an event but it says right here what thresholds that will trigger a warning or a critical or any other things like this is a disk check so mount points ignore amount of points ignore labels and down here we could see the check parameters we could see that it's going to give us a warning if the cpu is exceeding 85 percent but if the cpu exceeds 95 percent it's going to get a critical we haven't bothered to fill out any of those other parameters and right here i can actually want to check i can actually say test this check on the disk against a ci and and here's a configuration so it's pulling ci straight from the cmdb so you must have a populated cmdb as you must always with event managing management if i click ok on this oops now what's going is it's actually sending this check this command out to my agent and it's going to tell it to run the check and determine if it runs now you would definitely do this if you customize some parameters or certainly if you have a custom check altogether and you would say okay we're going to chat test this check on this agent and um on the chosen age basically what i was selecting was the agent on which it's going to run this check so this could have been a proxy but in this case it's going to check the system disk on that agent oh let's come on let's hope it doesn't time out if it takes more than 60 seconds it's going to time out we don't want to see that or my patients might time out in fact i hope i selected a vowel if i didn't select a valid agent it would definitely time out and give me an error which case i'm just going to move on because i know my agents are all working anyway okay i'm going to abort that but anyway so once you've got these checks deployed and once you've got these um policies edited the way you want to then you update or return the policy update if you made any changes i'm going to actually update this to something like a 30 second interval and then after you update the policy and any changes you make you would do this button that says republish this means you're then republishing whatever the check is you had republish publish and so now we're deploying these windows checks these windows pop this with those policy for events out to all of my agents and then by the way i could see here this is the list of agents to which it's being deployed and the current status so you can go into any policy and you'll always have a list of agents running that policy you can go to any agent and you can see which policies are running which agents and now i want to show just one more and i'm going to show the http entry point so i'm going to actually focus this on http now this is the end user monitoring again we have events and we have matrix so the events this is a policy that's going to run entry points against a business service and metrics it's going to run a command against a business service but it's going to collect response times and it's going to send that to us as as a metric event for anomaly detection now this is something i want to point out when you go in and do an http entry point for events or matrix if i go into this and say i edit this in the sandbox in order to do an http endpoint we need a mapped business application so this is where we use servicenow's service mapping now it doesn't have to be mapped through automated service mapping it just means you have to have an application mapped now i have an application out here i switch over to my topology map it's called customer service customer service is a fully deployed application we've if you know if you're not familiar with servicenow service mapping in a nutshell basically we determine servers and nodes and modules out there network devices that are running some kind of component and based on the signature of that component we've determined that is part of this business service so customer service is our business service also called an application service and it is mapped to these components so when i go into my um policy for monitoring this application it says monitor applications of whatever name the filter is going to determine which business services so i don't give this a url to monitor i give this a business service in this case a business service i said a name contains customer if i click on preview there's only one business service that contains customer and i can see here it's customer service what happens is the url that it uses is the one from customer service from the entry point so if i click here it will monitor these entry points these urls so it automatically goes into our customer service business service the cmdb pulls out whatever urls are there and that's what it sends to this policy and so it's going to then run that url every in the case i've i've configured this for like 15 seconds and then it gets a response and it's going to send us that response time back to servicenow and we'll be able to see that application from the end user point of view so it's a fairly simple operation that it's using but it's leveraging the fact that we have automated service mapping so if you map if you haven't mapped applications you can't use this but you don't have to use servicenow's automated service mapping you can manually map business services but if you are then you can definitely make use of this so this i think is one big advantage of the using the servicenow agent client collector because it's not just gathering raw data on the back end it's also gathering and user data that you would expect from an application okay so now we've got this data deployed and we've got all these checks deployed how do we make use of this so um if i go into my application so this is one of my business the this is one of the nodes my business service when i go to my my topology mapping in fact um the topology map i've got is made up of two tomcat servers one running on 17231 1-214 and 172 31-10 10-52 so this is 172 311-214 so i'm already monitoring this now for metrics and as a business application i can go into this and let me go into my tomcat and just for the fun of it let me shut this down and in fact because this application is deployed on two different machines that are load balanced let me shut down the apache tomcat that's running on the second server so apache tomcats shut down if i go into the user interface and refresh this site can't be reached i just killed my application so let's see what happens and we've already seen some effect i could see some of the events coming in here now i'm going to go into the agent workspace by the way and i'm going to go into these lists of servers so i'm going to go into my lists of open alerts now one thing i want to note here the source of the events is itom agent so the itom agent when it is what represents the agent client collector so when the agent runs its check and you can see here where we've got check system memory check system memory etc and we've got it on some of these servers 1-214 10-52 i could actually go into some of these these alerts and because these configuration items happen to be mapped to a business application when i go into an alert like for example this is a tomcat alert which happens to run on certain checks we can see impacted services i can see this is impacting my customer service application it's a mapped application but let's take a look at some of the details we get back at one of these events so this is an event now that came in from the agent client collector and um we could see here the source severity group task okay so um there's a message key but here's the description windows service critical the service tomcat9 is not running um severity reduced now there was a failover node so this is something i've set up in my instance where it actually reduced the severity from critical to major because it noted that there were two nodes there but still we did get this from the agent client collector and this is some other details that we have the tomcats events this is the reporting mid server the acc and the the windows service check okay so um that's the the details that we wanted and we could see here also our customer service map and um let's note some other things we've got um you know processor queue length um errors warning the processor length is 380 but we could take a look at the totally severe ones that the severity critical and um this windows service tomcat9 is not running tomcat nine is not running um that is definitely of um interest to me and um but we've also got this one execution timed out um open alert on customer service to me itom agent and that's just on the configuration item customer service that represents our business service so not only is um do we have all these tomcat servers that are out thanks that we know that from the agent client collector we now know that the customer service application is actually down and we know this because we've got this alert that says execution timed out when i go to my topology map we can see here the customer service is definitely in the red and this is the event this is the http check follow redirect so we have an itom agent that's running from a proxy that basically connects to this url and and it determined that the application has the critical timeout so regardless of what is going on with any of the other servers we know we can't reach this application okay so that's the advantage of having a url end user monitor as long as as well as having as well as having monitoring different components servers windows metrics application metrics etc but now in addition to this we have rolled out metric collections so what else are we getting from the agent client collector so now let's talk about this insights explorer so i can actually go into my application and say let's view the metrics since it's being monitored and let's take a look at some backend metrics so we could see this is data that the agent client collector is collecting from um um it's monitoring it's quite collecting network data um ram usage percentage cpu queue length so i can actually see the details of the metrics and i can see for these are my tomcat machines again the memory we can see all this data now i can drag all this here into this explorer but it just so happens i have the insights explorer set up right here and on the insights explorer i have set up a view called customer service and so i created a view called customer service to reflect my customer service application and in my customer service application i've been decided what metrics i care to monitor now i can go into all these details and in fact one of the other things that on the application service level that i can go into well is this internal round scene so this is the total end user time so i'm not just monitoring um these backend metrics which i can drag over here in fact i can go here and add some possibly troublesome metrics that we've seen over time like this networked device and add this metric to the chart and so i can make this thing very busy and we can compare metrics over time so i'm showing the last hour i could show the last three hours what i want to make note is this blue line right here this represents the actual response time that we got when monitoring our url you can see right there the url http internal brownstein customer service that is the actual url and the response time was here was .5 seconds and we can see it over time up here however there seems to be a peak oops wait that was the network time and we could see here that oh it took up to wow seven seconds so there was a time at approximately 12 45 it took seven seconds and at that same time when comparing something like the disc the network device let's take a look at the ram usage percentage on one of our tomcat servers so it's showing in the green right now but let's see the memory and so at this time when it was taking seven seconds to get a response um the cpu usage was 59 i don't know if that's that's good or bad but now we can compare over time you know the disk usage available the the pv network device and so it's just interesting to see what was going on in fact let's take a look at cpu load so when we have that seven second response time the cpu load is this blue line and oh my goodness okay that that definitely says something um because i think i see the cpu load as um definitely looks like a very high point looks like cpu load was was pretty high and this this light blue line the whole time and so that that probably will definitely explain things because we could see our cpu is definitely being over taxed on this machine but it's only giving us really bad performance time some of the time okay so the the thing to know about this is all this data is being generated through the servicenow agent client collector um i got all this data available because i rolled out the client collector and um and i configured those policy checks um for monitoring the um windows data as well as some other information such as tomcat metrics and as well as monitoring end user response time metrics now i'm going to go back to my application and let's view the map again and so applications still in a red state execution timed out and i still have these um errors of critical process and so this is where i think i want to go into some of my critical events where the windows service is critical on certain machines and i can open these alerts and i think i'd like to do something about this it just so happens that um i have some remediation actions that i've got built on this and one of them since i know that this is a apache tomcat service that's gone down one of my remediation is restarting my apache tomcat and i can click on this run remediation what that will do [Music] is deploy a workflow job and hopefully restart my application service so it operates and goes against the flow designer while that's happening let's see with this other question ah does view metrics and insight explorer only show metrics from the acc or does it show metric some other collected metrics oh i am from scom and i will type the answer and answer it live no view metrics will show any metrics that are coming into the um into the mid server that happens to be associated with that business service so even if they're coming in from scom or solarwinds or any custom metrics that you're sending in through the web service so the vmetrix is not just the acc the advantage of the acc is that it's much easier to configure those metrics now i want to also differentiate something this insights explorer is the generic application and um i can i i set up a view called customer service i can go in um and i don't have to have a view and i don't have to have monitoring metrics for nodes that are part of a business application so i can go in and just add any configuration item i want to this so we can go down here and say just add a configuration item and if that configuration item has metrics on it we can add it to the windows explorer so it's so you can add anything in windows server windows service linux server anything you have to want here's a list of my all of my configuration items so if any of these just happen to have um so if i were to add this one to my view so i just added another ci it doesn't have to be part of a business application or not um now when you're using the topology map and you're doing this view metrics then you're only going to see configuration items that happen to be available and part of this business service so i'm not going to see any configuration items otherwise so i'll be able to go into here and say this is in the red and i can drag here to um show me this chart and it brings up the metric explorer for this device that's in the red and i could try to compare this to looks like cpuq length or any of these other metrics that are going on and compare them side by side to say hey what's the reason that we're getting in the red in fact i can actually go into this and say show me the boundaries for these anomaly scores you know what okay there we go shows me some of the boundaries and show me the anomaly scores and then what you get is the upper and lower bounds so over time we do analytics on all of these metrics and with that data we now have upper and lower bounds for what range we would expect the metric to fall into to be considered normal okay i seem to have trouble bringing up my anomaly scores for that one metric so i will move on and it's completed okay so good so it restarted my apache service so i would expect the monitor to suddenly tell us that things are okay and the execution will not time out and as long as i'm at it i'm going to tell it to restart my second service do i still have a critical warning reopened okay well okay uh things things are switching back this is moved to a warning state and eventually i expect this thing to move back to a regular state and um and i don't know if there's any more questions but pretty much got just about everything that i um would have that i can demo so if there's a lot of questions i met richard share today there was one other question that just came in going back to silent mode does silent mode trigger its own event or would it be a good idea to trigger its own defensive business rule or something okay so silent mode actually does in a way trigger event because um silent mode is triggered and it gets reflected back into the agent the agent does communicate that it's in silent mode and so from that um it's it's like it's own silent mode type of an event and so from that you could automate based on agents going into silent mode um but it's not um a normal item it doesn't necessarily send an event let's say the itom agent is now in silent mode but good question ryan yes you know you definitely don't want everything going into silent mode and not know about it and in fact if i go look at my agents let's see if any of them are in silent mode which is always a good idea to verify and nope they're all not in silent mode i'm sorry but what were you saying patrick i guess if there's no other questions um uh another question just hopped in here and from ryan is there a way to deploy ac agents in mass to target hosts versus manually on each host um yes there is um it's a silent install but you have to deploy it using whatever mechanism you would use to deploy configurations so it's um there's a silent installation and you could run the agent to install as a silent installation um on all hosts um now the expectation might also be what um uh what i've seen done at other companies where they have agents and they would have um for example um a windows image or a linux image that they deploy which always has the agent on it so like i there it would be part of a standard build would probably make a lot more sense but certainly also deploying agents in mass as well and i could see that yeah my business application is no longer in a critical state so definitely the end user monitor kicked in we still have assorted agent events here processor queue length windows processor execution timed out windows processor they're in the warning state but we still don't we don't have a critical state for our customer service application even though we do have occasional what we call in the red you know critical monitors here for some metrics you