[Music] welcome everyone thank you for joining us today for vulnerability response success strategies planning and the customer journey I'm Tim Lee with Serna solutions and one of the reasons we are excited to do this webinar today is because we've been talking to a lot of customers that have questions about vulnerability response and a lot of customers that are looking to solve some new security problems with some new technology and we are joined by our distinguished panel we have I'll just introduce the siRNA and ServiceNow people for right now I'll jump in to our other guests in just a minute but right now we're joined by Tim Kristen with ServiceNow and Josh to cero with service solutions real quick by way of introduction guys could you just give us a little bit of background about what you do in the ServiceNow space I'll start with Tim yes good good afternoon everyone my name is Tim Krista and I'm a security and risk specialist in ServiceNow so for any topics security and inner in IRM related I'm responsible for yeah I'm the practice lead aterna solutions for security and risk so kind of work tightly with Tim on helping customers kind of achieve success when you know dealing with various products or tools specifically ServiceNow yeah and I'll say josh is definitely my go-to when I have questions about security and integrations so he's we're very excited to have both of you guys with us here today and we're excited about our guests as well like I said I'll get into that in just one minute but before that I just want to handle some jumping ahead just want to handle some quick housekeeping a question that we get very often is is this session being recorded and will it be sent to me yes we will be reporting this session we will be emailing it out to you you will most likely also be able to find it on certain solutions website on our YouTube channel and we will be sending that out on LinkedIn so if you're not already following Serna on linkedin it's a great chance for you you however if you do have questions during this session please feel free to use that Q&A feature the Q&A module on that the zoom and all don't do chat I think it's just a lot easier for us to monitor the the Q&A portion and that really gets a little bit mixed up so that's really the best way I would not wait to ask your question we will try to handle it as soon as possible if we can integrate that into our current conversation will be happy to interject and get your questions answered as soon as possible and it really helps our conversation go better for everyone if you guys participate that way and we see a lot of success so I'm very excited too just by way of agenda we're going to give you a really brief introduction on vulnerability response I imagine you already have the one-on-one understanding of what that is but we're going to get a little bit more into some of the specifics at some of the perspectives of why it's a unique offering from ServiceNow but we're also going to be talking to our honored guests today from ApS they have done a successful vulnerability response implementation and we're going to pick their brain a little bit about what that was like for them what were their lessons learned and give you that insight of just seeing what it was like to pull this off successfully and what that might take for you if that's something that you're considering and then finally we're going to talk just a little bit more about how to get the most out of your VR implementation when I say VR I'm obviously referring to the owner ability response I know sometimes it gets mixed up with virtual reality but when I say VR from this point out well we'll just assume that's what I'm talking about right now but let's go ahead and move real quick into my service now for vulnerability response and I'll send that over to Tim and to Josh Tim from your perspective I mean why do you think right now that ServiceNow has a unique offering in that solution space sure there's three really big things that circus now brings a vulnerability response and you can kind of take each of these in its own swimlane as a work first put all of your vulnerability response data in one place secondly a catch workflows to those whether they're out-of-the-box workflows from service now or once you design yourself and then thirdly create service level agreements that go around all of those workflows so you can kind of get the undue burden off of security teams and backed in the stakeholders that are really responsible for this across the organization so though I would say those are the three big big things that help us transform teams and transform current cultures yeah and Josh what about you I mean from from your perspective really more on the partner side what what have you seen that's exciting about this space yeah I mean Tim said it really well I mean I would I would add on to that tool capability perspective you know organizations as security becomes a larger market or larger focus or you know struggle to struggle to address all of the things that they have coming at them from many different directions right and figuring out what to address is really the hardest thing and ServiceNow does a wonderful job of helping you drive really robust risk and prioritization metrics around your vulnerabilities and then group those together in logical ways that you can resolve them so there's some real efficiency that teams can see and hopefully Shawn and ApS can tell us a little bit about that but the real efficiency teams can see through leveraging the tool on top of their existing kind of environment yeah I think that what what I see a lot when talking to customers is they're still using spreadsheets and that's ultimately there's there's no shame in admitting that you're still using spreadsheets because it's still so widespread but when we look at the security space today you hear a lot of people say that it's a moving target and I'm good at Excel and I wouldn't want to build an Excel spreadsheet that can hit a moving target I don't think I know how to do that so yeah I think that's that's the perfect way to to sum up what we're talking about today next let me go ahead and introduce our guests we're joined today by John Cunningham with EPS and Shawn I'll throw it to you first and if you want to go ahead and introduce yourself in your team thank you Tim I appreciate you guys inviting us to share our journey at APs I'm joined by Joseph Maloney and Tyler Anderson both are cyber security risk analysts on the cybersecurity risk management team at ApS and we were the these two are two of the main reasons why we're at where we're at today if some of the folks on the call attended the virtual knowledge 2020 we had a presentation there and these two gentlemen were the primary ones that were presenting our vulnerability response implementation fantastic outright quit yeah and it's you know the as far as the journey is concerned what I'd like to cover today briefly is that that talk touched very little on top of what these some of the metrics that are drive conversations but it did talk about some of the challenges and we we talked about the challenges because of the journey that we went through over a period of about two and a half years was not just implementing the integration between service now and tenable through vulnerability response but it took place much sooner where we were engaging with our business partners to show them you know what risks were associated with the applications and services that they rely on maybe yes we break our systems down into business and mission critical systems the business systems are largely systems like payroll procurement those systems that support the mission the mission in the case of a public utility is to make sure that we're delivering power safely those are the most critical assets that we have to make sure that we're protecting and it's once we started being instructed having these conversations we quickly realized that we didn't necessarily have good correlated data where the data that we had was not necessarily holistic and that's where the power of this tool really really shines the for all the vulnerability management solutions that are out there that such as Qualis and tenable and some others they're very good at scanning networks but it's not necessarily aggregating that data and including it in a digestible fashion that can be quickly disseminated to the asset owners for remediation and that's when those metric conversations when you go back and have those after this tool is then it becomes a data-driven conversation not an emotional one and that makes it their difference yeah and for everyone that might not already be aware I just put into the chat a link to the k20 presentation that he's talking about definitely save that link put that somewhere and watch it later they do a great job about 20 minutes of going into how they set up service now he'll be set up vulnerability response and really what they're doing to kind of lead the charge in in this type of process and so if you have questions about that that will answer a lot of them for you we may get into some of those questions here and if you have them obviously you can put them into QA but but this will also probably save you a lot of time and and answer a lot of those questions for you as well well that's awesome so and how long have you guys been on ServiceNow we've been on ServiceNow off for over ten years we were one of the first couple hundred customers when you used it as a classic work management tool the IT work was still performed and orchestrated through this tool but we've really started to expand it starting with the vulnerability response tool from our workspace from the security perspective we also use the ServiceNow GRC tool the performance analytics as well as the vendor risk management tool all of those in most of our processes at ApS within the security department have all been migrated into ServiceNow so there's very few processes that we execute with a level of rapidity that are not included or not been ported over in some automated fashion into ServiceNow ok well so let's do this right now I've kind of broken down a little bit of the customer process in terms of making decisions and what I want to do here and I'll break this down is really briefly just talk a little bit more about identifying the need sort of going into the problem that you had that required you know to look for this type of solution talk a little bit more about what that decision-making process was like and then really jump a little more into the implement implementation itself again talking more about those lessons learned was there well and kind of giving people the that invaluable insight of what that looks like so just jumping in first as far as identifying need I mean what was what would you say was really the core problem or that you know the the major deciding factor that made you look into this the inability to address vulnerabilities in a rapid fashion and ensure that we're assigning those identified vulnerabilities to our assets and the asset owners in a quick fashion sure we could address the critical items but we weren't nearly at the depth of a breath that we needed to do we were working from quite frankly from spreadsheets data extractions from cannibal taking that data analyzing it collecting it together creating a manual ticket and ServiceNow attaching that spreadsheet and then assigning it to the asset owners that we believe had the capability of remediating them so that definitely was I think the biggest identifying need our desire when we when we really have recognized we needed to do something different it was about two years ago maybe a little over two years ago when we went to knowledge 2018 and we brought with a with kind of like we started with what is our what is what do we want the future to look like and so we we built out that view who and brought that with us and started having conversations with ServiceNow folks with other companies looked at how other companies are implementing the technology and bear in mind VR at the time I believe was fairly new it was maybe just six months to a year old and there wasn't a lot of business cases out there that demonstrated where where we wanted to go so we were somewhat starting early adopters if you will and through that process we made some mistakes you know we we did some integration decisions that ultimately we had to undo had to do some rework and that wasn't for the lack of you know desire or it was a lack of experience quite frankly with my team and so they had just take a step back and take a look at really dig into the tool and that's part of the decision process to do this is do you have the right people you know the technology anybody can buy a tool anybody can implement the tool but do you have the people in place to ensure that's that success and so it took the team of about obviously probably a year to really to get to know the tool to get the in to know the integrations to learn how the matching rules from tenable through VR occurred to ensure that you're you know quite frankly you're assigning the vulnerability to the right group and what it also will do for you it will tease out inaccuracies within their steam to be hey why am I getting these vulnerabilities when Steve over here should get them well it's because the information that you haven't assumed to be is not accurate and so consequently it dries a different level of conversation not just related to vulnerabilities but to the CMDB accuracy and efficacy and I think that's a great point I think that someone came in and actually had that question right off the bat for us which was you know how much of your success was based on your CMDB maturity and on your video you had talked about how that was kind of a unexpected challenge for you was you know absent in the CMDB information what was that process like for you guys in terms of trying to overcome that what is the process like continual when you see when so if an asset owner says hey how come this is not accurate you're giving me these these vulnerabilities when it should go to maybe the application team is because your business services are not properly mapped to the assets work and vice versa so it's it's it it really becomes you know when we have these conversations when we recognize something like this we have to make sure we're prescriptively telling the CMDB the owner or the CI owner what corrections they can make to that this doesn't occur again and ultimately you know we are not the arbiters of the CMDB there's his own bite by IT we are not part of IT so we too are customers of the CMDB so it's really taking whatever those instances you see that you are making self Corrections along the way it doesn't happen overnight it hasn't been completed at ApS we still have a lot of work to do but we can but now that that is a data-driven conversation whereas before it was quite frankly more of an emotional one now we have actual examples that we can show them point to so that they can make corrections yeah so let's when we talk about the decision-making process I think that one of the things that comes up all the time is getting the right decision makers on board is that was that a process for you and how did you approach kind of getting the right people in line with making a one-year commitment to getting something implement decision process to move forward with this was quite frankly because our program could not reach the level of maturity that we desired for that end state the only way we could we could get to that point through the old saying of people process and technology we I know I had good team members on my team but we've been growing that team over a period of years several of them are from former interns and they've been with us for a certain five-six years in this case we had the processes documented we had the policies in place but we lacked the technology to take it to the rapidity that we the you know we had to put it on you know like a turbo if you will and we couldn't get there without that so the decision-making process was a relatively easy one the ROI and on the investment into moving forward with vulnerability response was was a no-brainer and then because then you can take that information once you have that integration in place and you know it works and you can see that it's working then you can start to have not just a high level metric measures adherence to uracil A's but then you can start measure how am i patching each one of our critical assets as a business service itself and then you share those reports with each of each executive for his or her business applications and and show them well this is why things are slipping or why things are doing well and then you have those same conversations with IT your challenges are still here you know you're not your assets are not necessarily correcting the CMDB or you have applications that are installed on your assets that may not be patched with the rapidity that they need to be such as Java and Adobe and all the different debt load that a lot of companies struggle with noticed us and then the operating systems you know making sure that they are using it the automation that is available to them through satellite or SCCM and that technology that will allow them to have technology that can that can support the quite frankly that the absolute need to remediate vulnerabilities as quickly as they possibly can I hope that answered the question that yeah yeah absolutely yeah and so why don't we instead of talking about it let's let's show it real quick I think that um you know talking a little bit more about what the implementation process was for you you know we want to show off a little bit of how you had put this together give your team a chance to talk a little bit more about you know what they had done you mentioned having the right people in place and what their process was sure let's go ahead and jump into that okay so it starts with the are what we call cyber security vulnerability response dashboard and I'm gonna clear this filter right here in which you're going to see when I clear that is a metric in red this metric is in red because it's below the threshold the threshold is supposed to be 97% that is the agreed-upon SLA that IT agrees to me for all the critical the standard and the high SLA related vulnerabilities that is for two days for critical 14 days for high and then 60 days for standard currently they are at 92% however if you really want to know how they're performing because this is set for January 1st of 2020 because that is when the SLA for us we kind of that was our kickoff point if you will with IT even though we were using it prior this was our agreement on the date when we would start the SLA however if you look at the 12-month rolling average is really where they're really performing they're at about 80% and then I'll kind of dive into why that is occurring and it just recently because of this number and this number we were able to bring some observations to some senior leaders from ITA showing them you're not going to meet your metric if you don't make some substantive changes very quickly and here's the reason why and it was because of this integration that we were able to have that level of mature conversation with them rather than more of an emotional one this was a data driven conversation once they saw the data it became easy so the first and foremost are not going to talk a little bit about this and they're going to kind of turn over some of the questions to Tyler Anderson and Joseph Moloney and maybe ask them to share some of their experiences so first and foremost we have three different areas that we can select from the assignment group the CIA class name and the business service breakdown if you want to see first and foremost when you come to this dashboard there's six panels this is the Tier three metric this is the total count of vulnerabilities this is the total count that has missed the target here our top ten vulnerabilities that we are seeing approaching their target this is the 12-month rolling average then if you click on my groups work this will show you specific areas of work that is literally in your queue that is assigned to your team and you can build it however you choose to build it that's the beauty of the dashboard and the performance analytics tension that comes along with it and then view from a management perspective I want to see the stuff that's approaching that's about to go either breach or as breach and then find out why that is occurring I'm not going to click on this one P this report takes a long time to load but it's basically a table format it shows all the vulnerability groups assigned to each frontline leader and and broken down by critical assets and then we have a critical asset grouping that takes all of our business mission critical systems gives an overall metric the number of vulnerabilities that are past due and this kind of a trend line here and then a breakdown of most of the defined business what we call business or mission-critical services then what you can do so this kind of drives again the team's work then you get into what quite frankly the senior leadership cares about is those things that are defined business services and quite frankly are defined as as mission critical so if I type in something like pi prod that is a that is a system that we use at APs for telemetry information on our on our power plants and once once I pull that up first off I can see right away that there are 98% attaching to the metric which is very good only five have breached the remediation timeline and you can by the way you can do the same thing here on this screen and this screen and it will give you the same number but it will tell you on this screen the total number of vulnerabilities associated with those assets and bear in mind when I say assets I mean the server's the database is the application itself that make up that service and then what their 12-month rolling average and that's really good you can see that the dispersion rate between those two is almost negligible which is really what you want so this helps kind of drive that conversation as I spoke to and less teams when I the thing I do I like about the tier 3 metrics but I don't like about it is it masks areas of trouble so if you're not if your management team is not paying attention they may not know where the laggards are pulling why things are being down you can clearly see in some of the instances we will look at critical statistics what some of those areas are we have applications that are performing here as one as a 38% here's when it's 53% here's a 62% and most of those themes we were able to identify due to a lack a manual versus an automated actual function no we had decision points by certain support teams to say we want to manually patch we don't want to use automation because we're fearful that we may cause an unintended outage well if that's the case then you don't have to adhere to your SLA so to argue have processes in place to ensure that you can meet that obligation and this is really starting to change that conversation from just the tier 3 you know the and bear in mind what is in tier 3 we all have like a 60,000 we have 60,000 manage assets at atm that sounds like a lot for some of the organizations on the call that's probably an a nap but at ApS of that 60,000 probably a quarter of those are desktop endpoint devices that is the lay up of vulnerability management in other words you can patch those if you break a few okay we can reimage them but you should never miss on a desktop vulnerability but because you can patch repetitively repeatedly and you can use automation to do so the the servers is where you know again you get teams that support teams they get nervous but this number here like I'm saying the I use a term about desktops and I say that it's it's the high tide that raises all boats so it is masking because it's relatively easy again it is masking the overall performance of IT within this know it's not revealing it and so it is not until you start to have these data-driven conversations with senior IT leaders and the asset owners about why things are slipping but more importantly why they're slipping in their specific area and so that's where it really the our bateau really comes to life were these dashboards this is a question that came in were these out of box with with VR or was this something that you had to customize and setup yourself no I'm gonna let Joseph Maloney who is the author along with Tyler Anderson there are two main guys of work in this program and Joseph has done most of the development work of this dashboard and so this takes a lot of hard work you can use the out-of-the-box stuff but bear in mind it's it's out of the box it's it's not customized for your business it's not customized for how you do business and so that's where you really need to learn the tool and be kind to yourself and not try to adopt and bite off too much too soon so Joseph do you want to answer some of those kind of give us your overview of that yeah so in this case the whole dashboard is all customized we built all the automated indicators and the calculated indicators the percentages the breakdowns all of that however the tool does offer a lot of the out-of-the-box reporting would prefer it comes with that performance analytics package the main thing that we notice is since we have been using ServiceNow for so long there was a lot of things we were not doing out of the box and that ended up needing to build something specifically cater to our company and how we work and one more thing that you might you might want to show entire if you want to speak to this we all remember and we always you know when we had these what we call accelerated high-impact vulnerabilities at this they they come on fast we have to react quickly and so I'll give an example in 2017 when the shadow brokers vulnerabilities were exposed or a tool sets permit suppose there were massive vulnerabilities that were exposed within those two and most organizations that own windows assets which is just about 99.9% of us had to act quickly in the case of that particular vulnerability it took the company at the time about I want to say 75 to 80 days to patch our windows assets Wow that well that set that sounds not horrible in the case of something as bad as that is that is not certainly not optimum and so when we implemented this tool we wanted a way of tracking those high velocity type tickets so that we could see how we're performing as a company and as an IT group and as a security team to see what you know where things are falling down and where maybe we need to poke focus of attention so Connor do you want to talk about this briefly yeah no I think you I think you pretty much covered it but yeah I would like to echo that this is a the accelerated patching is probably some of my favorite functionality that we found through this we've been through heartbleed and yeah for the water crime owner abilities right and just going through those exercises without something like vulnerability response it's pretty much talking week of creating spreadsheets and sending those out to various business stakeholders who who want to know whether their applications are impacted and what the risk is right being able to set up one dashboard and maybe an hour worth of work that kind of updates all that information real-time remediation teams who are who looking at fixing stuff can run scans to update the the the numbers in near-real-time and we can do kind of hourly imports to provide those updates to the business stakeholders they definitely made this last round of the of the accelerated patching when we had that last critical vulnerability come out of the NSA I think when I came out it really helped and addressing those vulnerabilities and as you can see with this dashboard we have various panels this one covers the count of group assignments this one is the count of affected systems and the stuff on the bottom or clothes the stuff on the top maybe we had a reimage system or something that was stuck in a drawer that so this is Desktop Services so I can clearly tell that this is an endpoint device was probably in a drawer probably it wasn't patched because we couldn't even detect it and the patching the automated patching tools couldn't even see it so more than likely that's probably what we're dealing with here but it lets you see to tyler's point immediate visibility as to how things were performing and did Toddlers earlier point literally we were sending spreadsheets to the IT asset teams that did the patching they were pulling reports from their systems we were correlating them in trying to find out where our deltas and hey do you guys have this system do you have this server on your best how can we do and you don't so it really it it's not what we call a a smooth orchestration this really raises the bar and so it provides that information to an actionable information not only to us but the asset owners themselves do you find do you find that this like increased visibility right and the really robust dashboard you've created is driving kind of more calling organization wide adoption are people more receptive to the conversation because it's more visible yes and no at first it drives so I don't know the seven stages of acceptance you know or whatever it is the six stages I can I don't know if it's next or five or whatever but everybody starts out with first denial then he gets a anger right and it eventually when you have enough of them and you don't use the blame game but you say hey this is what the data is showing us what can we do to help you and one of the things that my director always Fosters what how can security help you we're really big into not just throwing things over you know pointing out through the dashboard what the problems are but really partnering with them and fighting each of our partners whether it's on a business side or IT on our expertise and maybe methodologies that can be used to improve their response so it's it's not perfect but the conversations continue to happen we've probably done between one on ones team meetings special sessions well into the hundreds and that is that has been occurring over just the last year but it's something that has really fostered and engendered from my leadership and because of the really strong partnerships that he's developed at the VP level and that my team and myself developed at more the frontline level it is made it's making significant impact in input we probably had seven or eight meetings last week with specific teams going over data within the dashboard and not looking at the holistic stuff really diving in to what is driving their area specifically and then saying all right so when we look at the vulnerabilities if I type in a system as an example Swim's is a work management solution we use at Pala Bertie it is palaver is the largest nuclear power plant in the United States one of the largest in the world we can see that there are vulnerabilities and to those things that are past the target if I look at the past the target items when I dive into that I can do a comparison and then if I shift the day back to say let's say to April 1st I can see that in this in its data that nothing has moved out since April 1st because these are all past due right had I pulled up the other ones which was the 70 or so they're all current vulnerabilities and then also those that have breached I might be able to see some that have actually moved out of that number but you can dive into the data and kind of get just just by scanning it I can see you guys are struggling with Java and we looked and we have a lot of Linux kernel stuff that has not been applied and the question is why well I can tell you this because our Linux space systems largely don't use automated patching these manual passion and with as I mentioned my team of a big stickler on process to me if I asked my team to make it each to make a cake for me and I don't provide a recipe I'm gonna get seven different variants of the cake well I don't like the same cake and I like a process and so this is to me as a recognition that when I have this conversation with these guys from IT and teams like this I will ask them what is holding you back from using the automation is it fear measure your process tell me where you can where your concerns are so that we can share those along with yourself with the teams that can help you leverage in this case red hat satellite and then maybe overcome some of those fears start with the dev systems all right are we seeing are we seeing things working after after it's been applied you know moving on to the QA systems and the test systems finally before we get to the you know the production systems but more importantly you know do you have alignment with the business on the maintenance outages that are required to be able to apply patching that those conversations cannot be delayed and they cannot not be had because the business ultimately has to agree to those outages and is look if they understand what the risk is versus what we're trying to what we're trying to remediate and why most cases they're going to give it to you now certainly in the context of a nuclear power plant we we're not going to take certain systems down that are planned critical during an outage you just can't do it so there aren't gonna be certain times that you know obviously the business operations are going to dictate the rapidity or the the desires of IT and security to remediate things quickly now what do you think in terms of you know coming through on the other side of this what do you wish you had known on day one that you know now Tyler kind of get there from my perspective we didn't know what we didn't know this is a new control it required a whole new way of thinking of looking at the data and it I am oh I'm somebody that is always in a desire to move quicker the one thing I heard and I took away from the conversations I heard it tenable at knowledge and at the tenable conference is don't try to sprint take it take a slow map out what your problems are map out what you want your end state to look like hey bear boy we're not there yet you know we're still trying to achieve what our end state may look like but map that out be patient you're gonna make mistakes self correct when you identify them quickly listen to your team I am a leader I am nothing more than a functionary listen to the people on your team what help they need whether it's training whether it's you know having maybe a senior leadership meetings where you can drive some of the desire why why we need to change to move away from spreadsheets and there with a more automated fashion if you lay those things out in a data-driven conversation with clear targets and measurements that you yes you can demonstrate I think you'll get the support you need if you go in with them but an emotional one that well I need it because we need it well you know senior leaders don't like to give money unless they know what they're investing in and why so that would be my my point Hammacher hey yeah I think I think you touched on this most of the something we wish we would have known when we were initially implementing a lot of our assignments are based on dependency and service mapping with no service now so specifically that I think if we could have put a little bit more focus on trying to true up some dependency mapping between servers and the applications that they support we probably would have a little bit of an easier time doing some of the grouping rules and assignments initially since we we did a pretty big import initially I kind of result in a lot of stuff being assigned none of Scilly incorrectly but based on the CMDB data that really needed to be updated and there was a decent amount of work to go back through and kind of reassign some that stuff since then I mean that effort did put a lot of focus on the CMDB so I think it helped a lot and we did get a lot of movement and correcting some of the stuff that we we identified but we like we could have identified some of those problems before the import and saved us a little bit of time reassigning and doing a little bit of the rework but but that is there's one thing that's wild cost us some time internally it really did shine a light on some of the areas where we needed some improvement so so it's a little bit of a hardship for us but also almost a value-added in itself that we were unable to be able to show that hey we're unable to assign X amount of vulnerabilities and that represents this amount of risk we really need to put some focus into addressing these particular issues within the CMDB in order to be able to do that properly yeah yeah I think it's a similar challenge also CMDB related is how we manage assets in general what information is in there and how up-to-date that information is the moment that we are trying to matchable nur ability with a specific CI we encounter a lot of issues there because uh you know normally they want you to do something on like IP but if you reuse I piece you might match to something that has been retired or you may have possibility of multiple matches so again it's really understanding how you're managing your configuration items are you how are those relationship to network adapters or router interfaces and all those little details the more you understand the underlying structure of your configuration items the better you're going to be able to match that information coming from your scanner I feel like the Davos one one area that we struggle at the beginning I think we underestimated how how much difficulty we will we will end up doing so at the end we ended up using about ten sequential rules in order to match to configuration items you know starting with the DNS name and going all the way down to the last one is like an exact match at IP but I think that was again CMD V the quality of your CMDB is going to drive a lot of the success that like Tyler said this tool also it's going to drive quality of the CMDB because nobody likes vulnerabilities assigned to them that they don't have to resolve and once you prove to them that it's just a sign based on data that they maintain they will update it yeah let's and that I think that's super valuable insight if I'm being honest and and knowing what you guys know now knowing about the the large investment that this was and you know both financial and time investment would you on the other side would you say it was all worth it I can tell you my boss I definitely think so because I three four years ago I could not address to be lovable of his desire of remediation because we couldn't identify vulnerabilities in the breadth and depth that we do today bear in mind that much of what you see is a lot of it is under what we call the corporate network or the wild wild west because we are utility and what we have to protect matters most delivering power to people's homes we have a very complex network we have about 60 domains we don't have visibility and all of them we still got work to do but without that disinvestment we couldn't take care of you know and really have a good feeling about where our security what our security posture was at the moment and that's not a comfortable conversation for a director to have with a VP you want to be assured that those those investments were making because ultimately you know the customer a lot of those costs end up in customers we it has to be something that we can justify and in this case reducing risk to be to our to our company inherently reduces risk to our customers to it the insurance that we can we can meet our obligation of delivering power safely quickly so absolutely 100% the best I've been in this role for eight years this is by far the best investment I feel that an outcome that we've ever had for the technologies that we are responsible for great yeah and so I we have a few more questions that have come through the Q&A I do want to get to some of those but before that I wanted to talk a little bit more to the team about the implementation process itself what you know what we've seen in terms of you know what do people need to know what are kind of the questions they need to have just so that everyone's aware just like the video that I had to turn up earlier about the Kate's wanting presentation from APs I just now put in a link to a document or a PDF this is something we'll also provide to you by email afterward which is somewhat of a VR implementation guide it's a chance for you to ask the right questions and kind of check the right boxes to get you more strategy you can't say it'll take you a to z but it's definitely a great starts to set yourself up on the right foot for your VR implementation but talking to that a little bit Josh I know that you had kind of set aside some areas that people should be focusing on these implementations if you speak a little bit more to what that muscles is like and what we're looking at here yeah specifically you know the first thing we're gonna do is go in and have a conversation obviously with the stakeholders to understand you know what is the process today and specifically how do we identify and I think I think Sean's team talked to us a little bit but how do we identify for a given vulnerability on and give an asset who's responsible for mediating it a lot of times that's something that's kind of tribal knowledge at organizations so it's one of the things we kind of highlighted in the readiness guide and what we do is we give this to two people that are looking at vulnerability response and we say the more of these things that you can kind of get your arms around right and begin to understand what does this mean to me the kind of fewer questions we have to answer is part of the implementation so it makes things go a little smoother and you know most organizations don't have the answers to all these things that's okay but the the more items you can speak to kind of the smoother and the higher maturity you're going to get out of the first implementation you know the first round so you can you can really kind of get quite a lot if you can answer some of these questions and the big things are going to be like I said around understanding how things are assigned and remediated you know understanding the footprint of your tool so if you're using tenable or Qualis or rapid7 you know do you have comprehensive coverage across the environment as Sean said they're still working on that today so that can be a really tremendous challenge for some organizations but understanding at least where your blind spots are is a good start and then you're from the CMDB perspective a bunch of people were asking about hey we're you know what level maturity or completeness are correct and correctness do we need from assuming to be perspective to be successful the answer is you know vulnerability response is always going to move the needle for you you can have a non-existent cm maybe your scanner will bring in assets as you as you turn it on it'll give you a nice place to start right even if you have another source so you can you can always move the needle that being said your scanner may or may not you know unless you've configured it it's not going to know who owns the things and some of the information around maybe compliance in fact PCI environment etc so the more things we can do to kind of drive some maturity into the team to be the better you're going to get you know I'm more mature outcome so it's kind of you get out what you put in if that makes sense mm-hmm yeah and talking to that CMDB that was a lot of the questions that we got coming in where were people still saying you know how amateur does this have to be what do you think in terms of if someone says they have a healthy CMDB and they they implement this process how big of a change is that going to be for how they're managing that see MEP I know it's kind of a weird question but if if I guess if we're saying that people are generally happy is it still going to be a change for them or is it going to be business as usual well the tip will reveal those areas that are that's respect yeah and that once you're able to do that then you can start to have those conversations and bear in mind that the conversations that you really want to drive with IT teams initially remember we talked about the five stages of acceptance you you we would have these conversations where our teams would say well we don't want to patch well okay where we want to deviate we want an exception okay is the business aware of this and do they understand what the risk is associated with that because ultimately it is not your role or job to accept risk for the organization related to the business the business has to accept that so we have a process built in service now that relates is captured in the GRC module where we capture out your risks and associated exceptions to vulnerabilities those have to be signed off on the lowest level as a director in most cases any risk moon is above a 2.1 on a scale of four that has to be signed off by MVP so at that point you're having the conversation for risk acceptance at the right level the CMDB can reveal some of those problem areas that we talked about earlier where assets to joseph´s earlier point where all abilities are not being assigned correctly or to the right group hey let's have that conversation have you bothered to look at your you know you you have currently these ten applications you seem to have a pretty good idea on your production systems but we're seeing significant lag and vona vote is being remediated on your dev QA test are those mapped equivalently to your tech to your production systems and because we think they're not and so it drives that conversation we've had recent conversation where parts of mighty want to renegotiate the SLA and I'm saying I'm not going there I'm happy to have a conversation where you can show me the data why you can't currently meet the SLA and what schedules and processes you have in place to meet that and if you show me all right during the summer in Arizona number one you can't take a plant down during the Arizona in this summertime you know you minimize that because it's hot we produce a lot of power in the summers to keep people's ACS on those outages can become impedances to notching Oh have you overlaid those outages with what your current schedule is and your assets and can you if you can clearly show me that you can't do that then fine let's have a conversation about adjusting the essays for a certain system or apathetic applications but not holistically I'll never we will never move off of the SLA to occur they have I would like to become more aggressive for desktops I think the escalation be thirty days hmm period desktop I don't I'm not thinking to say this wrong it's not that I don't care I do care but a desktop if I break somebody's decima I live with that okay I do understand the more the the hand-wringing that can occur with respect to business and mission critical systems and applications that run our business but the CMDB as I say to everybody on these on these discussions if your CMDB has problems it will reveal nobility response will reveal them yes I think that you know we hear a lot of companies that just don't know how to handle CMDB you know they don't really even after they set it up they're not even sure whether or not it's healthy so having having something in place that will help reveal that I see that obviously as as a plus I do want to get through it we have about five minutes left in it we had a few questions left here on our docket that if if we have a moment I'd like to try and pop into these um and I think if you guys had questions and we didn't get to them go to the Q&A panel because I see that we have answered quite a few of them via text so it's a great place to go in and take a look but Sean for your team how many resources did you have dedicated to vulnerability response my team is made with people we manage risk vulnerability management as well as security awareness currently I have one two I would say roughly three that work within the ball to build a response space but they have other assignments so and they probably do about 50% of their work with invulnerability response so if you do the math it's about I would say probably kind of calculates to about two full-time out of the seven people that work directly for me yeah and you know as we said putting that putting the right team together for the implementation is important obviously putting the right team together for the operation is something completely different it sounds like you guys definitely have you know a highly skilled team taken care of this which is which is awesome um you mentioned that you guys have been on ServiceNow for about ten years and the question here is how long did it take you to design develop and deploy the new VR system you said it was about a year for you but was there were there other steps involved that are worth going into just briefly in first we had to stand what stand up the scanning technology we moved off of an old platform to tenable that took about a year then the implementation of VR took about a year and then quite frankly about six months of that year were devoted to what the integration was not that difficult it was the refinement okay what you know we chose this field why do we why don't we use this field instead we're using these matching rules refining those matching rules developing the dashboards the performance analytics the executive reports all of that took and then communicating the ocm that took place and continues to take place is Prabha the hardest to overcome because it takes you have to meet and engage people on a one-on-one or a small team basis to get them to the point of understanding so that they can be self fulfilling you know further success rather than us constantly hand-holding them did that answer your question I'm probably just just adding a note to that you know we've you know Sean your story's awesome and I think you guys did pretty cool stuff with performance analytics dashboards and whatnot I mean you know we obviously have come up with lots of customers and and and we actually didn't help push ons implementation but one of the things I want to call out is you know we do see implement pretty standard implementation times tend to be 8 to 12 weeks for vulnerability response so it is possible to achieve an outcome pretty quickly that is excluding like the scanner setup time so that's making the assumption that you've got tenable or Qualis or rapid7 scanning your environment today but the ServiceNow piece can be pretty straightforward and you can get some value pretty quickly you know from an implementation yeah all right well guys I think we're pretty much right up to time I do want to take this time just to say thank you to everyone who participated either if you joined us today as an audience member or if you were on our panel we greatly appreciate the time that you put into sharing with us today Sean and your team tyler and joseph thank you guys so much it was fantastic to hear your story if you guys have any questions for APS feel free to reach out to us at the contact information here Tim Kristen was was the ServiceNow rep that worked closely with them and the siRNA solutions works a lot in the security operations space as well so if you have questions about what those implementations are like and how we can help you out definitely feel for B and let us know again in the chat I put up a couple of links that are extremely valuable to hear interested in this topic be linked up up to the K 20 presentation from APs on their process is invaluable seeing the way they set that up they go into a lot more detail of their structure there and and what that looks like and as Tim Tim C mentioned before they really are leading the charge in terms of putting that type of process together for themselves and there's also a link in there for the VR implementation guide if you guys want to ask yourselves with some more questions internally get ready and try to shorten that process of getting implemented that's a great way to give you the toolkit to get started Josh Tim also thank you guys for joining us and lending your expertise today and everyone else we will see you next time thank you guys so much thank you guys [Music]