[Music] you hello and welcome the code creative I'm Travis Tolson we are going live with a more spontaneous than usual edition of code creative more folks are opening up to offering external services inside it's been really really cool really exciting to see how things have been developing but I've been getting a lot of questions about how do I make service portal forms public how do I make Catalog items public in in ServiceNow how's it going Jenny Jenny asks how do I get Travis back into my life well here I am so yeah I really wanted to address this this public catalog item especially with some of the work that I've been doing with some of the government services and healthcare recently you know this one has really been jumping out at me lately and let's jump over to the screen so I wanted to set up a quick demo here to give an example and of course it asked me to log in as soon as I start now one of the things that gets glossed over a lot when we start talking about public forums is the security element you know a lot of times we just tend to shrug off and accept oh well you know we know we're making it public we know it's not going to be secure and that's fine it'll all be contained to this table it's not that big of an issue the problem is that whenever you open up a public endpoint you're opening up a couple different potential avenues for vulnerabilities and you really need to think about this one very carefully on how you want to approach it now of course you know some of its real simple of you know accidentally exposing read access to personal information if somebody types into a field their personal information I've seen some sites deal with this by saying don't enter personal information into this box you know and there's a lot of issues around stuff like that and visibility but specifically today I want to talk about the denial of service vulnerability that you open up because this one's pretty significant as far as I can tell just to give you some quick background on this one I have accidentally done a denial of service on at least two production ServiceNow instances once was early in my career and once was within the last year and then I started Weis inning up to throttling my web service calls to ServiceNow instances and because I've been dealing a lot more with the public stuff recently I've done some more testing on this and realize just how significant of an issue you could potentially have so you know real quick you know I've set up this little demo form here where you know you can imagine fields actually being in place I didn't really have any if we take a look at how these widgets are implemented you can see that all I'm really doing is taking a seedot server call back to the server script in the service portal widget all I'm doing is passing an action and a record and this is where the issue comes into play is the database insert itself this is this is where we actually run into the denial of service issue you know so this is all it's doing you know you can see this is being kind of a simulated collecting data from the form and then submitting it to the server side [Music] there we go I've just started working with full screen stuff what am i one of my friends and co-workers Robert Webb was badgering me about using my minimized windows and really small windows and so I've started experimenting I gave into the peer pressure of using full screen but I'm still getting used to all right so basically I come in here when I hit submit it's going to send that information server-side and what I'll get is an incident record getting created and you can see these no validation test ones popping up whenever I hit that submit now the problem is that if I look behind the scenes I can find out pretty quickly using my network call you using sends a post request to this URL now for anybody who's accustomed to working with post messages you know that the browser is not the only place a post message can be sent we can send post messages from any client a browser is just one of many potential clients another potential client is called postman so I have set up in postman a it is calling this same URL endpoint passing the same data that the browser client is now I did have to grab some cookie information and a couple of other things so it's possible that this particular call could become invalid at some point at which point I'd have to refresh those cookies but if I come back here and send this message okay so I got an empty result this time so this time the result didn't work but that's okay because I can come in here and I can renew the cookie and of course if I were you know approaching this from a more savvy standpoint I would write code that would grab the cookie for me and updated automatically I'm doing all this manually right now I hit Send and don't know it didn't work was it working and I just don't realize it okay nope not working gotta love the live demos let's say what am I missing so if I take a look here I wonder if my token updated maybe my ex user token updated what scrap yep it did now I hit Send ah there we go now I get a result back and that results my indication that the message did work I jump over here and there we go we've got a no no Val postman no validation test incident that was created now you might be wondering yourselves okay that's that's great so you can call it from another client why is this an issue well I can also write a script you know Python nodejs another ServiceNow instance pretty much whatever I choose to use and I can bombard this same message over and over and over and over and it is letting me do it again and again and again and again and imagine this running at the speed of a computer so with a script I can turn around and I can scale this out I'm in case you're wondering the the accidents that I have had were single computer client I've got a 10 year old machine sitting in my living room that I usually offload these types of background jobs to and you know it's running a bunch of other stuff it's got six gigs of RAM it's really not a powerful machine and that was the one that took down the production instances now that was on authenticated endpoints so it was trusting me as a user and I imagined you know there's a little more security behind those on production instances but for these public ones it's hard to imagine exactly what they might have in place I can see them having some security on production instances that may deflect these situations for unauthenticated users obviously I'm not going to go out and try this unless a customer calls me and asks or ServiceNow calls me and asks I'll be more than happy to but I believe they'd probably rather have their security teams run these tests but with that I can beat up a server pretty quickly and let's see I think I've got a like this one yeah so I've got a video that will give you an idea of what will happen so this is a video within a video for you guys but basically it is I'm currently trying to get to a personal dev instance this personal dev instance in fact while I am performing a and not really knowing he'll service nails infrastructure setup I was told that I could be natively impacting other aspects of their infrastructure so I've stopped running these tests so if you're seeing this please don't go out and run these tests yourself but I did manage to capture a video of it before I learned better you can see that for all intents and purposes this instance is toast I mean it's it's completely unusable you can hopefully make out the little loader symbol there and basically what you end up with is a lot of 4:29 errors too many requests I believe is what it is I'm going to fast forward fast forward you can see it's starting to minutes later still loading you know this is over the course of at least two and a half minutes maybe longer of just sitting here waiting for this to load before it actually comes up and this is not coming from a heavy load and you can see that some pages will get cached as sorry and error occurred so creating these public endpoints is really a significant issue now some of you may be thinking yourselves aha but I can run Google reCAPTCHA and I've seen plenty of blogs out there that have taught me how to use Google reCAPTCHA and I have to raise an issue with reCAPTCHA because while in reCAPTCHA is implemented properly it can provide a defense against this a lot of times it's not implemented properly so I have another example here of a public submits this one is actually technically with Google reCAPTCHA validation but and I've seen a number of blogs and I've seen a number number of turnkey type solutions out there that implement reCAPTCHA exactly like this which is to say that they turn around and they get the reCAPTCHA response and if the response is good or if the response is not good they return if it is good then they continue along the process and they hit the server now the problem is that we haven't changed anything about the server call this server call is the exact same that it was before in other words while I have prevented my trusted client from sending bad messages I haven't stopped my untrusted client at all my untrusted client my scripts you know people out there that would want to take my instance down they can still do it with ease now taking this one step further some folks have implemented server-side validation which is great and the way a lot of times this will work is you'll have a seedot server dot get past some kind of object dot then function and basically the get will say something to the tune of let's say valid a caption okay and so will pass to the server and then if it comes back will say you know if you know response dot good you know if the response comes back that the the captcha was good we've confirmed that this person is a human then run your server call and once again we end up in the exact same issue this function call this server call here is still an unprotected endpoint we're we're doing a server-side CAPTCHA validation but we are still not validating it with the database insert that database insert is still public exposed not validated now one potential solution for this is to do your CAPTCHA validation server-side at the same time that you do your database insert so you do your CAPTCHA and your database insert basically in the same if input statement and this might work I have not tested this one the issue that I can potentially see is dependent on service nails implementation and your particular implementation of sending the rest message if a database insert is still required to generate the service nail rest call then you could still end up in a performance intensive operation that could still act as a denial of service on your application server it really depends on the performance of that rest implementation so you know just to kind of recap some of the things that we've covered so far I'm gonna jump over to my iPad screen so basically what we've got here is we've got a client this could be your browser this could be a malicious client and we have discussed using some form of CAPTCHA and we've also got far or actually let me get rid of the CAPTCHA here real quick and we've got our client we've got our server so what we've discussed so far is taking a simple post message and sending it unsecured to the server and I'll add the database over here and basically what we have is we then hit our database endpoint and we have some form of long-running operation here before our response back now this unprotected post is what becomes our issue because we have an long-running operation this could be database this could be the this could potentially be a blocking rest call asynchronous rest call this could be any number of things that could potentially tie up the instance the database call is probably the worst of them because the database seems to be what runs out of resources the fastest at least as far as the limited testing that I've done so it's those database inserts the glide record insert glide record update those are the two parts of the operation that seemed to be the now we have discussed injecting a captcha check where we basically say wow I actually wanted a little bit more than that where we turn around and we do a part in my handwriting here you should alright that was a little too bad we do a CAPTCHA check and then we send the post but it doesn't really solve the issue because the server's not checking and then the last thing we discussed was having the CAPTCHA check over on the server side in order to validate with Google that you're a human before we actually make the database call and this operation here where we were we hit Google from the server side you know this call would actually go this call here would actually go out to Google this one is the best setup that I think we can do from a purely unauthenticated standpoint but then that raises the question what what about authentication makes this better you know why why is authentication better because in reality authentication to me is the right solution this this CAPTCHA public stuff is a risk for denial of service pretty much no matter how you slice it so what is it that makes authentication better well frankly a user registering with your system from a public standpoint is still going to create this risk but it isolates that risk to a single known input rather than spreading it out over over a couple different potential factors one things a registration isolates that vector to a single point so basically if we want to consider our database server and all of our performance and I spelled performance wrong spelling is not my strong suit I don't even know if that's right so we've got our database and performance intensive operations when it comes to denial of service these are the things that we want to protect so you know CAPTCHA is one thing that we can do but as we've demonstrated if we are not very careful about how we implement it that can become a liability in and of itself authentication is the other so authentication gives us a couple things first of all if a user is abusing the system we can lock that user we can prevent them from continuing to send messages hey Justin how's it going at Andrew how's it going guys so with authentication first of all we have a way of blocking users we can just inactivate that user and now they can't continue to send messages second of all it does isolate as I said to a single endpoint your authentication endpoints become the potential vector for attack so now we can focus on hardening this with firewalls and set exactly any rate limits you can start protecting that authentication with you know other protections rate limits and everything else and you know to your point Gani rate limits could also be an option for those api's as well you know so like if we were to use the cap show we could also throw rate-limiting on top of it but ultimately why I would prefer using service nails made of authentication is simple first of all is the 401 request or sorry the 401 response when authentication fails it passes a 401 response back to to your client now the difference between a 401 response and the 429 is 429 means your instances for all intents and purposes dead a 401 response though is a get operation ultimately there's there's a read operation in there which is cashable your 401 czar cacheable if you send the same credentials to the server it can look up in memory quickly without hitting these performance intensive operations it can look up quickly and say yeah nice try we've already tried you go away for a one go home now can that endpoint still be overwhelmed absolutely but it's going to take a lot more than my six gigs ram sitting over behind my couch to down an instance with the 400 watts how do I know cause I tried that one to again before I learned that ServiceNow was able to churn out 400 ones until my heart was content and my system choked before service now that not the case with the 420 knots not the case when I was hitting the performance intensive Andrew absolutely rate rate limits are just a small stopgap you know so basically you've got basically what it comes down to is you want to minimize the footprint you want to minimize the size of the the vector of attack is it one rest endpoint or is it your entire API authentication lets you narrow that down to one Justin says put your a new server on the other hand no Justin I am NOT trying it with my new server e something or other it's an older workstation with 96 gigs of ram and all the power and I'm really excited but right now it's sitting in the corner because I've been too busy with other things like this like like dealing with denial of service attacks so yeah Z for 28 Justin knows what I have sitting in here better than I do HP is e for 20 but yeah so we want to minimize the footprint of the attack by minimizing it to just the authentication endpoint we greatly limit the avenue for these easy public database inserts and what we've limited it to is the public user authentication now is service nails public service public user authentication bulletproof right now maybe not I'm gonna live in my conversation on that one but what do you get with the service now authentication well you get something that is tried and tested by thousands I don't know what service tens of thousands - I don't know what service nails customer base is up to but you get all of the benefits and knowledge if you go into the knowledge base you'll see that there are security reports on different things in service now and it is a collaborative experience of all the customers submitting this stuff and hardening those endpoints Mike's in here I'm like Mike Lombardo Gordo so uh I was told I needed to come in and makes morality comments to you yeah it's just it's it's not the same without it without you and here picking on me [Laughter] so Sarah has entered the chat but so when you're when you're using service nails native external user authentication in CSM you are benefiting from the collective experience of all service nails customers you are lonely crap Kristin end is in the house too man we're getting everybody in here today you're talking about important stuff I'm talking about how I blue how I accidentally blowed up ServiceNow instances I got scared yeah I I had I had my customer who I have a still had I had a great relationship with called me up and was like the instances down what happened and I was and I had this go I'm so sorry my bad that's muffle I did that one and I had to call call the hi desk and ask them to restart the instance Casey says sadly some very lockdown customers aren't allowed don't allow a user to be in the system at all no external customer record before they have been through a process yeah and I mean in in some cases you know that even if you have the external customer where it goes through a process then you've really got it harden because now you have trusted users if you are dealing with pure public registration or pure public submission where you don't even know who it is all you have is an IP address and the rest message you know you have no idea where that message is coming from it could be anything and it's those case as we want to mitigate but you know what I was talking about with CSM's external user registration is that at least with that you benefit from the collective experience of service male customers you have lots of security reports I can tell you now and I really need to make a list of these customers and start reaching out to them but I can tell you that there are a number of active customers right now that have spun up custom public implementations using CAPTCHA that as far as I can tell are just as susceptible despite the fact that they're using CAPTCHA hopefully there's not anybody malicious listening in on this and I know I'm taking a bit of a risk by saying that I'm not going to highlight which customers here I'm not gonna highlight you know which blog articles which turnkey solutions if you guys have content have stuff out there that's dealing with CAPTCHA you may want to go back and review your stuff your your code may be covered in this discussion in terms of stuff that is at risk so you know but with service nails external user authentication or external user registration you at least benefit from the collective experience and continued development for a lot of these other customers that have their own custom spun up implementations if I wasn't here talking about it or somebody else wasn't here talking about it if that customer doesn't subscribe to my blog or subscribe to whoever else publishes this type of content are they ever gonna know until it's too late you know and I you know especially as more government customers as more as we get more higher profile customers implementing CSM implementing CSM like behaviors custom CSM like behaviors this becomes more and more of a risk if you will Jenny says if you want to go next levels secure you can process your inbound REST API via reverse see and have the look load balance or handle DDoS protection and Justin replies you need K eights and Cooper needs and I think I said that right I don't know and ingress load balance containers see these guys are hard hard core infrastructure yeah this is way over my I'm just I'm just like accidental oh noes I blow stuff I blow stuff the prod if you have a public instance do you know how service now handles licensing around that I have to default to the you're going to have to discuss that with your representative it really depends on how your licensing is structured I've seen customers implement all kinds of different things in attempts to circumvent the license I mean that has never happened I forgot we've got service now folks on the one I'm so that I think there's some problem with the connection we're gonna have contact your I I do not condone any of the behaviors which I have demonstrated today not the accidentally destroying your Protestants not testing it on your personal dev instance not the attempts to circumvent your service no licensing I condone none of these behaviors you're gonna need like a legal I don't think I can publish this one this is like live stream only delete forever [Laughter] Andrew Barnes says service now images are coming after you now ah well you know you were talking about moving to North Carolina so you know I mean not North Carolina completely different unknown location you know there are some some North Carolina some ServiceNow clients in North Carolina I'm sure we can get them for all the ServiceNow needs I wonder if they have public on secure endpoints [Laughter] Casey says I think this discussion is great in theory for anyone that can afford CSM but there are many customers who have pure ITSM needs for external forms it really is and you know this goes back to where there are a lot of customers that are you know publishing these endpoints and I think the best that you can do is to implement CAPTCHA in the proper way with CAPTCHA server-side validation in tandem with your performance intensive process now again I'm not a hundred percent sure that the rest call does not count as performance intensive and that's my concern with that and you know one of the things with service nails external user authentication stuff is or external user registration stuff is that it is directly tied to CSM to the point that you can't even use a different portal other than the CSP portal without cloning the widget many of the customers I've dealt with on CSM are CSM ITSM hybrid and I really think that we need to make the best practices affordable to those customers especially yeah Casey says oh uh all my government customers are still I TSA unlimited licensing model and I'm with you Casey most of my customers are ITSM or pseudo CSM some kind of weird licensing model and so much of CSM has that ITSM tie-in and I really think that we need that service now needs to find a way to make the licensing accommodate those customers you know of course I've had a soapbox for years around licensing not accommodating smaller customer use cases and prioritizing you know ultra multi kajillion dollar this will be so much better but they can't afford it I would uh III would page and then Here I am bringing the average not at all page page has been here yeah Andrew said it service mail elder knowledge ten person right there mine was mine was twelve I'm not like yeah I'm not service now antique but I don't remember knowledge twelve for reasons Justin Robert says where is Express when you need it yes yes just kidding just kidding in all seriousness though I really wish there was a way that we could find something to support though those smaller use cases there are so many of them I have customers I would love to run on service now but they're just they're too small for the use case and getting a bunch of them cobbled together to make an MSP immediately viable it is a challenge you know but I mean here we are in this situation circling back to Michael Lombardo at service now it's breath rest in peace you know but you know circling the conversation fact there's absolutely those use cases that are not on CSM because of the licensing model and licensing restrictions and I think the best it the best thing that we can do is some form of low process intensive validation server-side rate limiting is is a possibility you know and I'm not I am NOT a hardcore security not a hard core infrastructure expert you know Justin has been holding my hand as i've been derping my way through docker and k eights and whatever important things he's making sure you weren't buying a jet engine to put in my house right right you know so i you know i'm not real strong on on this particular area Justin says I was thinking about this state governments should use something like single instance with domain set yes yes Justin single instance with domain step in it for each city yes state first and then what the county from there but yeah the city and county governments heck we were talking with the county yesterday about this exact thing with with with CSM or domain set this hard pack 2k 20 things but but yeah you know circling back to the to to the whole public forums and in public forum submission that is built in it's not bulletproof yet but it's better than nothing and over time it's going to get better whereas if you can troll something and you do it wrong odds are you may not find out until it's too late and again there are customers right now that as far as I can tell are vulnerable because of improperly implemented CAPTCHA inspect element you can read the client-side code you can see whether or not there is a sequential process or whether the CAPTCHA is submitted with the insert data I mean we've kind of already covered it I know I know well you know it happens but you know using service nails external user registration over time is going to be the best solution push on your sales rep if you have to talk to ServiceNow security something it is going to be better than implementing it yourself if you have to implement it yourself I completely understand you know we've been dealing with this stuff we you know we've been dealing with this tough time we've dealt with the the custom not CSM implementations krishnan has already called the SWAT team on you here do already code the stuff hey can you tell them to hold off till after dinner I'm hungry but come on man I'm helping too I'm helping to encourage and push on the new UX framework that's gotta count for something man but but yeah I mean if you have to do it on your own do what you can to protect those endpoints submit the validation token with your your registration implement something and no a token Jenny says I built a mid-layer that handles the public user authentication than the Midd calls service now interesting I would like to see that we'll have to touch base I'm very curious about that one you know but again ultimately the the the be-all end-all here what we are trying to protect when it comes to denial of service is database processes and performance intensive operations whether that's CPU bound or i/o bound which in this case could potentially be as I said a Google CAPTCHA why are you fixing my highlights because so you know the the Google CAPTCHA rest message could potentially be an i/o bound performance intensive operation I'm not sure but at the end of the day you want to protect the database and the performance and sense of operations from a public external submission implementing CAPTCHA improperly does not succeed in that you need some kind of you need some kind of token you need some kind of authentication this is what authentication is ultimately for is validating trusted users authentication is the best answer in absence of that some form of CAPTCHA some sort of human validation some sort of rate limit should likely be put in place to try and protect those endpoints so if you are implementing these public forms again I've seen so much talk about this here recently if you are implementing public submission forms you have to protect those publicly exposed endpoints places where you will find publicly exposed endpoints is going to be we've got that's one we've got rest your your script addressed we've got processors UI pages just to name a few anytime you expose any of these again he also says he's used something as simple as is I'm guessing that's pronounced je wat JSON web token I believe is what it stands for I can see where J watt mites might be able to be used Chris native request that we remove processes will pretend that didn't happen but you know any of these can be made public any of these can be made public and if they are public and performing database or performance intensive operations they need to be protected they need to be protected properly and if you want to know if your endpoint is potentially exposed all you have to do is try hitting that endpoint from postman post name creation christen and is's creation of processors has now been reserved for meit only know is that in an ACL can I override what this is how the Deaf gets broken but no processors by default run on the default rasa and can easily overwhelm the instance making it so that users using the UI can no longer access the system essentially a DDoS attack this is how we break the death she got you Chris oh they block it me [Laughter] but so using a client that is not a ServiceNow browser to hit those endpoints in the same way that you would expect to hit it from ServiceNow is a way that you can test if you can hit it from postman from a quote unquote potentially malicious client then there's a good chance that your endpoint is exposed barring some kind of rate limiting by ServiceNow barring some kind of DDoS protection layer that they've got on their production servers and the only way you'll know about that is if you contact service now and ask them but if you can hit that endpoint from something other than service now without authenticating then your then your public endpoints probably expects so Mike Lombardo says I got the main troll on the cheap DM me is that a hashtag why not glide fast there Mike but but yeah so that's really what I wanted to talk about today with as many people have been asking me about this one lately you know I've been dealing with this a lot lately I've been dealing with exposed endpoints recently I have identified them on customers recently this has been basically my last two weeks has been nothing but this and I just it's right there that Justin says hashtag why not heck SM do not have do not hack a set y'all quit trying to do this watching us I'm trying to make ServiceNow better not you have service now hunt me down I want to join the ServiceNow ninjas but then I will come after you but you know where I live exactly that's why they'll hire me um Justin when I get together and make a cloud flare integrations click the process Justin's wife hashtag why not have service now but but yeah so that's that's really what I wanted to talk about today you know you got to watch out for your public endpoints anytime you're making these forms public you are opening up insert and update operations which by definition are not cacheable you're opening those performance and sense of operations up to untrusted users which can eventually lead to bringing down your instance you know it's really a not a question of can it happen it's not really a question of if it's more a question of when when somebody determines that taking down you know a government ServiceNow instance or you know a university instance or you know somebody that made a stupid comment on Twitter taking down their company's instance you know it's really just a question of when somebody gets interested enough to take it down so you know make sure you're doing your due diligence on these public forums and doing what you can to protect them Chris nanda says beware arrest api's are monitored so you know they already known that they are watching for me at this point I swear it was accidental twice it was on purpose once for the purpose of testing until I think it was Justin told me that I was being a bad community person I mean though now that Justin has educated me on some of the infrastructure side and why I get no cookie for intentionally attacking service now Chris brings up a good point he says but if you are a consultant who builds a something for a customer that could have license implications in the future you can be held liable if that customer gets hit with a post charge and we don't want that so yeah yeah yeah absolutely and you know one of the things I've learned over the years and this is a little off topic from the specific at hand conversation of the denial of service potential on forums but you know with with customers you run into a lot of scenarios where you know the licensing is complicated the licensing is challenging but ultimately we want to try and work with service now to make the licensing appropriate so that there's a win-win for the customer and service now you know we don't want to go through and implement stuff that circumvents CSM is a very powerful tool very powerful for many different reasons yes the licensing right now can be sticky in some areas but we want to try and make that better and the best way to make that better is not by going through and implementing something under the hood that circumvents the best approach is to talk to other similarly situated customers come up with a use case and push it to the service nail sales reps Kristin Anitha is murky yes murky so you know overall we want to make the ecosystem better for everybody involved I personally love a lot about CSM and a lot about the direction it's take it's it's come a really long way since the early days I was on one of the one of the first CSM projects and it has come such a long way with some of the ITSM integrations in there it's really really improved and the external user registration aspect I've had a number of conversations with a number of customers recently where it's worth it it is worth it to go the proper way for customers servus you know I had even considered implementing an application that did customer service II type stuff for municipalities cities and government websites and since then especially with some of the recent stuff that you know we've been doing it glide fast I've really changed my direction on it because of the speed with which we were able to stand some stand some stuff up using CSM and some of the areas where I didn't fully use CSM we're looking at going back and trying to tie it more into CSM to make it stronger more robust and were insulated against some of these security issues in the future so you know you really really want to push back I you know talk to those sales reps talk to folk talk to folks in the community you know and overall let's work together to make this platform better that's what this whole little spontaneous session was about was you know communicating some of the potential risks for these denial-of-service on public forms you know so let's work together on it if you have any questions about potential for denial of service if you're working on public forms and want to know how to go about it or if you have a use case where you feel like you're being pressured by the licensing to go in a way that's not CSM hey you know reach out to me I more than happy to help be the lightning rod to to collect different to collect different sides of a community and bring that case to the right ServiceNow folks so that we can help promote these use cases and get them covered under a licensing structure that works well for everybody and provides the win/win it doesn't have to be combative we can all work together and you know find a way to make the whole ecosystem better so you know with that if you've got any questions feel free to reach out and it is time time little baby has joined the chat hi how you doing you got anything you would add no he's good no hashtag why not the pulses yeah all right good all right well thanks a lot everybody appreciate you joining and I hope to see you next time yeah it's really great conversation today thanks everyone