actually and so um please don't make any stock purchasing decisions or anything like that based on any of the conversations that we have here today so um let's we'll quickly run through kind of the highlights of the agenda and how we operate um and then uh so we'll you know why are we here right um will and I decided that this was something important for us to kick off to really help teams um and customers engage with how with each other around how they're utilizing Cloud native Technologies in specifically in relation to servicenow as it stands today um there's a lot of um then we found a dearth of information in the world around how companies are utilizing servicenow in conjunction with their Cloud native Technologies and so we thought we would step up and fill the gap so um who we are right uh my name is Mike Mike Gallagher I am the Enterprise application platforms manager for drw trading um and I have a lovingly joke that I if it's got a one or a zero in it I've probably Managed IT at some point in my career um I love all things around service now but in particular I.T operations management I'd love to focus in I play with kubernetes in my spare time because that's how I am um and uh I love to train Brazilian Jiu Jitsu and hang out with friends and play board games matter of fact I'm going to a board gaming convention next week so if anybody's there um stop by and say hi nice and I today I'm gonna change it up a little bit first I remember to actually drink my beer after my announcement um and second of all um I normally really only like Stouts and and dark beers that I can't see through um but today I'm gonna drink an apricot Blonde by Dry Dock Brewing their local Brewing Company and it's it's so delicious that I I break my normal taste profile just for this beer nice well okay um will Hallam I'm uh an itom architect I work at servicenow and uh been in technology for about three decades uh last decade or so really focusing on the I.T operations management Arena managing um managing things in the cloud on-prem trying to find ways to automate repetitive tasks and that's kind of just organically led me to the servicenow platform about six years ago and then I've been working app service now for a little over two years and in my spare time I enjoy hanging with my family playing pickup hockey and video games today I will be enjoying an uncurious New England IPA from a local Brewery in my area the upstate New York area near Albany uh druthers brewing company makes this delightful India pale ale and I will be enjoying it thoroughly as we talk about today's topic which is external credential stores and using them with your servicenow Discovery orchestration and service mapping capabilities um and I just kind of Came Upon This topic just because uh it's been in the platform for quite a while in terms of connectivity to uh out you know um a supported plug-in with cyberark we fairly recently uh announced that hashicorp had come out with their own integration with their vault product so it's come up in conversation quite a bit but because it had historically been supported with cyber Arc which is a product you can't generally just download and play with and that's kind of how we do a lot of our put together a lot of our demo content is you know if if an external tool is required it's a lot easier to put together a demo of that if they have like a free or an open source version so that was always somewhat of a barrier to entry to playing with this capability um and I just happened to come across a few handy articles that you know kind of give credit to as we get later into the talk that talked about you know how to create your own and so I thought that would be a worthwhile exercise and and you know Mike and I chatted about it and thought based on his experiences at drw with the hashicorp integration and the fact that we both kind of encountered little gotchas here and there it would be worth uh you know worth doing a session talking about things to watch out for what we had learned as we kind of journeyed through getting those uh external credential stores working in our own environments so um again I I we don't really stand up formality here so anybody I'm trying to be diligent and make sure everybody's got the power to unmute themselves in the zoom um feel free to do that or just click the hand raise whatever you're comfortable with you can certainly throw something in the Q a it looks like there's something in there now uh psychotic GitHub was removed um Isaac I don't I did not know that that we had a psychotic repo out there as well at some point the two that I ran into um were the um cyber Arc and um and hashicorp examples I I'm not I'm not sure why that disappeared but um let me just make a quick note and if I can get any insight I'll share that um in our wrap-up notes just um because yeah that's interesting it could be I I know they um I know they so I see you put a link in there is that yeah so that's the to the GitHub repo that I it used to have like all the code and data and now if you go to it it just says this Repro repository is empty so it's just okay add any insight to it and it used to have like instructions on how to install like the jar files and how to like pull them down from psychotic's website and stuff like that and what to upload into your system to get it all running kind of so it's interesting that it was removed then and then I it used to be another extra there used to be a link to this GitHub page in the documentation right I can't remember which version I had seen it in on the doc site but now looking at Tokyo and Utah it's now since uh gone yeah um I'm well you've piqued my curiosity so I'm definitely going to look into that if I get any insight I'll post it include it in our um our wrap-up notes um the one thing I I do know because we use this we we act that's the tool we use internally for like credential vault um and I've noticed as a user of that tool that it's rebranded now so maybe the fact that the company changed hands might have had something to do with it but that's just a pure guess on my part I'll um you know if I find out anything definitive I will include that in our wrap up notes that we send out after each session thank you yep that was my curiosity as well too since I'd seen that rebranding gotcha cool Okay so uh again just uh anybody with a question or comment feel free to interject as we go through our go through our conversation um so the external credential just some kind of high points about the external credential capability um per the name it lets you piggyback off of extra existing external credential stores currently um we servicenow officially supports the Cyber Arc integration and then hashicorp has their own integration that is available through the servicenow store which Mike has first-hand experience with so that's just a little you know a little subtlety there in terms of support for the Cyber Arc integration you know first stop could be serviced now because it's something that we provide whereas if you're uh a hashicorp vault user and you're using their integration then your primary point of support for that would be would be hashicorp uh it's support in terms of kind of the you know we've got credentials in use by a lot of the platform capabilities so this capability is specifically supported for Discovery orchestration and service mapping um one notable kind of exception that I'm aware of is service graph connectors as far as I'm aware service graft connector for external credentials is still outstanding pretty much across the board for any kind of service graph connector that's currently available um and so the way it functions is instead of a full-blown credential in your credential table you end up creating essentially a stub record in that credential table which then just has a kind of a handle that refers back to the credential ID however it's maintained within the particular external credential credential store product hey Will yes do we do we have to ask Navan what the roadmap is to for the service graph connectors or um you know I I think that is a great idea uh I'm gonna take an action to do that and if I do get some insight um any kind of you know ballpark or guidance there I will include that in the wrap-up notes as well okay thank you sir absolutely a couple questions in the Q a from Angie which I will answer live um does it support the SFTP or jdbc so I um by that do you mean like um the flow actions for SFTP or jdbc connection no so sir currently the credentials for getting that connection is stored within servicenow instance you cannot put it in cyber Arc and um kind of retrieve it like what we do with Discovery right so that's kind of a gap with our security team is is there any thing in the roadmap that would enable external credentials for those data source Integrations so like the the data source piece you can either put in a local SQL account where you can I'm pretty sure you can configure the connection to use whatever the account is that you're running the mid-service off of and so the mid server um service itself connects to the you know SQL Server database using its 80 right credential or rights roles um if I remember correctly the configuration still asks for the username and password to be put into cert to servicenow even though I know it's encrypted but it still is out on service now that's really the problem yeah true um I've made a note I will see if I can get any insight into where that is on the roadmap I do know um they are adding some it kind of in a related vein they're adding some enhancements to the credential table and the ability to offload the encryption keys that are used to encrypt credentials to be owned by the customer so I know that that's available today in terms of a way to mitigate this kind of mitigate that concern in a slightly different way um so if you're not familiar with that that's part of the overall Suite of functionality referred to as Vault um so not too conclusive so we well we are doing something similar so I hope this answers the question so using the integrated authentication so if you have an ad account versus an application account so on the data source you have an option to select integrated authentication and the mid server that you select the mid server service needs to be run under that ad account so so basically your ad admin or the security team could potentially set that password up and that password can be rotated um can be rotated through the Cyber Arc or hashicorp world yeah true although that only is possible if you use mid server right because I think for some SFTP Integrations that we use the SFTP drop boxes outside of our Network so service I was actually grabbing that directly without going through so that's right correct this is connecting to SFTP that that's definitely a foundational um uh feature or limitation uh take your pick of this the way they've put this capability together because of the fact that it the mechanics behind it or you're basically putting a jar file on the mid that is you know handling that credential call so there's definitely that dependence on the mid um I I'm gonna make a note and see if there's any um you know any thought to having an external credential hook that resides on the platform itself but I'm just guessing that that's probably not active something that's actively part of the the feature backlog but um I will uh make a note and if I can find anything out I'll include that in our in our wrap-up notes direct because it is I mean that's kind of the reason that service graph connectors don't do it today is because most service graph connectors start out by default directing traffic out establishing connections from the platform and not some of them but not all have the ability to say oh I want to go through the mid but I think because that was kind of added as a subsequent feature after the initial versions of those SG connectors come out um it was kind of originally coded not to include that that call to the external credential hook but you know because there's definitely been that I've heard that question asked before about tying it in with the service graph connector so there may that may bring along with it some capability to do those external Integrations direct from platform as well uh let's see Mike it looks like you had picked another question on the Q a did you wanna oh no I just we had already answered it live that service graph connector yeah um not today we're I'm gonna see what I can find out from a roadmap perspective and I'll I'll include that in our follow-up notes so that one's covered um Bearer token credential type so um let's hold that thought because Mike found some Mike made some observations with regard to the um credential type supported by the Vault um integration and when I get into kind of my custom integration that I created by following some of our documentation uh we can touch on it then as well because it does look like in addition to um and that kind of is this last point on this one slide in addition to the officially supported offerings our documentation includes instructions for creating your own external credential resolver and I actually went through that exercise for AWS Secrets manager and overall it wasn't terribly difficult I'll try and share some of the lessons learned and I actually did a little write-up in a community article about my experience doing that and there's a link to there'll be a link to that in the link section of this deck which we will send out to all our registrants afterwards so if that's of Interest those all those resources will be available for everyone's perusal just a quick rundown on why you'd want to do external credentials uh company standards if you know and I think um Angie you kind of called out that your internal security team would prefer to have more credentials managed by cyber Arc than anywhere else and that's a recurring theme across the customer base I I came from a cyber Arc shop before I started working at servicenow so it can the ability to tie into that existing tool is uh a great way to kind of reduce friction in the larger efforts to deploy and maintain the the server service now platform uh increase trust that kind of speaks to the same theme which is you know these are tools that kind of store the keys to the kingdom so to speak and so they are heavily vetted and heavily controlled and so if you can just piggyback on that instead of having to uh you know enter into the same kind of risk assessment and mitigation around something like the credentials table especially for credentials that require you know orchestration credentials even Discovery credentials sometimes are afforded a pretty high level of privilege and so anything you can do to increase the Comfort level with the maintain maintaining of those credentials can just you know kind of take a few concerns off the the service analogs platform owners plate whoops sorry I didn't mean to jump ahead um most of these tools if not all of them provide the ability to automatically rotate rotate credentials and so again just piggyback on that existing capability obviously you can rotate credentials via Automation and servicenow platform but why reinvent the wheel if you've already got something in your corporate tooling which does that for you another big win that can be obtained by going with the external credentials store is just tying into existing provisioning and logging auditing Etc processes um yes you can certainly see when a given credential is accessed in servicenow but if you've already got that logging and auditing in place with something like cyber Arc hashicorp Vault then why not just piggyback on that with an integration like this and yeah again just kind of another kind of way to say similar thing is just tying into your own internal resources as opposed to just having a separate place for these credentials or at least minimizing the rigor behind it you know we've already touched on the fact that not every single credential that servicenow uses can necessarily be stored in the external store but the more of that that you can leverage generally the higher the Comfort level and the lower overhead on your platform team in terms of the how uh how to move forward with uh implementing an external credential resolver the our standard documentation talks about how do you activate and configure the Cyber Arc integration uh the hash Corp integration as I mentioned is available on the servicenow store hashicorp has documentation on how to configure it and install it and there's a link to that at the end of this deck which everybody will get shortly after the session and then there's also those example GitHub repositories that we talked about earlier um apparently the psychotic one was out there but now it's basically just empty the remaining two that are out there are the Cyber Arc and there's an example basically and that it's important to note that both of those repos are intended for as like a starting point um the for cyber Arc you can get the official jar file and that's what you should um you know put in place if you're looking to implement this in production in a supported fashion those repos are out there for use as um examples or if the desire is to do a custom solution because your specific needs are unique relative to you know the the off-the-shelf Cyber art jar that's provided it gives you visibility into what's going on in the covers it gives you the ability to add some extra print statements add some extra support for perhaps a credential type that for whatever reason out of the box is not included in the stock jar file and I def after going through the exercise I did I definitely recommend if you do want to play around with doing a custom provider the starting the great a great starting point is to clone one of those repos and then just tweak tweak it to add the specific calls to whatever um if you're using a different product or just you know you can kind of get started with them as is to talk to cyber Arc or hashicorp vault and then apply the requisite tweaks that you'd like to uh big kind of a a big awareness point is you can't combine these so if you've got multiple credential stores you have to align them to a given mid server you can't throw multiple custom jars on a single mid server it will get confused and it'll either pick one and try and use that or it'll just say I've got multiple jars that Implement same class and it will be super unhappy uh question in the Q a from Darren cyber configuration allows the credential to be resolved several ways depending on how the credential ID is entered for example can look in multiple safes or search by IP I did notice that in the documentation depending on the format of the credential ID you can actually have like a unique credential for like a given IP address which is is really nice does the hashicorp Vault provide the same flexibility um Mike do you happen to know since you've had some Hands-On with the hashicorp integration yeah it so it actually can resolve multiple different back-end Secrets engines as well as it can read as long as the um as long as the April has access to all the right namespaces um then it can read from multiple namespaces and you'll see when you configure the the credential ID later you'll see that it calls out in there the path for the namespaced all the way down to the credential ID so yeah you can pull it from multiple different places okay the gotcha slide so Mike and I both have kind of stepped into some potholes as we've done our respective implementations and experimentations with extension external credential stores so we just wanted to kind of capture some of those and hopefully help the folks uh who are here today avoiding any of those uh any of those pitfalls um it's just it this is the first thing is called out in the documentation it's just really important because you can have it all set up and if you don't have this property set it it won't work um so definitely make sure the external credential property is set to true there's some additional properties I noticed in the hashicorp documentation that are required so definitely just kind of be vigilant with running down the documented steps for if you're using one of the two kind of um you know commercial Solutions um and even if you're doing a custom one you have to have this property uh turned on or else it won't it it actually won't even show you the option to turn on external credential store when you bring up a credential record um and so yeah that was one thing I ran into I think because I had like freshly installed the plug-in and I went immediately to my credential table and try to create a new credential I was like it's not giving me the checkbox it's not letting me select external credentials store and that was because I think something was cached to my browser so it wasn't switching to this external storage view when I was listing the credential table so then when I clicked new it was just using the default view and so it would bring up the empty record but not give me that external uh you know storage checkbox option so that's just it's more for like if you've just installed it by the after it's been installed and your cache refreshes then there's a business is actually a business rule that gets put in place such that if this property is enabled when you go to your credential list it'll include it'll select by default the external storage view which then gives you the option to check external credential store as a as a an option on the form SSH private key format so I learned that and I think Mike ran into something similar with the hashcorp on um the mid server SSH engine is a little Persnickety when it comes to the format of the private the the private key my what what I noticed is that so I was using AWS Secrets manager which just basically sticks everything in one long line and what was happening was it wasn't carrying the the new line character at the end of the begin statement and that you know and immediately preceding the the end statement and when it was feeding that into the SSH private key engine within the mid-server code it was saying it's an invalid key and and I was fat you know my credentials are failing so um it was a simple enough fix once I figured that out I just kind of re-formatted the private key string within my plugin and I'll kind of highlight those lines of code when we do the when we dive into the platform side of things but um yeah Mike I guess you guys just had to kind of play around a little bit with how you stored those keys in in hashicorp to to get it to work yep and and we ended up actually kind of formatting it exactly in this way and got it to work it took us a lot of attempts to get it right though was it like a Unix Windows thing or it's expecting like just 10 or 13 versus 10 and 13 char like the new the character no see I thought I thought when I first ran into the issue I thought that what it might have been so I actually spun up I think I had started trying it with like a Windows mid and then I spun up a Linux mid to see if that would because when I built on my build when I built my custom jar it would put up a little warning saying something like something about the Locale and by specifying the i a n something something Locale I was making my jar file platform specific and since I was building on Linux I thought that might have been it but I had the same issue when I tried it with a Linux mid and what ended up fixing it was I specifically I just I took in the whole key as one big string I deconstructed it into the begin the b64 payload and then the end and then I put it back together kind of like what you see here with the begin and then an explicit new line then the base64 stuff then a new line and then the end and then the mid server was perfectly happy both platforms so okay it was it was well I'm glad you guys had that hardship and the rest of us yep um Java build tools familiarity mid jar file soap if you do have occasion to play around with those two example GitHub repos you'll notice that the um the pom file that spells out that's basically how you list out like the dependencies that your jar has got some um it actually pulls in some jars from the mid server itself and so they kind of leave it as an exercise to the developer to either like pull down a mid server zip file and stick it somewhere in their build environment so that those jar files are accessible or just configure you know run a mid server I'm not sure why you'd do that but maybe some people have a small lab or whatever where the mid server is also their build machine so in that case you just have to point the build project at the mid JRE um directory and then I can pull in those jars and these are those it needs some mid jars for things like um logging like you want to write some log entries within the mid log then you need the standard jar that instantiates the classes that are used by the mid to right to the mid log and um one fun aspect of this for me was I I don't think I've ever built a Java class before I don't think I ever had to have had to do that so I got to learn how to use Maven and you know pull in a project and and build builds a Java class and put it in a jar so um if you're like completely not familiar with it don't let it be too intimidating there's a lot of stuff out there as far as like basics for using like I said I use Maven I'm sure there's mult there's a lot of different Java build tools one of the articles that I link to in here and in my community article um gives you kind of a little getting started how to install like the Java build environment on a Windows box I was using Linux so I kind of was on my own there but if you're a Windows person there are some basic instructions for getting that set up on a Windows um you know Windows desktop laptop that you can follow to get you to where you need to be whoops keep it in my scroll wheel when I mean to just kind of gesture with my mouse pointer um so again back to Angie's question about the supported um credential types so they're the documentation does specifically list out these are the supported credentials types um also if you go to either of the example GitHub repos it's got part of what the code has is in fact this is the last slide I think yep so I can actually just jump over to one of those repos and kind of show you what I'm talking about um so for example with the hashicorp version uh let's see when we get so when we get kind of down here it's just got a big switch where that kind of shows you all the different um credential types for which support is coded into that specific jar so this has a pretty good list um Bearer token credential type um that would I wonder if that is actually separate from kubernetes um I mean I don't see kubernetes included here either so it's probably [Music] I would say it's yeah let me just make a note and see if that's on the roadmap and or what I'll do is I'll take my example my test setup and I'll see if I can add support for a kubernetes bearer token with my custom and that way I can give you kind of both answers I can tell you whether you know there's anything on the road map or whether you should go create an idea to try and get a Groundswell to add to that feature and I'll also let you know if I'm successful in doing that from a custom perspective because as best I can tell the the only limiting factor is just mapping the whatever field on your external credentials store is populated with that you know that secret value and mapping it back to the appropriate field that the mid server is expecting to have um so what I mean by that is [Music] um let's see I think that's in my let me just bring up my little custom Java class here so here's the code for the custom resolver I created to use AWS Secrets manager and it was just you know like I said you can't just get a free version Community version of cyber Arc and I wanted to be able to show the mechanics behind a an external credential store integration and um I already was using AWS Secrets manager for some other whoops some other demo stuff so it was just kind of uh an easy option to take because I had some credentials I could reference so what I mean by kind of mapping those attributes and this code here I just took this directly from uh there's kind of a generic example provided in our documentation as meant as a starting point if you wanted to create a um uh your own custom external resolver and one of the things that example code does is it shows you kind of all of the parameters you can pass to your external resolver and then all the possible values that it can send back so these things here these top lines these are the values that you can populate so the two that I kind of focused on are the ID the credential ID and then the credential type and so the credential ID when you're creating one of those stub records when you're creating one of those stub records and you select external credentials store uh one of the fields by the way quick plug if you're not using SN to the SN tools browser plugin get it today it is freaking awesome yeah it's really good um it's in utils snutils thank you um so it asks you for a credential ID so that's what gets populated this value here is what gets passed in this ID property for the structure that it's passing in and then the other key one is type and that's basically like the class like in this case it's SSH private key I did find there was a little bit of uh that you know the the example GitHub repos were really great in calling out exactly how those are spelled out because they show up as you know shows up as a nicely kind of human readable SSH private key here but the way it gets passed into your jar is like this SSH under bar private under Bar Key so that's kind of key information that was great to have this example code to look at to refer to and then you can also pass in like the IP address that you're trying to talk to and the mid server that is making requests and then all the stuff that comes back is listed here and again all of this is it's on the doc page for which there'll be a link um that talks about you know basically gives you an example of hey if you want to create your own uh cost um external credential resolver here's how to do it and so there's the user username password uh passphrase P key is what um and this was a little bit of trial and error this P key um attribute that's what actually stores the private key and I suspect that's what you'd have to populate with a bearer token if you're returning a bearer token type um I know that's where like if it's an API key type credential this is what gets populated with the API key um not to be confused with priv key which is down here and I found super confusing at least initially but that's actually so and I I didn't really find any examples of what proof protocol and prove key are used for so I don't know whether that's maybe a function of what Vault does or it lets you manage your own external encryption key and it ties into that um similar you know auth protocol and off key they're in the code but I did not see examples of what that's actually used for so I don't know whether that's therefore future functionality or whether it's just in there because of a desire to kind of make the credential table compatible with all kinds of different credentials even ones that aren't perhaps kind of part of the main the mainstream so um you know once you get into creating a custom resolver I think there's at least an opportunity where if there's a credential type that's not supported out of the box you potentially could add support for those other credential types um provided you could populate the correct attribute to kind of come back and be used by the mid but I guess more to come on that later uh you know after I have a chance to try and you know try that use case with the kubernetes bearer token and and see if I'm able to get anywhere so one way or the other I'll I'll let everybody know by by way of the wrap up notes um Mike I'm trying to remember did you say you ran into some other kind of weird gotchas with the hashiki stuff or was it just kind of a matter of just kind of continually just double checking those docs and making sure that everything was in place the way you did I I know you said it was kind of challenging getting that thing completely uh sorted out yeah there's a couple of big key key components to it right you have to have the CLI running um on the hashicorp CLI has to be running on the machine that you're running from right so the mid server has to be able to access locally the CLI and then the CLI has to have its own configuration file already preset with the app roll um and the configurations around exactly which namespace to get to and and how it should be able to reach out and do that and um what we found is things like you know uh if that process dies you know be it's effectively just proxying the requests what that jar file does is it effectively just processes the requests and proxies them through the CLI that are running locally on the mid server and then the CLI Runs Out to the actual Vault agent or excuse me the actual Vault running in the background so um it you know you have to configure the address for the CLI to connect up to you have to ensure that your app role has all the necessary permissions to be able to read the credentials in the namespace that you're trying to gain access to um and then you know you have to go through and ensure that even things like you know local firewalls are not blocking access into those individual local ports so it it takes a little more tweaking than it would seem at first when you embark on the process but I think as long as you really follow um I found the hashicorp documentation is very detailed and very thorough um as long as you can follow that pretty closely you should be pretty good um yeah it just takes a takes a few tweaks to make sure you got it right yeah that was definitely my experience as well and looking at the instructions for cyber Arc it seems like it's similar where you need the Cyber Arc client and or agent forget what terminology they use but basically have to set up your mid as a cyber Arc you know a trusted cyber Arc client first and then the custom jar piggybacks on that to make the calls into cyber Arc so Mike and I were talking uh earlier today and comment Mike you know you commented that the fact you have to do all that kind of does make it difficult to implement this a little more challenging at least if you wanted to have containerized mids so you know because we're all about Cloud native on uh you know beers with Engineers definitely worth mentioning that it it it does add some additional layers um if you were going to try and do that there'd definitely be some iteration to try and get a container image running amid which also then has those CLI and um client capabilities to talk to your you know your cyber Arc your Hachi cord fault yeah at that point you're definitely rolling your own mid-server image um and um in ensuring that you know for our case you know the hashicorp Vault CLI is installed on that mid server image and then you'd have I I would probably recommend that you would you know configure the The Vault config file as a secret right and mount it into the mid-server so that way you're not having to constantly you know hand edit that or configure it around but um that's not something I've actually attempted to do so um I would I wouldn't uh don't take don't take my word for it that might be why we still have the kind of public GitHub repo for the hashicorp resolver because I notice at least in this comment on the code it talks about using a zero dependency Vault Java excuse me Vault Java client so if there's a desire to pull hashicorp Vault stuff and you're trying to roll your own container image to do it this may be um easier because it if I'm reading this you know accurately it claims that you know you basically just set this up and it'll be able to talk to the Vault without perhaps having to go through the same um applying the same layers that that hashicorp provided integration requires so just something to consider and just kind of an example of why you might consider a a custom implementation in in addition to the fact that there's you know there's lots of other providers out there and um you know I I found Secrets manager to be a great test case just because I already had an AWS account so you know I was able to just kind of piggyback off something I already had I didn't have to try and spin up vault if you can even do that without buying it um you know just seem to be a lot lighter weight of an effort just to kind of put together uh some custom um credential material so um just to kind of you know demonstrate it in the works it's not particularly uh you know um flashy it just kind of does what it's meant to do um you know like I I I will probably change this RSA key even though you can only see a piece of it but it'll be out on YouTube so you never know what's going to happen um but uh yeah I mean in Secrets manager I just create uh little Json payload with the user and P key values and then on my instance I've got this credential record created I you know I'm not going to kind of reiterate what's in the documentation documentation talks about once you've got your uh jar file created you just upload it and I think this is something I'm pretty sure this is something we've improved on over the years because I can remember looking at this a couple years ago and you had to manually distribute jar files to your mids which was kind of a pain but now we've got this handy mid-server jar files table so you create your custom jar file you upload it and just attack you you know you create a record and the key thing is this record needs to match the record that you create when you're following the documentation um in the vault configuration table and that's what kind of produces the pick list when you're creating your external Prudential um Alex is asking if a mid-server can support multiple um and the answer is no uh you can only have one external credential resolver per mid server so you can um I'm gonna so theoretically if you had multiple vaults volt you know applications uh what I do not know off the top of my head is if you do when you want to distribute different jars to different mids um how you would do that without it without it just sending every jar to every mid and then the mids are going to be confused because you're overloading the same class multiple times so I will take that as an action research and I will put my findings in the wrap up notes multiple vaults different jars so [Music] um the way I wrote my little custom um custom provider was I would hear to a naming convention where this would be the ID that I would put in my stub credential record and then slash and then the credential type would go after the slash and then I just kind of wrote my credential provider when it gets up here to uh you know when it first gets called This is the main kind of uh method that gets called the resolve method um I would just tell it to take the ID and the type concatenate them with a slash and that would be the name of the secret that it would try and pull from Secrets manager and the nice thing about Secrets manager is they give you they literally hand you Java code right here for how to retrieve this particular secret so I pretty much just stole this code right out of here stuck it in this thing and then tweaked it a little bit to make it a little more um parameterized uh did not fully parameterize it so it's hard coded to use my favorite region right now I definitely don't do that for real but um for playing around perfectly fine and um yeah so I just created my stub entry lock it down to my specific mids because it's kind of a catch all test environment and um I was able to successfully run a test so this will be either a show of success or show of how to debug uh good night oh I killed that VM all right let me find a good VM for a second uh AWS ec2 so I mean the nice thing is um and the reason that it's great to include the standard mid server um logging class is that you can just throw log messages and they'll just go to your mid log and so you you know when it comes to debugging this which I did a lot of as I put it together just because I was um you know because I was learning um it was great to just be able to go in the mid log and pull out my debug messages from the SSH session there's also SSH specific debug Flags you can turn on so if your SSH is a great use case because you can actually turn on explicit SSH debugging on your mid server and it gives you a load of SSH diagnostic messages all right so let's pick this uh oops didn't mean to close my browser change tabs oh but that's nope I didn't want the external I'm going to be private there we go credential validated full success awesome awesome and um so I also went ahead and created a Windows credential um um windows there we go um 59 86 maybe I'm taking a bit of a guess on the right when RM Port so it doesn't come back I may have to do a quick Google search unless anybody bonus points if anybody knows the port for win RM secured don't ask me what you get if you have beers with Engineers bonus points and we'll have to figure something out if anybody actually has the bragging rights win our Imports all right I will search I think it's uh 59.86 yeah I thought you had it right yeah I just took the last tested um IP address I may have to grab a valid IP address or do that try well there you go there we go and so then I just went ahead and I defined a uh Discovery schedule to hit those devices and I did a full discovery of my whopping like four VMS or whatever it was um but all using those external credentials and again that was uh a great success and so it just you know I think it's pretty cool because you can you know immediately resolve issues with like oh how are you going to re rotate your Discovery credentials well we'll just piggyback off the existing facility we have that does that and one sorry to jump in but one interesting one to point out is that when you're uh specifically with the hashey court Vault integration when you're in the external credential store View and you go create a new new um credential and check the external credential storefault it pops up hey here's here's you know here's your little drop down for the external credential stores and hashicorp Vault does not get added to that list it's only shows cyber Arc so if you leave it set to none um then it passes the credential request down to the mid server unless the mid server figure it out um and so if you're using Vault it's it's not broken that you can't see that in the list it's it's really odd uh behavior um and also like um regardless of the documentation it looks like basically every credential type that is on the platform does have that external credential store checkbox so you might be able to um you might be able to do this with multiple different kinds of credentials even if they're not officially supported as long as the jar file knows how to retrieve those credentials and pass it back into the mid server so um it you know kind of depends obviously the the the whole auth um like end-to-end method has to go through the mid server in order to use the jar file to connect up to the client and then over to the Vault to the uh credential store itself um so that's why the service graph connector is likely wouldn't work um however uh I I do think you could probably make this work with just about any type of of credentials that you would like but that would be totally hacking the crap out of it so yeah I mean it's definitely worth reiterating that like any customization in the platform once you you know once you undertake that effort you kind of you do own it as far as you know ongoing support making sure that it's um all working following platform upgrades and that kind of stuff but um you know based on the documentation Pro kind of providing you with the starting point for creating a custom resolver and the fact that it is pretty straightforward it's basically take a payload do some stuff with your external credential provider and then return a payload it's you know it's pretty straightforward there's not a whole lot of trickiness going on there um the um the vagaries of the SSH private key notwithstanding um oh yeah let me kind of highlight that that I had to kind of build some tweaking into my uh my resolver so I started let's see I started up here through kind of some trial and error I was able to generate a regex which corresponded with what the SSH engine within our mid-server is expecting this regex here on this line this is what the mid server is expecting a private key to look like and um so what I did well sorry one step back so what I did first as I went to regex 101 and I figured out what kind of regex our mid server was expecting it to look like then I compared that with how the private key from this SSH private key that was being provided by Secrets manager was providing and then wrote code to recognize that and that's what this regex is and so I use this to kind of deconstruct the private key as it comes in from Secrets manager and then based on my regex 101 experimentation I then reassemble it here which is basically you know like I mentioned when we were on the slide deck you take the begin string you add a new line you stick the base64 payload in the middle you terminate that with a new line and then you add the end string and that satisfied the mid server SSH engine I didn't have any of the same challenges with like straight up password type credentials like the windows one that one was much more straightforward so we're like seven minutes past time um so my flies when you're having fun drinking beer there's a couple things in the uh let's say q a Angie says thank you she needs to drop thank you Angie in retrospect you can see me thanking you on YouTube um Donnie hopefully you're still out there you're able to hang in for us to get your question if not you'll see it on YouTube as well um search now engineer has us at a record to the Vault configurations table to be able to select different vaults if I recall caveat was that there needed to be a mid-server jar with the same name to handle the requests to that Vault you're correct the yeah so you upload the jar and that's got um that will have a specific well the G I don't know that the jar the jar record has to have the same name as the Vault configuration record so if we look here at the jar files um so I should have a single jar file record named cred res AWS SM I click into it the actual jar file that's attached is a completely different name so the jar file itself could be named whatever at least in my exercise I the name did not match um perhaps bad form but what have you uh the name of the jar file record res AWS SM and then if I go to vaults configuration table I think then I've got a vault configuration record that's named cred res AWS SM and then that does appear as an option in the pull down for the credentials so something like that may be the method that you can use to kind of tell it hey load in specific jars I'm just a little concerned that usually I I don't know that jars aren't loaded kind of pre-loaded in which case you'd have the class that's defined in those jars is the same class so if you look up here uh it's defining a credential resolver class and then if I look at one of the example repos that's also defining a credential resolver class so if you've got multiple jars on the same mid um and it could just be my you know like I said this is the first like Java build I put together so it could be Java's just smart enough to load specific jars in and out in which case good for Java but I'm gonna unless somebody knows that authoritatively I will take it upon myself to kind of look into that a little bit and and run that down include that in the notes uh let's see done done so we've got nothing in QA um I'll just give a quick call for any other kind of topical stuff before I turn off the recording you focus the idea the opportunity to air you know General Grievances and ask general questions if they would so like going once twice perfect right