hello everyone good morning good evening welcome to virtual agent academy today's topic is defining custom virtual agent nlu topics so i'm joined by my colleagues today david van housten and nabil asby my name is marcel goodsol we all represent product management in different areas of servicenow around virtual agent and conversational interfaces in nlu so to get started our goals for today are to share some learnings and some best practices on how to start building custom virtual agent topics typically you might look at the out of the box topics but oftentimes we found our customers really want to build some custom topics and the question is always is what's the best way to do that how should i think about that how should i approach that so as part of this exercise we will break this down into two parts so the first part being the nlu component how to build a custom intent how to apply entities to that intent and then lastly how to build a virtual agent topic leveraging that brand new intent and how to utilize entities for better topic discovery and recognition so just some some brief recommendations right we always recommend think big but start small it's very easy to start small and scale up from there because as part of virtual agent nlu implementations you want to start at a manageable level you want to be able to iterate on your implementation you want to be able to start small really define a narrow scope and see success with that and understand that users might be slow to adopt and really build relevant topics that will help deflect cases and incidents and really benefit the business so the foundations are oftentimes very important and in a lot of situations customers might think oh well i should you know start with a lot of different topics and we have all these ideas but actually the best way to start is to have a narrow scope a very defined solid foundation upon which you can build so it really becomes an iterative process and especially if you plan to incorporate nlu early on you want to make sure that you're refining your topics and really optimizing what you're building so that as you scale up you're scaling up on your best practices and you're scaling up on topics that users love and will keep going back to over and over again and to do this it really requires you to plan and be very prescriptive about how you begin that process so a current challenge that you might see and this may look familiar and again this can be applied across hr can be applied across i.t legal finance but oftentimes there are opportunities to build custom topics for example hr does not ship a paid time off or a time off request or a vacation request out of box and this is oftentimes a common topic that really makes sense to put into virtual agent because it's easy to build and it can deflect a lot of cases for these types of requests so this follows a typical process in the typical channels that an employee might engage with so there's opportunity here to look at this and say well what are my options to build a process like this in virtual agent and how would i even approach something like that so one way to start is looking at what's your process right how does a user or an employee request something like time off how would they uh go about that would they go to the portal would they go to their mobile device do they talk to their supervisor so you might get a process map like this so you could be an hr administrator embarking on building out this particular topic or you could be a delegated developer that has handed this process map with the expectation that you will create a workable topic for hr to use so it might look very complex at first there might be a lot of touch points a lot of different decision points and some things you might want to consider and this can oftentimes be very confusing on how to build something from start to finish so today's session is really going to break this down how you would attack and approach something like this and start with um how to incorporate nlu into building this topic so briefly some assumptions right we're not trying to boil the ocean we're not trying to build out every single branch or every possibility of this process but we want to make sure that we can build something that will at least deflect some requests for paid time off or time off requests um some basic assumptions is that this assumes that you have entitlements to human resources um and you want to use virtual agent right but you're you're using it not to solve every potential every you know combination of events that could take place but you're you're picking a particular part of that process and you're saying this makes sense to move in a virtual agent let's solve for this and we want to build it as an action action-oriented topic because remember virtual agent is different from search it's different from a service catalog or an hr catalog or a portal experience in that you can use it to build guided processes so having action-oriented topics is oftentimes very beneficial we will take the approach of a u.s centric topic design and the assumption that you are importing paid time off balance or leave balance into service now and so that is going to be one of the main assumptions and we are going to be using the default setup topics for the nlu component making sure that you have clearly defined this intent for nlu so when we say things like clearly separateable from other competing intents it just means we're building a crisp paid time-off view request type of intent as opposed to something building in one intent that says view pto balance another intent that says request pto so those tend to be a little bit similar so we're assuming that we're just going to build this one topic and it has very clearly defined and separatable intents or utterances so david i'm going to turn it over to you to speak to the nlu component thanks marcel uh hey everyone uh yes so let me share my screen and i'm going to share my desktop over here all right let me just move my zoom okay so um basically i'm i'm serving the back end so i'm already started up studio where we have our uh nlu model which already has several intents um ready to go and they're published as i can see over here at the top uh we're working with a confidence threshold of 75 but we don't have the intent that marcel basically just laid the groundwork for so the first thing that i'm going to do is i'm going to start the the new intent and i'm going to give it a specific to the point name and a good guidance is to not use any spaces when you create the intent to use upper cases for separating the words and as marcel said right it's really important to make your intents really specific but also very separatable and i will get back to what that means as we get into the examples when we're testing the actual intent so uh we created that intent now and now we need to do is add several utterances so we want to know things about taking a pto or vacation as we call it in europe a lot and we like to take vacation a lot in europe as well so my u.s colleagues tend to tell me at times so i'm just going to add several of the utterances that we have um over here and these are just pretty standard utterances and what is important when you create utterances right and you you should basically start off uh creating a list for yourself and basically uh asking your users uh in a sort of a survey what kind of questions they might be asking around this type of pto or you could mine uh the the asks in the searches to see how they try to ask for this type of intent and what's important is the shape of the uh the utterances and that you cover basically the variations in shapes and with shapes i mean like things like like to take vacation right you could also think like to take pto status status and pto is also something that's balanced is an important work with with pto so with that you're trying to capture as let's say a broad spectrum of potential asked by your customer you're not going to add each and every utterance here right that's not the purpose of machine learning you want to let the machine figure out but it's important to highlight important words that that belong to the intent so to capture that broad spectrum what's another one which is really important is obviously the the date right so aside from saying i would like to take vacation we're going to be specific and say hey starting december 3rd right we want to add that as well so let me see the status of my pto and then i'm going to add some more as we go along different variations as you can see here want to start pto and then as aside from want you could have like to start pto you can also see that i'm already starting to capture upper and lower case right and this is for um also for for yourself right as you're building the model to still understand what's going on what what you're trying to capture here and this one you can see here right i'm now starting with that word pto sometimes people are let's say giving a command in this case so you just want to say hey pto on the first and don't make too many words don't add too many words to it how many variation days do i have left so we're basically capturing all these different variations as we go along there we go all right now let's train it for the first time and see uh how far we're doing as marcel mentioned right we we don't have uh yet captured any entities at least i did not specifically designated any of the the words or parts of the sentence as a potential entity but as you will notice is that the the machine is is smart enough to already designate several of the uh or or within each of the sentences what an entity might be especially when it comes to system entities such as dates uh it does a a really good job already out of the box so you don't have to do anything actually to already get the benefits of entities if you want so and and as you will notice marcel will uh talk about that a little bit as she is going to build out the the topic uh you you can but you don't have to use the uh the entity all right let's try a test sentence i would like pto starting november 5th just to see a quick test how we're doing all right this seems to be going well right we're reaching above the threshold 89 i do have one that's really close right that's within the delta so that means that the the first and the second intent is uh nearby and that means from a virtual agent perspective that most likely this will result in a menu being popped up saying uh the virtual agent isn't being able to basically pick one of the two over the other or in this case one of the three actually and it will give you a menu with all the options which is something we don't want and there's another thing that worries me here we have something called general hr inquiry and i know this is an out of the box intent but that's actually not a not the best practice as you start out building your own model to incorporate too many generic uh intents because they tend to overlap very quickly with your very specific intents so general guidance is try and be as specific as you can and keep them separated because as you uh basically identify more and more intents and topics that you would like to cover the general uh intents will start to quickly be a a pain so i'm happy with this result although i do understand that um people could also say things like maybe i would like some free time so different wording from pto and let's see how that responds oh so that's that's uh nasty right you can see that we're not picking that up but we are picking up the entity right you can see that this is the system the entity date and we're picking up the value november 5th so at least we're good over there but in order to make sure that we're also covering off those i'm going to add some more utterances to the to the list just to make sure that marcel is not going to run into problems as people start calling things slightly different than we thought they would be calling it and this is a good practice to basically um keep an eye on as well right as you progress to keep an eye on what people are uttering as they converse with your virtual agent to see if you have any gaps in your utterances or in your vocabulary i haven't talked about the vocabulary yet but essentially let's say something like the time off pto um vacation that is something that you could basically enrich the the vocabulary with to make it really easy for the for the machine to understand when somebody is talking about uh something like pto so let me show you an example of how we would like to solve that and i'm just copying and pasting i know because that just makes my life a little bit easier aside from adding it all together we're going to use a red a regex a pattern because that's the easiest part and as i i will explain this a little bit because it looks really complicated but it actually is not we're going to call everything vacation and the first thing that we do is basically exclude the synonym word from the regex because you have to as you will get into an endless loop error and it will not be able to train the the machine so that's why we're excluding it here and then we're going to mention all of the different variations that we could find in people uttering like pto paid time off time off holiday and vacation and they would all basically replace that with it it would all be replaced with vacation so whenever someone says pay time off it would be replaced by vacation that's what essentially what the vocabulary synonym feature does um and in this case i've also made it uh upper and lower case agnostic right so any of these uh characters could be upper or lower case it doesn't matter right we we took care of that with this regex that's why you're using a pad okay let me save that so now it's added to the vocabulary it works across the full model right so anywhere where you would use pay time off time off holiday or vacation it would all be already be replaced with vacation so you have to be mindful of that and careful as well right so don't be too rigorous in that because you could uh for example if you would have something like software and you would have a software system called human or people you wouldn't want to put that in a vocabulary word and say that always always when you encounter people or human it needs to be translated into software because that's just not true in this case i think we're we're pretty safe um and this is going to work so let me train the model one last time before i hand it off back to marcel to start the topic i can't i can't switch time and um i'm just going to end up with yeah some addresses as it's tested let's see once it's done and i publish it give the machine a little time thank you all right so we published it looking good um let's see how this one works sorry i forgot about that one see if we get a good match now 79 okay we could even even improve it a little bit more if we want to but uh right now we're above the threshold uh we no longer have the uh the general intent bugging us anymore and we're picking up the date so that's looking good let's see one last one test need time off that's perfect it matches general hr inquiry if i really don't want it right um and and and i think that's true i don't want it i'm just gonna go in there and i'm gonna say hey i think we imported this one but we shouldn't be using this one right and it has all these generic requests we just need to make specific uh topics of the ones that actually are important to us and then get rid of this one so i'm just going to remove it most of the time right because you have an out of the box model uh it will it will still be there obviously you can re-import it so don't worry about let's say importing and and then deleting if as your model matures um marcel i think we're ready for you to start the topic all right and i just want to mention too i mean as we're showing you how to build custom topics oftentimes this comes on the heels of you've maybe built out a general topic and you've tried that and maybe it's time to deprecate the general topic and start creating some additional topics so while i'm sharing my screen david did you want to just talk briefly about you know what's best practice around test utterances versus um actual utterances you're putting in the model and maybe how many that customers should keep yes good one so as you saw that i indeed had a little set of test utterances and it's very important obviously to next to the the set of utterances that you have in your intent to also have a let's say a smaller subset of test utterances that you will not use in your intent but that are an actual representation of uh your expected utterances that will will hit the virtual agent because you will use those to test the accuracy and or what we call the performance of the nlu model and to make sure that it remains stable over time so um yeah and it's it's right now you have to keep them let's see like i did in maybe a list but we're working on uh giving you let's say uh in platform uh abilities to to basically store those test utterances and to basically periodically test the system with those thanks marcel for reminding me oh great thanks david so i'm as you can see i'm on the virtual agent designer screen and the next phase of this process is to now create the topic so the topic is essentially your your framework for housing nlu and creating an engaging and interactive virtual agent topic and i want to reiterate you don't need to be a developer to do this i'm going to show you how you can create a topic using the builders so today we're just going to call it uh be a pto topic and we can give it a category so if you want to categorize it it's an hrsd one thing to note too is if i'm building an hr topic i want to make sure i build it in the human resources virtual agent conversation scope the reason is the builders are designed to work in the hr scope specific to hr components so you want to make sure you're just in that scope if you are going to build an hr topic and we can give it a type so we're going to call it a topic but you can see there's other options as well and notice when i select topic i have a bunch of drop downs right so i can create conditions so is it a manager only topic is it an exempt employee topic are there certain roles that should be available and i can come down here and then select my intent so i'm going to select the intent that david just showed you that we created so the view and request pto and if you don't see your intent you can hit this refresh list of intents and you should see it um just note you cannot have your intent active in another active topic it will give you an error so just make sure your topic is inactive if you want to test additional topics and see which one has a better flow and so we're going to hit create from there and we are now in the canvas for virtual agents so i want you all to think of yourselves as the artist behind how to create a virtual agent topic it doesn't need to require a ton of scripting or customization but the goal today is to use the builders and create an engaging experience without going overboard right because remember the process may have looked complex but virtual agents power and capabilities lie in the simplicity and making something easy and user friendly so our first builder we're going to select a date time so it's coming from the user input so anything blue is something that the person would select so we might want to say something like select pto start date and we might want to say what date do you want to start pto so here we're going to do a special step we're going to select an entity because the user experience i want to create should skip this step if somebody enters in a date so that is why i'm enabling entity in nlu and i can also put in an acknowledgement or confirmation message so i might say got it or great choice or you know yeah you're going on vacation and for the format maybe i just want to put date right to keep it simple now i can put conditions on here to to skip or show this node we're not going to touch that right now but my next step now that i have the date selected is to look up pto so again this is taking one of those assumptions we talked about that i am storing pto in service now um so what i'm going to do is just say look up pto balance and we're going to select the table i happen to have it in the hr profile table so you can see for this particular employee i've created a custom field that is housing her pto balance so that's just the assumption that i'm bringing that in through an integration and i don't need to create a script or anything but i do want to add a condition that the user is the person engaging in the conversation so what i'm going to do is select user and then from my list i'm just going to select an input variable of user so this is the best way to dynamically create that particular piece we can save it and now what i want to do is show the user their pto balance so i'm going to choose a bot response because again i want the bot to show them what their balance is so i'm going to title this show pto balance your current is and now what i'm going to do here is use something dynamic i'm going to use an input variable i'm going to use a reference to what we just looked up and i'm going to pull in the value from that particular string field and i can say something like hours if i store it in hours or days if i store it in days you want to make sure your metrics line up right so i'm just going to save this as we go along so we're now looking at show pto balance and the next step is to ask the user if they want to request pto one of the the reasons we do this is if your balance is two hours and you want to request a full eight hour day off maybe you don't have enough time or maybe your organization doesn't allow you to dip into negative balances so what we want to do is just make sure we are confirming that you want to request pto so we'll just say um do you still want to request pto and we're not going to do anything with the entities here but we are going to create a choice list so we're going to say yes and yes and then we're going to say no because that's our second choice and no so notice i'm not putting any conditions on this and i'm just giving it a choice so i can either say yes i'm gonna proceed or no i want to get out of this conversation and do something else so at this point you are making a decision so let's introduce your decisions utility decisions utilities are branches so typically this will be when there's a decision point you want to confirm or you want to give users a choice to do something so the best practice is just to create your choices and since we want the no to just end the conversation it's easier to start there so i'm just going to say no and i'm going to give it a condition and say confirm pto request is no and then for this i'm going to move it to yes and i'm going to say confirm pto request is yes so i can proceed building down this thread but notice everything we've built so far so nothing custom we've just used builders and we're starting to create that dynamic experience so now i want to ask the user how many hours so to do that i probably want them to type something in so i'm going to bring in a text and i'm going to say hours and notice there's a validation so as of the paris release you're going to see some validation formats you can have a custom format so if you want to put some logic around that you can i'm just going to leave it as text and i'm going to say please enter of hours you want to request off example five right says we want to give them an example right because people might enter i don't know a decimal or five hours so we want to just make sure they're entering something and we give them some guidance and we can always give them an acknowledgement or a default but at this point we are going to do something a little bit more complex we are going to create a script action that then calculates what their remaining balance is going to be now i know scripting tends to be that area of oh my goodness i don't have the skill set i don't know javascript this is going to be a very basic script so again think of it as you're creating this topic and you might need to learn a little bit of a new language to do that but the impact is going to be powerful to the user experience so to calculate pto we're just going to call it pto and we're gonna come here and we're gonna give it an expression now when i open this up i'm gonna see some some helpful info in here i can just delete that and i can write the following script so i'm gonna create a new variable called balance which refers to my pto balance and i'm going to first include a input variable so i have va inputs lookup pto balance and it just fills it in nicely for me and i'm going to tell it what field so again this field is coming from this area so if i were to look at the dictionary actually it just shows it when you right click um this is where i can see what that column or that field name is here so i'm just telling it go here and we're going to use a system variable a va vars and we're going to create a new script variable here called new balance right because that's the new calculated balance equals balance minus what we just created va inputs dot how many hours and we are going to close that up so that's going to be our script action very basic right remember we created a new variable and it just equals this lookup this specific field related to the lookup and now we're saying a new variable equals this balance minus the number of hours so we're just calculating it for the user and don't remember to close or remember to close your bracket so i'm going to save that and we're just going to create a script variable here that says new balance right because we need to tie it in and give it a default of zero so that is probably the most complicated thing we are going to do today so we calculated the balance and now we want to show the impact to the balance because remember when you request pto usually you want to see how many hours you have left over because you might be thinking about your vacation or or the time you're requesting after the time you're requesting so it's always good to see that so you might want to say your new balance will be and again here we're going to go to our script variable now and we are going to use the new balance and just say it's it's going to be this many hours is the new balance so so far we've calculated the impact but we haven't created the request yet so let's do that so we're going to use this utility called a record action and we are going to create a an action node so we're going to say create pto case and we're going to tell it to create a record you have the option to update but we're going to create here and i happen to know it's in the total rewards hr case table so what i can simply type is total rewards it will pull up that particular table and this is where i can add the fields so what i'm going to say is i want the hr service to be general pto request and i want the open for right because hr uses open for and subject person i want that to be the user so i'm going to go to input variables and then select user i want the subject person to be the same so i'm going to come here again do the same thing and then you can do things like assignment group and things like that so i'm just going to do description and i'm going to say pto requested for uh this particular date so i'm going to go to input variables and see where it says select pto start date so i'm going to say this date and i can just do something like assign to and maybe i just want to make sure this goes to a particular user or i could say assignment group and have it go to let's just say hr tier 1 is where i want to send it for sure usually you set this at the template level but you can also set it at the virtual agent level if it's not established at the template level so i want to save that now and just make sure down at the bottom that this is going to execute right so i validate that this is returning true i can save that and now i have two more actions to do i want to create a display card and tell that user that your case is created so what i want to do is now have this refer to a record and the record is going to be the create pto case that we just built in the fields here i want to select the hr service so i'm going to just go down and i can say open for and the case number so we're going to pick the number right so then this is going to be our record and i have one last node where maybe i want to write back to that custom field in the profile and update the balance so this could be something like update pto balance here so for this action i'm going to go update a record and i'm going to update the lookup pto balance right because that's where we originally were using the utility to go ahead and look up that balance and i'm going to add that field so i'm just going to say pto balance is and here's where our script variable comes back i wanted to update it with that new balance we calculated using our script action so and i can set this right if i wanted to run this every time or i could set this to false if i don't want it to execute so i have the options to do that and now let's end this node here so we built this area we're just going to drag it over to end and that is going to be our topic so normally you would save it you would test it and publish it in this particular case we have built it so i'm going to now test it as an employee so i am in as christy right and christy's going to be my employee and we're going to go ahead and test this out so before we do that um it looks like we have some q a and some questions so did we want to pause and take questions for now yeah so marcel we have a question um what if you build in global scope rather than the hr scope what is the impact yeah that's a great one so the biggest impact i've seen is when you create the case card and the case by the case card i mean the card that just shows you the confirmation of your case um usually customers want if you're creating that virtual agent they want it to open in the portal view and if you build it in global scope it doesn't it's not going to open in the portal view so that kind of produces a less than desirable user experience and it's just the way hr was written it looks at a system property that tells it what portal extension to open and so i would say that tends to be the biggest issue that comes up is you're building topics um that involve cases but i would also say the layer of security right so if you you know end up scaling up and you want your hr team to really own and manage their own topics and you know maybe they kind of want their own work area typically like the security and the scope around hr gives you that freedom as opposed to having everything built and global so there's the security aspect and then there's the aspect of we want to make sure the builders are optimized thanks marcel did we have and it looks like we had a couple other questions in the chat okay so it looks like these were all answered i'm gonna close this out okay so let's do the fun part let's test this so i've already opened virtual agent and i can type in something like i want pto on november 25th and so i'm coming in and again right we are aiming to create experiences for how employees would naturally interact as opposed to how people with a more technical mindset would think that they want to interact so well it's thinking the goal here is to have it skip that first step of the calendar okay so it says got it right so right off the bat we receive our confirmation message so that tells me the entity is working properly so david awesome with your utterances you built it in a way that you know i as a user maybe a non-technical user would go in and just ask for what i'm looking for right so i'm presented with my pto balance and sweet i have 19 and a half hours i can definitely request two days off so yes i want to proceed and i want to enter in this number so you know two days eight hours a day so i want to enter 16 because i want to take a total of 16 hours and now i'm presented with a case right so back to the original question if i click on this link because i built it in the hr scope it's going to open up on the hr what we call the hrm ticket page right so i can manage it i can ask a question i can see any updates right and notice our case number is is 1209 here so back to the user experience i can say no that was super easy um you know i was able to request pto with my start date and let's validate on the agent side so as roger roger's going to be my agent we can see that this case came in for christy and if you have predictive intelligence set up this is an excellent example of how predictive intelligence can help supplement the power of virtual agent so maybe you have this link to pto but maybe this is something that should be actually a leave of absence so again you know this is how you can leverage predictive intelligence with virtual agent but i can see my case has been created its source is through virtual agent and i can see my start date of november 25th so that person can then start that case mark that is ready for work and go ahead and process that request so that really brings it full circle all the way from beginning that topic to building that topic out leveraging nlu and all without having to have really a developer skill set to do that and that's really you know when we talk about what's what's best practice for virtual agent what's optimal again you know taking a paid time off process it could be incredibly complex there could be points of validation there could be i'm an exempt employee i'm a non-exempt employee maybe i accrue hours differently maybe i get a bucket of hours at the beginning of the fiscal year starting august 1st maybe you know i can't draw negative balances so there's all sorts of things that might be going on but again we want to think of virtual agent is how do you solve for the most common request and make it easy and make it actionable so there could be a ton of fringe use cases and different twists and turns to that process but virtual agent the power is really solving for you know the bulk of those requests coming in and usually those are going to be pretty straightforward i want to request time off how do i i even begin from a process map and take that start to finish to give something that's that's powerful for users to go ahead and engage with so i think we can turn it back over to q a and i'll just share with you guys our community slide we encourage and invite you to visit the virtual agent nlu community we have a ton of great contributors business users developers everybody in between and it's a great place to just get answers and find resources and of course recordings to virtual agent academy so it's just a fantastic place to go if you have questions so nabil any other follow-up q a um no i think we answered it thanks marcel appreciate your time thanks david it was a great session