hi everyone good morning good evening wherever you are welcome to virtual agent academy my name is marcel goodsell i'm a product manager with our employee workflows group here at servicenow responsible for customer adoption of virtual agent and conversational interfaces and i'm joined by my colleague nabil asby who is responsible for product management for all of conversational interfaces so today's session is going to be an add-on to some of the branding sessions that you may have attended in earlier sessions we're going to focus today on advanced branding to really fine-tune the user experience and so what we're going to focus specifically on today are um a couple things around system properties so how do we control the number of intents returned using natural language understanding um so how many intents are the user going to see in virtual agent if it doesn't detect the right one off the top and then we're going to go into just the different conversational states for virtual agent and take a look at how you can adjust those to optimize the user experience so especially around things like timeout and how often that that conversation will stay available so um before we get started feel free to post your questions into the chat we'll be taking periodic breaks to address any questions that come up and as always we'd love to hear from you so getting into virtual agent nlu intense return so if you look at the behavior of how virtual agent returns nlu intents you're gonna see a couple ways in which we categorize a correct prediction so correct could mean um i'm going to virtual agent and i have a question about password reset and you know based on my utterance it drops me right into the password reset intent um but i could also have a correct prediction where i'm requesting something related to password reset and it returns you know maybe two or three different options related to that because it says okay it looks like you're asking about password reset but you could be asking about these other things so we're gonna give you the option to choose um so that's where you're going to see you know correct predictions categorized a couple different ways and the question is always how do we use virtual agent to control how many actually get returned so if we look at a an experience of an admin going in to test the model and if they use a pretty broad intent something like i need to update information um something that could easily match multiple intents then you know typically what you're going to see is the top three intents returned based on the test utterance and as always a best practice it's it's a good idea to you know keep a separate list of test utterances they could be a sampling of your utterances collected from crowdsourcing or they could be you know a different set of test utterances but really we want to see how the model holds up under these utterances but anyways so the top three intents usually get returned and you would expect the behavior in virtual agent to return what you're seeing in the model and what actually happens is when i enter the exact same utterance i'm actually getting six intents and oftentimes customers will say gosh you know this is a lot you know we want maybe two or three but we don't want to bring back you know five or six um depending on how closely the match is or how um how you know similar they are based on the the confidence delta so to address this we have a system property that controls the number of intents that get returned specific to the virtual agent client so you can see this this property here so we're going to take a look at that in the demo today and then if you adjust that value so let's say i change it from the default value of six to the default or just a value of three or two the result with the exact same utterance is far fewer intents that are matched and so oftentimes what we're hearing is this is a better experience it's less overwhelming and it just brings back you know the the really highly rated top three if it's not a a correct prediction slam dunk so um before we get into the demo are there any initial questions on this topic so far marcel uh we have no questions okay all right so if we go ahead and take a look at what you know typical experience might be with the default so i'm going to initiate a chat and this is from you know a typical employee service center portal and you know virtual agent conversation is launched right so i can type in you know something very um unspecific so we can try that example and you're gonna see here i get quite a few different responses and if i was a user i would think gosh you know i need to update my info but um maybe i was intending it to be a phone number or an address but you know this is this is a lot of information so you know now i could be confused because i have too many things returned so what i'm going to do is go into system properties so switching over to my admin hat so if you go to sis properties dot list and that should bring you to the system properties list and if i type in display so keywords are critical here because you can see there's a ton of system properties i'm going to get my max number of display properties and so what's great is i can see it defaulted to six um you know so remember when we looked at the experience of virtual agent you know i may have gotten up to six but then if we looked at you know if we're gonna test on the model you know and i say something like i need to update info you'll notice that you know the model may have returned some different results here but you know it'll say okay here's the the typical top three and i'm using a vague utterance on purpose just to show you you know some of the ambiguity that might come back out so if we come in here and we adjust this to three or even two um that's all i need to do so then i can update that and so i can say okay you know i only want it to return you know maybe a total of two intents coming back so now that that's been updated we can come back in here and i'm just going to refresh it and just to make sure that that change took place and when i initiate chat again and i'm coming back to the virtual agent we're going to start a new conversation [Music] and so as we're going through and you know launching into some different topics i'm going to enter the same exact utterance and at this point what it's going to do is only return to which which often times is arguably the better experience so using the system property really helps control how much you see get returned back to the virtual agent so i'll pause there do we have any questions no questions so far okay looks good all right okay i'm assuming this is clear okay so if we come back all right so the next topic is conversation states right and this is this is a pretty exciting one right because you may be familiar with live agent states from connect support um but really virtual agent conversation states take on a couple different functions and so first the different states you're going to see span everything from open to to cancel to completed to faulted to chat in progress and you can see the different definitions we attach to each state so you'll notice you know some might take on um you know a one-to-many relationship some might take on a one-to-one relationship so you'll notice the open state could map to somebody selecting a topic um somebody selecting to transfer to speak to a person or somebody initiating a new conversation so any of those could be categorized as open um you know conversely faulted could take on a couple different shapes as well so the most often time we see a faulted conversation is when a conversation will timeout so if a user abandons that conversation you know it'll count it as faulted so that's where you know some of the the advanced branding comes into play we don't want users to necessarily abandon the conversation but we may want to um you know curtail some of that so that you're not seeing as many faulted conversations the other you know reasons it could be faulted was it you know errors out or somebody cannot transfer to a live agent maybe a live agent isn't available but you'll see the other states completed that would be the the completion of a conversation start to finish and then a chat in progress so if a chat has you know been initiated and somebody is actively connecting with a person so these are all available in docs but this is essentially how the states are laid out and you know if we want to go ahead and modify um specifically when virtual agent will timeout right so out of box the default is 24 hours so if you think about it you know you're you're engaging with virtual agent and you have a question about adding a beneficiary to your benefits and what you're doing is you are you know typing away you're looking for information you find a knowledge article that's helpful and you say okay great like i don't need virtual agent anymore i found my answer but you're not necessarily marking something is finding the answer and so because it's set to default to timeout every 24 hours that conversation goes idle the user has no idea that they need to do something and then it gets counted as faulted when in fact the user actually found something very helpful and it should be counted as completed so that's where we can use um properties like time out to adjust that to maybe minimize the number of faulted conversations so what comprises timeout is going to be a combination of a system property so it's actually you have to add the system property it's available in docs and then a scheduled job which then will execute against the value set in the system property and so we're going to explore both of these in the demo how to set it up and then how to adjust the scheduled job and then finally you know when you're looking at the the conversation table so this is a screenshot taken from cs conversation you can see you know what conversation states um are most utilized which ones are completed which ones are in progress and which ones are faulted so this is just a way to essentially report on the progress you're seeing and the different states that you're seeing users experience in virtual agent so i'll pause there nabil do we have any questions uh not yet but i would like to touch on the uh the state when you start with virtual agents so you might be monitoring how many of these of your conversation can have for example completed which everyone's looking for the most important when you look at the performance and the effectiveness of your chatbot it's not where you start is where you turn to where you grow into so you might find like you have 20 percent of completion in the early stages you've gone live the most important that you improve this 20 and you do not decline so we found a lot of the conversation in the early stage my get faulted you know because you do not introduce many topics right you go live with very small number of topics five or ten so the end user will come to the virtual agent he won't find the right topic that he's looking for so you would just leave the conversation and that will end up in a faulted session that's okay that that what we want to focus on is trending the number of completion up and defaulted less so the starting point is uh it is important but as what's more important is the growth into better accuracy yeah thanks nibil and that is a really important point to make sure you know even if your baseline isn't looking the way you want it to look you want to make sure you're you know working toward a greater greater conversation state of um virtual agent conversations in the completed phase so one way to get you there is to review how these properties work so i want to show you where this lives and docs and we're going to actually use this to set it up so you'll notice that this is part of the virtual agent documentation so you can see there's a page in conversation states there's a page on closing idle virtual agent conversations and we want to add this property so it doesn't come out of box by default um so i'm simply going to copy that and it gives me instructions on how to set this up which is nice so i have this property highlighted i'm going to come into my instance and you'll notice i don't have a system property in here um i searched on idle i couldn't find it and so what i'm gonna do now is add this property let me make that a little bit bigger so all we're gonna do is give it a name and again the name has to match what is in docs because the scheduled job so the scheduled job here that sits to execute against the integer value is looking for the name of this exact property so just make sure it's exact um we're also going to copy the description just to be thorough so we'll put a description in here and i'm just going to give it a value of 15 seconds right because in our docs it says you know the value is um a number in seconds so you know just for for purposes of this i can say 15 right and go ahead and submit and so now we can see that property if we search on idle so when i come in that property should be already set up and you know as a next step what i would do is go into the scheduled job and go ahead and look for it there but in the meantime you know if we wanted to look at some different virtual agent conversation states we can measure those in the conversations table all right let's try this one more time here we go okay so now we can see that our property is here right and we set that up the next step is to go into scheduled jobs right so what i can do is i have it on a favorite but i can look under scheduled and what i'm going to do here is search for abandon and i'm going to pull up this particular scheduled job so what this does is it sets a parameter on when to run the job which will then look at the integer value of that property so if i wanted to adjust this for example i would select periodically and i would say um you know i want it to repeat maybe multiple times a day if i select days and i said one or two it would repeat every two days but let's just say i want it to um you know set every 15 seconds right to match the integer value so i can run this job every 15 seconds and i'm going to save it right so we've made that adjustment here and then you can execute now to see how that works and so we've executed the scheduled job so we should be able to see some different behavior now that we've made those adjustments so coming into um the virtual agent before we start looking at what that experience looks like um are there any questions i'm not here marcel one one note i would like to add about the scheduled jobs so in our exercise here in the demo we're gonna make it 15 seconds but uh in your production don't do that just just think of ex every few hours time after conversation um it's it's unlikely that someone want to come back to the virtual agent within literally a few minutes to ask for the same thing um so be careful not to use uh very short uh intervals for time out your conversation yeah so nabil would you say like an hour is probably average for the customers you work with um so it depends on the usage if it's an um if it's an external or in case of csm or versus employee experience right um i think 24 hours would be good but like minimum is uh is an hour uh this way if someone would ask for multiple times during the day which is unlikely to happen that would still work for him and you won't overload the instance with events yep great okay so if i start a new conversation here right we're gonna go ahead and start a new conversation um so you know i can choose from a topic right i might say i want to go into a pay discrepancy and then let's take a look at the different conversations going on and so notice my session right now i'm in an open state right so it's open if i come back and i say you know i expected a different amount on my pay stub you know and we kind of come back and see is it progressing is it you know changing it's still staying open um but what i can do is i can say you know i expect a different amount of amount on my pay stub but the knowledge i'm seeing is not helpful you know and i'm gonna go okay this this isn't helping i don't want to proceed with this i want to connect with a live agent so i'm initiating a chat and then notice on the back end if we keep refreshing this table it changes to chat in progress so while i'm waiting for my chat let me go over to roger who's then going to accept the chat so roger now accepts the chat and that conversation can keep going so roger might say you know hey what is your question about so he can kind of keep going on his end and the user could say um i missed entering my overtime hours last week right so maybe it warrants a lengthier conversation the argument or the um you know that the question or support required is a little bit more substantial and so the user is reaching out to an agent in this case right so they would continue that conversation and then ultimately the user might just end the conversation they might say okay great thanks for your help thanks for the case you created and then go ahead and end that so once we end it notice that the chat in progress moves to completed right because we've now successfully according to virtual agent completed that conversation um but let's say you know on the flip side i start a conversation and you know just to show you how the property works i could say pay discrepancy you know and i could go through the motions again but let's say i go idle and you know i'm just like okay um you know i got sidetracked maybe i got a call and i just basically left the conversation open so within 15 seconds what it would do is go ahead and time me out so we can track this progress so we can say okay current conversation is open but it'll beep once we hit 15 seconds um you know and then let me know but in the meantime right if you wanted to okay so see where it says i left the conversation so this is an indication that that job is run we've hit the 15 second limit mark which is again as nabil mentioned um we don't recommend you set it in 15 seconds but for demo purposes this gets the point across but if we come back in we can see that it is now faulted because that person just never responded so um just to give you some deeper insight into the states of the conversation you know we can come in and look at the different tasks that were completed during that so was one cancelled meaning did i move off of that and go into a search um did i connect with a live agent did i close the conversation and then we have a record of the messages so was it inbound outbound you know when was it created um you know was the closing conversation invoked live agent support etc so this just gives you a more detailed snapshot and the percentage um of where that was completed so that's really how you would use the property to then govern um you know how often this times out so you may you may want to have a reminder that says you know hey so-and-so um i see you've been idle you know can we help you at all or would you like to end the conversation so something that may prompt user behavior to proactively end the conversation as opposed to just leaving it abandoned which then defaults to faulted um so there's some creative ways where you can work with that but just simply wrapping up um actually um nabil do we have any questions from you anyone before we go back to powerpoint no oh all clear okay um so that really wraps up today's topics so we took a look at how to control the number of nlu intents returned on the virtual agent client and then we looked at how to control when idle conversations actually go idle and how you can really shape the user experience around that and again these are requests that usually come from customers so i would say if if this got you thinking today and you have other topics that you would love to see covered or some creative ideas around a topic please feel free to share those with us either on community as a comment just to let us know what types of topics you would like to see in upcoming sessions so we have a question now from frank uh does the schedule job change the conversation state from faulty to completed uh so the answer to that no that scheduled job that we have is only for the abandoned conversations to uh move them from open to faulted or what we say abandoned once the conversation is faulted it's gonna end the user the end user won't be able to interact with it anymore so marcel showed us i've touched you around the job it showed you that the conversation the session is over and it is closed so that won't be possible you have any other comments or um you know any feel free to also to to share any upcoming ideas you might have whether it's related to how to use a system property or to create a specific type of user experience this is very helpful marcel especially for for understanding the states it's really important to understand them to be able to measure the quality and the performance of the virtual agent uh once you go live and again it's uh it's it is important where you start but it is important where you uh grow into with these states i don't see any other questions myself for today thanks a lot um thanks everyone and see you in two weeks all right thanks everyone