hi everybody welcome to virtual agent academy um good morning good evening wherever you're located howard i will kick it over to you to begin great thanks marcel uh good morning good evening afternoon everyone my name is howard shaw i'm a product owner in the iit customer experiences group where we have deployed a virtual agent for our now support operations and i'm joined with rajiv jayaswal one of our wonderful developers in our team who's built and helped us to build a really interesting slack integration for our virtual agent and we're using it in a different way so some of that today so what we'll do is we'll jump into showcasing a little bit about how we're using now support i mean virtual agent internal support and then uh jump into a couple of the scenarios that we work that we've built for our slack integration and then uh reggie will give oh and also showcase a little bit of the code that we put on that as well so just uh if you wouldn't mind we'd love it to see if you could give us some some feedback at the end of our session today it's about a half hour today and at the end we've got a survey in our zoom chat that we were hoping to get your feedback on so we can better tailor our solutions or what what we present future academies to you so with that being said i'll just jump right on in and first off i just wanted to have a quick introduction slide about our virtual agent and now support so we call our our virtual agent cody so um what we try to do is really what we try to provide in our now support environment is for our customers is to provide as many self-service options to you so that you can really access anything you need from servicenow at any time of day and night and the hope is that if we can automate many of the more repetitive or simple processes then you don't necessarily have to wait to talk to anyone you could get a lot of those things taken care of immediately so we've activated about just over 75 more continually adding more and more but we've activated about 75 conversations for our virtual agent for cody to be able to support you in just about anything across the board like if you need to create a case or if you need to put in a request through our service catalog if you need to manage plug-ins or if you just have general questions you want to know about our um i don't know our backup retention policy or things like that you can generally try to ask cody just about anything and the hope is that we've given you enough content that we could hopefully either answer your question or point you in the right direction or even help you to complete that task directly within the virtual agent so we're also leveraging the servicenow platform natural language understanding models because we wanted to make sure that cody could understand as much as possible so that you don't have to specifically type in a keyword or anything in particular you could just kind of talk in more natural language and the hope is that cody will be really good at picking that up and that's kind of what we're using our slack integration for today is to keep an eye on those types of things so let me jump into just a quick example of the use cases of a typical slack integration so with our virtual agent for servers now um you know you can integrate with slack with microsoft teams the typical use case is this first one here where you uh you activate a virtual agent in your instance with your slack so that your users your essentially your employees or anyone else can basically type directly in the slack window to your virtual agent and interact with your instance without having to log in directly into your instance so if they have questions or they need help and they're not even logged in they can just go straight into slack and interact back and forth with a virtual agent just like if they logged into the instance and activated the virtual agent itself and so for us in now support since we're using it for our customer base it's essentially our now support portal is for our all of our customers who need help from servicenow as well as our internal global technical support organization uses our systems to be able to work with you to provide that support and so since it's a mixture of both external and internal users we couldn't really activate a typical slack integration for external customers because not everyone has it those types of things so we did activate that integration specific to our internal users so that they could interact back and forth with our virtual agent without logging into our now support portal the bigger one uh for us that we're using the one of the primary reasons we use it here is kind of a more interesting scenario for us was we wanted to monitor how well virtual agent was doing in our now support portal and also just keep an eye on it you know make sure everything's going okay uh you know any problems that we encounter so we activated the slack integration so that virtual agent can post directly to a private internal slack channel that we monitor so that we can see what's going on so for example a couple of areas is if a customer submits uh surveys or feedback to us we want to know that right away so that we can hopefully respond as quickly as possible also at the same time if uh if someone typed in something and our natural language understanding model didn't understand it then we want to know that right away because then that will help us to say hey maybe we should retrain our model or or add more variation so that our virtual agent will be better the next time you use it kind of thing so if you see at the bottom here the uh the graph we're showcasing is our customers who use our now support in our now support region they access through our service portal as well as our mobile application that we provide off of now support and then our interests both us and it as well as our global technical support organization uses the portal and mobile apps as well but they also have that slack integration directly to our virtual agent so uh before i move on to the next slide i'd pause for a moment see if there are any questions from anyone no questions yet howard okay great all right so let me then jump into uh why did we do it so we kind of talked a little bit to that but essentially some of the benefits that we saw was you know we want to be watching our virtual agent and you know not necessarily find out hours or the next day that something went having trouble with it we essentially wanted to know right away anything that's going on in our virtual agent we want to be notified immediately so we created the slack channel a private internal slack channel for us to be able to keep an eye on it and the hope was if customers are providing um you know helpful survey feedback to us you know we want to respond right away because we want customer our customers to know hey you know we're totally watching and keeping an eye on on your feedback and we want to be able to take as much of it into account as possible so this allowed us to be notified the moment someone submits a feedback so that we can take a look at it review it maybe take action right away as well as respond uh much more quickly on the other side we also a second scenario we activated it for fallbacks so what a fallback is with virtual agent is essentially if you type something into a virtual agent and our nlu predictor model so basically our machine learning if it says normally what would happen with our machine learning is that we've trained the model for a specific topic and virtual agent to be able to pick up on what you type as well as a variety of variations so we took a lot of time to try to make sure we've pre-trained the model to say hey let's think of every possible way that somebody could ask for let's say you know information about the knowledge conference and so we try to train our model with with as many possibilities or variations as we can come up with uh but if say for example somebody types um you know what they're looking for in a completely different way that we didn't necessarily predict and uh our our machine learning will come back and say you know what i don't know what they're asking for here so that's considered a fallback and so in that case we wanted to be alerted to that because you know that's not the best experience you know virtual response quickly as possible but the thought was okay there's a fallback then we should uh post it to our panels so in i.t we can keep an eye on and maybe review it right away and say hey you know if we can add that to a model and and retrain it and continually improve our model then hopefully the next time someone comes in if they type the same thing they won't get a fallback they'll actually get the right answer in that case so with that i'm going to jump into a quick demo and i wanted to show you what we're going to see so first off i'm going to demo the um the notification of a survey so when a customer fills out a survey what exactly is happening here so i've got a virtual window open in our support service portal and this is just one of our qa environments but i'm what you see down here on the bottom is our virtual agent and you can interact with it in a variety of ways you can just start typing uh what you're asking for ask it in more sentence form you can just type looking for so if i typed out anything then if we've trained our model properly then virtual agent should come back and provide a response of uh what you know hopefully to exactly what you are and of course when i'm demoing it's very it's not working the way it was right before the it's a little bit on the flip side so um what normally would happen here is that uh the the system would then immediately start working with you so let me go ahead and uh refresh my screen i don't know what happened i guess i had my window waiting too long so just one moment and howard while you're pulling that up this is a question for both you and rajiv question from from an attendee how do you control multiple people responding at the same time using the virtual agent from slack so if there's several people um you know kind of maybe tackling a response or sending a message in sure that's a great question actually in fact the typical um the typical slack integration is that you as the individual user are interacting with virtual agent uh directly right so there isn't it's more of a one-to-one type of conversation right so you're talking to the virtual agent and the virtual agent is responding to you directly so it's not like it's a like a party or chat line kind of thing in that sense so you're working with it directly in other cases where we're monitoring there are multiple of us who are uh watching but we don't necessarily reply in our um in our slack channels uh when the ones that we're just monitoring on kind of thing from there so okay great yeah and then one other question from the audience um can you explain a little bit more about you know kind of the need for the slack channel for the help desk um because the question is you know don't you really want the these created as incidents if there's a real issue um you know kind of your approach for a slack channel as opposed to like the traditional incident creation yeah see there's a couple things in fact that's a great question what we're moving towards uh with our virtual agent is what we call well we call it as a guided versus assisted approach right so on the one hand you might have the the approach that you can ask a virtual agent they can provide you answers maybe give you links to where you can get more information like for example in this case with partner deployment registration we're giving them more information on how partners can register with servicenow things like that but there's also the assisted side and that's that's the side we tend to prioritize and we're trying to make sure that we have that meaning we want virtual agent to do the task for you right so you don't have to use don't just use it for a guiding type format use it so that it can help you actually to complete that task and one example is creating a case right so today um what we're building right now is the ability for a virtual agent to interact with you directly and ask you the appropriate questions and then go ahead and submit the case for you on your behalf so you didn't have to go fill it out yourself kind of thing so you're interacting with virtual agent um and then if you were in that integration that could be the same experience so you didn't have to log into the instance and you could submit a case um in that case now the other way is that um what we're moving towards is more of a using more and more of the ai like sentiment analysis or you know branching off in our topics to say well what if uh you know we couldn't quite answer their question it wasn't going in the right direction then virtual agent can say you know what i'm going to go ahead and either escalate this i'm going to generate a case for you or eventually what we want to get to is i'm going to escalate and bring in a live agent to work with you as well and i can pass all of the discussion history that we've had in here uh direct that agent and or into the case to save you time from having to do that so that's kind of what we're working on this year uh as well for that so great that's very helpful sure okay great so finally came up so i apologize for that always in a demo it slows down i just wanted to showcase what the um what it would look like if you asked a specific question and how a virtual agent would reply to you and then in the other case what you can do is you can keep typing more questions or if you just want to see what's available like for example in our case again i mentioned we activated about 75 virtual agent topics so you can just select from the list the one you're looking for as well and just go straight into it so like as an example with our feed and so um in this case uh the way we configured it is we have our virtual agent asking for feedback on a very small percentage of conversations that users have and we try not to inundate people much so that's why we created this the percentage to be very low so it's just asking quick question how happy are you with the service with the virtual agent and then from there do you have any additional comments and so in this case i'll say um you know i i might be able to provide some sort of suggestion but if i just want to type out anything it will record that comment and uh and then eventually what will happen is it's now going to be posting a um specifically a uh message to our slack channel so if i open that up now let me make sure that that's showing up if you see right here this is the slack channel that we've activated for monitoring what's going on in our in our instances and if you see what it brings up here is it shows that we receive feedback from some user this was their rating and whether or not they provided any sort of specific typed uh response and then the other thing we try to do is provide the last three topics that they were interacting with on virtual agent and the reason we provide that history is that in case someone types in something like oh i had a problem this didn't work for me we can we can kind of get an idea of maybe what they were doing just before they provided that feedback and that'll help us to um maybe do a little bit more troubleshooting when we go through our log records to see you know what happened to this user do we you know is there something we need to fix with our virtual agent essentially so it helps us not only from a feedback perspective but also from a little bit from a troubleshooting or getting a sense of what was going on when they provided that feedback so let me jump back into the powerpoint real quick and just showcase that again again this is what we saw here um where you provide the the notification i'm sorry you fill out your survey and the notification is something that then we get on the uh in our internal slack channel so um and it's generally what we've done is we've set it up for the last step of that survey the other scenario before i jump into that but any other questions marcel before i jump into the next scenario okay uh how would we do we do have one sorry um so once the survey response or freeform feedback is seen in the slack channel you were that you were showing in that previous screen right should this create an incident or a follow-up in service now to go ahead and address um you know perhaps a bug or an enhancement request so the way we've configured it at the moment is no this survey feedback is more we're looking for you know how is it working for you today if there is a problem because we're monitoring it it's more of a manual flow for us is that if if we see something that says hey i'm having a problem with such and such then we'll turn around and forward that and or create a case on their behalf once we see that we don't want it to kind of end up in a survey kind of uh you know black hole kind of thing so if we see that we'll we'll take action and notify the appropriate teams to to to jump on it but generally we've only activated this just so that we can keep an eye on things so far that certainly is a great idea for a next step like we could take that and say hey if you're finding a problem on this area then maybe we should immediately notify or or you know log a case in that in that situation so but that's a great idea okay great and one additional question so um once the it support gets a message of ask cody via slack what's the next step so that's where we have our processes in place that as a team we've got it alerting us and what we'll generally do is either we typically like to reply as quickly as possible to the customer and either you know um thank them for their feedback provide more more background as to what they're asking for maybe answer their questions and again like i said if it was a uh a problem then we would you know get the appropriate teams or support folks involved in that case but it's more of a process that we have that goes with this that says okay well once you've activated the notifications you know let's make sure that there's a process so that we don't let these fall apart kind of thing from there so um so it's more just business process for us that we have at the moment okay and i'll read our final question um came from the audience nabil has answered so the question was is is this also available with zoom or google chat um and the response was the integration is not available but it can be built via apis today so if you wanted to integrate with those um applications as well and last question that just came in um can you get the response via ask cody so howard do we need more clarification on that or or can you take that one um i think what that the question is is so if we see a feedback here in our slack channel could we type it back directly in our slack channel and then would that go back to the directly to the user who typed it let's say they were still in their virtual agent session at that moment that's actually a really interesting idea we didn't activate that mainly because um the question of well if we didn't let's say we didn't respond for a few minutes or something like that if the user was then uh you know went on to another page or was no longer in their virtual agents so we didn't do it that way only because we weren't sure that they would still be there so the thought was what we turning around and doing is um we generally will send an email back to the to the user um that you know hey we just got your feedback thank you very much and you know if i can answer more questions i will we will on that um but generally we do that through email just to make sure that they get it kind of thing so okay great yeah no more questions okay so let me jump secondly into our our our next scenario which is essentially the fallback so again if someone typed in something that was unusual or something we didn't pre-train then a virtual agent may come back and say hey i'm sorry i didn't i didn't get it you can either retype it again or you can pull from the show me everything list of what you're looking for when that happens we essentially get a message like this in our slack channel that says hey we got a fallback from so and so this is what they typed and that will allow us to better understand um you know what what were they looking for and is that an utterance that we didn't necessarily pre-train or think about and we should and so as an example let me jump back into our slack window i'm just going to scroll i'm going to scroll up just a little bit so you can see um this is examples right here these three are examples of utterances i just kind of played around with i typed into a virtual agent and it came back as a fallback because we don't have one we don't have a performance scorecard topic in virtual agents so we didn't have that trained but then secondly we don't have um uh you know supercalifraging we didn't train that one either so in that case it came back as a fallback and we would get to review that so what we'll see then in that case is we get that fallback message and then what we can do in that case is sometimes we'll go back to our our nlu model and check out to see what we've trained it on and so this is actually an interesting example because sometimes we see things like um you know we tried to train our knowledge conference and we try to think about all different ways people could ask about it but one thing we ended up finding and this actually worked exactly in this scenario for us we got a fallback where someone typed in k18 for knowledge 2018 and we saw that and we thought oh you know what actually a lot of customers type it that way we know we didn't really think about it when we were training the model so what we ended up doing was okay yeah that makes sense let's add that to our model and and add it as another utterance so that it understands a little bit better the next time in other cases if someone types in something that might be too obscure that we don't think anyone else will type then we probably won't trade added to our model because we want to be careful that the model doesn't get overwhelmed with too many variations and that might affect negatively impact its accuracy in the future so we want to be careful you only have so many but at the same time make sure the good ones are in there so one customization that we built this and another piece was we created a log record that every time a fallback is recorded it gets into this log separate log record that we watch as well in addition to the slack channel and then we can go in here and so instead of going into the nlu model updating it retraining it pushing it back through to production what we have here is the ability to checkbox the ones that we want and say you know what the this one makes sense let's add this to our model and by by initiating it here it will automatically go ahead and add it to our model go through the training process and eventually get get into production so it's just a time saver for us from an it monitoring side but this is an example of something that we thought would be valuable to add as a customization so with that i'm going to jump over to this slide to showcase a little bit more i know i think we took much more time that i wanted so i apologize for that i want to transition over to rajiv make sure he has time to showcase not only what um what he's got here talk to this slide but then also show a little bit of a demo of of the code so so i apologize for running along but rajiv if you want to chat here let's you can go ahead and kind of talk to this slide all right thank you award hey everyone uh my name is rajiv jaiswal i'm senior application engineer here here at servicenow uh today i'm going to talk about uh two parts one is the slack and another is flow designer uh the two demo that harvard heard just so now uh i'm just going to show you how we have implemented that and i'm also going to show you how we can do that from this class uh since we don't have much time uh let me try to do it a bit quickly uh how can i share my screen yep sure i'm sending it over to you right now okay all right yeah okay screen yes we can see it all right so yeah so this is the instance where we are doing the development part the first part that we saw was uh the survey feedback right so in the virtual agent i'll just show you the topic where we have implemented the survey so uh this is the survey survey topic that we have created uh let me zoom it so for the first time uh whenever we trigger this survey topic we try to ask uh uh to choose the rating from uh one to five right so once the user uh choose a response we try to divide that into a uh like we we have map like this like very satisfied is having value five somewhat satisfied desiring value four and so on and uh here we try to show that if user has selected four or five then we we tell that the user is happy and if the if they have selected value 3 2 or 1 then we we assume that the user is not happy it is a negative feedback and uh for both the positive feedback and negative feedback we try to ask the comment if they want to provide the comment or not if they want to provide the comment we we ask the comment uh as a text text box and then and then once we receive the comment uh we just display the thank you message same thing for the negative feedback as well and in the last when before ending the conversation uh we are trying to create a record in one of the table so this is the table we create a record uh with all the feedback details and based on these details we uh try to create a flow designer so to integrate the slack we need two steps one step is a setting of the slack slack application in your workspace the another is the flow designer so flow designer i'll just show you how flow designer is configured so in the flow design flow designer is altogether a very big topic which i don't want to cover everything here but i'll just show you how minimum thing what we need to do to integrate our slack channel uh with the virtual agent so uh for uh for feedback thing uh we have created one flow it is called feedback router so if you take a look at this feedback router there are two steps one step is what actually triggers that condition and the other step is uh the posting a message to the slack channel so as you can see in the screen the first condition first block that we have is the trigger condition and the second block that we have is the accents so this trigger condition is the condition based on which we need to trigger this flow so here since all the feedback details are recorded in this table so we have put this table as a trigger condition whenever the record is created uh we insert the record i mean when the record is inserted in this table this flow gets triggered and once the flow gets triggered we uh this is the action for posting a message to the slack channel so uh what you need to post the message to the slack channel is one uh webhook for the slack channel so you can contact your ita admin or you can contact you can configure uh your slack application in a moment i'll show you how to do that from scratch so this is the wavebook url uh this is linked with the selection and here we try to display the message so this is the message that va uses this is the record that was created and first name last name and all the all the fields that we can display it here so this is the this is how we post this message to the slack channel and these things the action two three four and five are doing some of the other logging activities and updating other records which are not required for this slack integration but only this trigger and actions are required the same thing is done for the uh fallback as well so since we do not have much time i'll just try to quickly show how how to do that uh uh in the slab so how to get the wavebook url so i'll i'll create one so there are two condition i'll just try to create that let me display this yeah so if you visit this api.slack.com so there will be your app section there uh in the your app section uh you can create a new application here and then once you create a new application uh you can uh create a wavehook for that so uh let me go back and then create a new app so that you can do it from this class so once you create that you need to click on this incoming wave hook so this is type some sometimes in your company uh your it it organizer might have controlled over uh control so that you will not be able to do it in that case you just need to contact them uh you need to activate this incoming wave hook and uh and you need to create a slack channel layer where you want to get the message from the va so uh channel uh let's say i created this channel for testing purpose i don't want to add any more people right now i'll just copy this slack channel and here i just need to add wavehook to the workspace so this is the channel just i created and i click on allow then it will generate it will generate this wavebook url so i i copy this wavefork url i go back to uh i go back to here so we saw that let's say i want to create a new flow now so my flow is like this so i go to this flow designer i create a new flow all right so uh we recommend uh to create this application in scoped app uh for uh better improvement for better reason and uh here i i want to run this as a system user right so i can click on the submit button so now the flow is created i need to add trigger condition and i need to add the action button so the trigger condition can be uh if the table if we are creating a record in the table or if we are creating if we are updating the table in the recorder there are multiple conditions for simplicity i just i'm just going to i'm going ahead and then create a record in the table so let's say we have uh we have a scenario where when whenever we provide the feedback to the custom feedback in the virtual agent we actually uh create a record in one of the table i'll just show you that table so whenever we create a feedback feedback from the virtual agent a record gets inserted in this table so i'll just put this table in my trigger condition and then i'll just save it and i'll add an accent condition so this is important point here or what we need to do to post a message to the slack channel i need to add action so if you search for a post message as a part of slack channel right and then here you have to provide the wafer url which we had received it here i'll just copy it and i'll paste it here and here uh what is the message that you want to you know like post to the slack channel so if you if you take a look at this comment let's say i just want to uh receive the feedback i just want to receive the message in the slack channel if someone uh post a message like out of poor and below average if someone post message post a feedback with poor and below average rating then i just want to get the message in the slack channel then what i do it here is i i i receive a message here uh who is the user user is so this is how we get the record so see this is the username and i want to get the rating what is the rating that they gave so this is the rating and i just want to return this message this message actually gets posted in the slack channel right so that's that's all so uh and another condition for us was that if the user has provided only bad feedback then when you we we need to get messages in the slack channel right so here we can provide a condition if our rating is poor or if the rating is below average so this is done i'll just save it and i'll activate this once you activate this this will get published all right and here i'll just try to uh provide the feedback sample feedback so that i'll see the response here in my slack channel and we are already out of time i'll just do it quickly all right so i'll just do survey feedback here and to trigger that condition i'm just going to provide some [Music] like dissatisfied rating right so uh what it actually does is it creates a record in one of our table and once that table is created once the record is created in this table here you see in the left side we got that application posting a message that rajiv.json has provided feedback with rating too so this value we have taken not the label so similarly we have a lot of use case so whenever there is insert or update in the table uh we can use that control to post a message in the slack channel so uh every time user post every time the user submits the feedback with the low rating we can get the message in the slack channel and all the members in that slack center can monitor those things and then they can reach out to them for further help and other things so with this time with this much time i i just want to hand it over to howard for uh other slide so is there any questions anyone uh yep reggie we have a question um someone's requesting to see the table as well so i think they want to see the table with the updated feedback that you just submitted so right now number of records are 14. so i just submitted one so number of record becomes 15 here so i'll just uh you know yeah this is already sorted by created so this is the record that i just created it was created with the below average and testing feedback is the comment and then we can also you know pass this value in the slack channel so for now just for demo purpose i just picked up this username and uh and the rating value okay great this is fantastic right okay so i'm going to jump back into sharing real quick and again apologize i know you probably wanted to see more of rajiv's part than mine so i'm sorry about taking all that time but um essentially a couple takeaways we just wanted to make sure that everyone saw was that you know we're careful as to how many um how many things we activate in our slack channel because we don't want to make it so hard that it's hard to monitor at the same time i think question was asked earlier about the process to follow through on those notifications and thirdly we want to make sure that our nlu model isn't overwhelmed with too many utterances just because it might impact its accuracy so we wanted to be careful about that as well so it's just a lot of you know finding the right balance with all those things and then i know we can open up for questions as well but also wanted to uh just end up on this slide to see if everyone can complete the survey in the zoom chat window um you know we certainly needed more time today so again thank you for staying on late late to help us through that so yeah and everyone if you could just please take a moment um to complete the survey these sessions are 100 driven by you so let us know what you want to see let us know where you're at in your virtual agent journey we'd love to meet you there um but we we do need to know what you're interested in so we can make the session sessions relevant for you so we appreciate your feedback and any follow-up questions or any additional questions please feel free to head out to the community post those on the community so that um everyone can benefit from seeing those and seeing the responses and again thank you so much for attending we really appreciate your time your contributions so much appreciated today thank you and a big thanks to howard and rajiv for a great session thanks everyone for your time really appreciate it thank you everyone thank you all this concludes virtual agent academy