good morning everyone welcome to virtual agent academy we are let's get started now a lot of participants here today thank you all for joining us whether it's your morning or your evening before we start just wanted to give a shout out to our um community website a lot of resources here for anyone from beginners to experts revolving around virtual agent or nlu learn about best practices update content documentation uh faqs and really rely on the experience of others and feel free to get in there to ask questions provide answers and and see all the latest information that we have on virtual agent and nlu so today welcome to uh virtual legend academy today's topic is nlu quality conversations in quebec so our quebec release is upcoming and we made a lot of enhancements to nlu that's going to really provide much better conversation experiences for your users and uh presenting today is uh we have david who is on the nlu side or for now intelligence we have paul from itsm marcel from employer workflows and myself from the conversational interfaces team so uh our goals for today uh we we're going to have david go over the nlu enhancements in quebec he's also going to show you uh in an instance uh kind of walk through the improvements that have been made that in turn that improves the authoring experience for nlu after that we're going to do exercises we're going to actually you know provide tangible use cases for it and for hr and paul and marcel are going to show you that and then at the end we will wrap up with q a during the time feel free to ask questions in the q and a panel and uh you know i and the other uh attend uh panelists will be fielding those questions uh at the time and if there are any uh pausing between the video or in the webinar cool with that said i'm gonna hand over control to david and he's going to uh kick us off go ahead david yeah thanks victor good morning everyone uh good afternoon or good evening the changes that were happening in quebec um but not only in quebec so in january past january so last month the back-end engine was updated for basically preparation of the general availability of the quebec release and because there was also an early availability of the quebec release for certain customers the backend engine was updated with this update um which is obviously intended to provide a much better and improved let's say prediction accuracy and greater support for let's say languages and entities it was also intended ultimately to become available for at least several parts for earlier customers on the paris release however um due to a uh slight mistake on our end um we also made that update available to every new model that was created in paris after uh january uh after we basically pushed that january update in the back end so any existing uh nlu model is not affected at all uh it will only appear once you've created a new model and um the biggest bonus is and obviously in the in the in the our goal is to make sure that everybody starts upgrading to this new back-end version the goal is that that engine that nyu engine basically has a lot more resilience for gibberish as you can see on the right-hand side in the picture where sometimes we used to provide a let's say a match to an intent when you typed in a gibberish word that should no longer happen or very few times at least another a great improvement is the fact that the number of direct matches has increased so that means that you get a lot less of those let's say a in-menu predictions um and also incorrect matches and the way that that that was done was basically twofold on the one hand we basically increased um the range in which the uh the prediction scores take place where we used to take um between point four five and one uh the scores we now look at the whole range between zero and one um and that basically spread out basically the the scoring possibilities uh of of the engine of the of the prediction and uh therefore causes um let's say fewer in menu as i said fewer menu predictions and a bigger chance of really of the direct match uh so that also means that you get to see when you when your virtual agent and your users sometimes would get options let's say three four or five options that they had to choose from uh the the right topic that that is greatly reduced to uh fewer times the good thing is that even though we inadvertently basically pushed it for all the the newer models that were created on paris you can still go back and make sure that you're basically running against that previous paris release back-end version through a set of properties in the system properties and the way to do that is just by following that link that points to a knowledge base article on servicenow support and in it it will guide you through all the steps that you need to do if you want to go back uh to that previous paris back end um it still exists obviously and it's up to you basically to test it out um and to see uh the improvements taking place and there's a couple of things that you need to bear beware of uh as you move from um let's say the paris release to the quebec back-end release uh but we we will provide some pointers throughout this session for that all right um biggest changes in quebec are obviously the extended support for 13 more languages so we had already support for three languages and now we're adding for 13 more languages and we're adding full support for english french german spanish and japanese so that means both intent matching as well as entity matching and we will also on the right hand side if you are a itsm hr or csm pro customer you will get features that will help you in model building through conflict review through intent discovery or topic recommendations as it's called and also in batch testing i will touch upon that a little later as well um we will also have out of the box models for uh predominantly the let's say the the the five languages that offer full support as you can see in the center so igsm and hr are uh very well equipped with out of the box topics and out-of-the-box nlu models uh and the other languages uh they rely on intent recognition so uh that that will allow you to basically uh provide a greater set of users in your organization to be able to converse at least the initial utterance to the virtual agent in their native language okay so what are those uh virtual agent and nlu authoring experiences um so the biggest change is the fact that we're introducing the nlu workbench and the nlu workbench is no longer a studio app or a studio part of the studio it's an actual application inside the servicenow platform and you can navigate to it through the [Music] application navigator um another big change is the fact that we have keywords that are now usable as a fallback for topic discovery so previously when you activate it activated nlu for your virtual agent you could no longer rely on keywords now we're allowing that to coexist and both paul and marcel will demonstrate uh let's say the impact of this feature so i will leave it to that to further explain how that works there are some new string entities introduced person number and currency so that means that when you make predictions sometimes the the system thinks hey this might be a person's a person or this might be a number or currency you can then use those values as variables in your virtual agent topics to further improve the flow and the user experience for your users vocabulary sources that's a big improvement because right now it means that from quebec onwards you have the ability to basically point to a table in the servicenow platform say this is a great source of entity values so not only choice list but basically all of the other tables so like cmdb related tables or business services or applications or hardware and you can point to them and you can basically use them as a supply for both your vocabulary as your as well as your entities so um with that change uh the list entity is now replaced by what we call the mapped entity the mapped entity basically maps to those vocabulary sources and fuzzy matching is now used to improve scoring uh when typos are made so sometimes when you have a typo which i frequently do [Music] that used to basically result in a sometimes a very significant let's say scoring um hit and meaning that it would not recognize one or more words for my sentence and therefore didn't find the right intent and i had to either pick from a menu or uh it would i would have to basically pick or change and re-enter my utterance uh with fuzzy matching that is greatly improved and sometimes or it now finds the right intent if i made slight typos the prediction of precision improvements that's one that i just touched upon with that back end update already so uh having the direct matches improved and the fewer in-menu results returned right so that that's just a big improvement and one thing that everybody needs to note and make note of is that overall because we basically exchange increase the range from 0.45 to 1 to 0 to 1 it means that there's a the spread is more evenly between that zero and one percentile and uh has an impact on your confidence threshold so we used to give guidance in in the confidence threshold let's say uh above 80 uh you should try and aim for and tune your nlu model uh what we've noticed with quebec is that we tend to tone it down a little bit to let's say a lower percentage threshold uh because a the direct matches are are happening more frequently there is a more uh and and the results are the returned accuracy accuracy scores are more across uh that whole range so that it's uh now becoming more uh important to uh yeah use the threshold and drive it back to where we usually or where we used to start which was 0.60 or sixty percent if you start a new model it's sixty percent so i'm not saying that you immediately have to go from eighty to sixty percent uh but as you do your tests with that new back end uh quebec or if you're using it in paris with paris in place you might need to take a look at your threshold numbers and i briefly mentioned those advanced workbench utilities so that's intent discovery which comes with topic recommendations on the virtual agent side and batch testing conflict review those utilities will will significantly basically help those those pro customers who are um yeah using virtual agent nlu extensively and want to make sure that as they move forward they get good recommendation as to let's say new topics that they should be looking at including in in their uh environment uh to find potential overlap issues with between various intents and that's obviously very helpful as your models grow and um also helps you basically sort of a fail-safe is you're using the batch testing capabilities to test your model before and after you make changes we will dedicate a specific uh a specified va academy on that on those advanced workbench um right uh paul um i think i'm gonna ask you to step in since uh uh time is running short and i think you're actually gonna cover the mmu workbench uh in in in real life so uh why don't you show uh how it looks and where to go and flip back and forth yup will do thanks david um so the specific topic on david's last slide that i want to just show a little bit about is the keywords as fullback for topic discovery let's assume i'm part of an organization that is in the process of implementing virtual agent i have a couple of topics published already i have niu enabled and i see there's a big demand for our employees around printer issues so one of the frequently seen utterances is for example paper jam if i type in paper jam now i see there's no intense discovered by niu and frankly there's nothing that we can discover because paper jam is not something we have published anything for yet right so what i'll do is i'll go into my virtual agent designer i'll look up anything regarding printer and i can see there is an out of the box template for printer issues so what i'll do is i'll open up the template or the topic um let me open this up a little bit further so we can actually see what this does i'm going to go through this in detail but just so you understand what it what it is doing first it asks what issue you have can't find the printer out of paper out of ink so it's a few options there and then the flow continues with anything else etc so that is um that's probably what i'm looking for so now as as best practice i'm going to duplicate this topic and i'm going to name this printer issues and save that so now i'm in the copied version of the printer issues i'm in the properties if i scroll down a little bit what you'll see is that there's both an nau model associated with this topic and an intent and there's keywords right so in previous version pre quebec you would see either one or the other in this case we see both for now i'm going to get rid of this out of the box nau model which automatically takes out the intent as well i'm going to publish this topic publish anyway um what i want to do now is go back to my overview of all my topics and test my active topics on the term of paper jam now if i type in paper jam i can see that there's no intent that was found and there's nothing in the results here as well so let's go back to the nlu model let me open up nlu and go to models i see that i have a nlu itsm model and if i open up that itsm uh model then i can see that i have two intents that i've already that are in in use instead of creating anything from scratch i'm going to import an intent i'm going to import the intent having to do with printer issues there it is so i'm importing that intent to my model open up the printer issues intent you can see these are some of the utterances all right so this is i can add my own utterances here but for now let's just go with what we have here train the model so i need to train the model and then publish the model let's give that a bit of time and now if i publish that model so now that model is trained and published i'll go back to my topic again director designer type in printer issues there we are now under properties i still have my keywords here and i can select my nlu itsm model that i just trained and published and i select the intent printer issues i want to save that and i want to publish that and let's go back to my designer to test all my active topics and look if this has the uh intended result type in paper jam it didn't match any intents but it did match the keywords so here's what's happening the uh nlu model that i created it didn't probably find anything related to paper jam so it didn't match any intense in my lu model so it wasn't able to match that to my topic but what it did do is it did fall back to the keywords right so now let's go back to my nlu model and find out why it didn't find anything around paper jam opening up my models and by the way here you can see a long list of models here you see all my languages so i can easily filter that to just show the english ones right so it's a shorter list list uh now especially with all the languages that we're supporting here's my only uitsm model go and open the model and test the model and say what does paper jam give us here we go it's 78 now you heard david mentioned that we've improved the accuracy of the nlu prediction so that we can we can safely lower the um at the threshold uh to a value close to maybe even sixty percent but in this case we don't even need sixty percent so if i would go to settings and lower this to 75 and hit save oh sorry hit save i then need to train and publish that new model again we'll give that a bit of time and publish that model now let's go back to my designer and test my active topics go in here type in paper jam and as you can see it now did find the intent printer issues it did also discover some entities but for some reason it linked that to my entity no paper um right so this is good for now what i showed you is that we have a mix of entity and nlu so if nau does not provide any intents that we can rely on it falls back to keywords for topic discovery from this point onward in my use case here i can now start to extend the topic to also include an area on paper jams and maybe probably even extend my entity value to also include a paper jam so that will match to that step in the process we'll leave that for some other va academy session uh for now um this is what i wanted to show so i think i'll hand over to marcel marcel thanks so much paul i'm gonna grab the screen from you and we can go into the hr portion of today's build exercise so just to tie this together right and and bringing this full circle into including different departments aside from it if we come in and we search on let's pull up the paid discrepancy template so what i wanted to really um convey to you guys today is you know different departments may move at different speeds depending on who's ready to uptake virtual agent who's ready to uptake in on you and so this feature really gives you flexibility to apply nlu to specific topics within a department specific topics you know by topic but really it gives the flexibility to come in and say i want to use nlu maybe for it but maybe for hr we're just testing it out with keyword and we're going to create a new duplicate topic from the out of box topic to demonstrate this and so once that saves you can see now it's just defaulted the the properties with the um out of box hr nlu model and pay discrepancies so what i want to do is just remove those right now and i'm going to publish this topic and make it active and just show you guys how this would work with keyword only so this is in the same instance we're assuming it is leveraging nlu but you can see under the keyword category i have some different keywords here so when we go to test some of these topics let's just test this against all topics and we're going to include the topic discovery [Music] and i'm going to type in something like direct deposit and you can see here that it didn't match any intents here so again this is where you know i want to make sure it's it's keyword um we're matching up with the keywords and i want to make sure that it's it's recording appropriately now if i were to change it and select an hr model here and let's pull in the pay discrepancy right so we're gonna save it and then we're gonna publish this so now we're just assuming that you know everything's using nlu so we're using nlu across hr and it what we can do is now test this against all topics so let's go back to designer we can test all topics and i'm going to try something different we're going to try paycheck assistance and you can see i have a 92 match right and this drops me directly into the topic um and so just to validate that against some of the reporting so if we go into the nlu the table is open now you predict blog.list and so this gives you the table and i'm just going to filter by audra which is the user um you can see here that depending on the different utterance right so for direct deposits you know i'm returning no intents and entities but if i look at something like paycheck assistance you can see that i'm using the nlu capability here so it'll show an intent match and i can go actually go into that record and understand nlu was applied and so you'll see the results in the in the log if it's zero it's an indication that um you know you want that to be zero if you're using keyword for that match and so that really you know kind of brings home the experience of of using nlu for some topics um you know it's a good idea like if we come back to virtual agent designer right so you may want to have it topics using lu um hr csm topics using keyword that's fine um but if we just quickly take a look at the properties again just to recap and so we can see here you know i can remove these um if you have keywords it'll look for the nlu first it won't necessarily look to the keyword the keywords more of a you know a fallback um but if i remove this it will just look at the keywords and it's a i believe there was a question in the q a it's a little different than um you know that the traditional zing search engine that you would see in the contextual search where the keyword is just looking for keyword matches to um identify which topic the user is asking about as opposed to nlu will will apply the natural language understanding capabilities but again just wanted to reiterate you guys can use either keyword or nlu so um again just give some flexibility on deployment options so with that i think we can go to questions so victor did we want to read some questions aloud and yeah let's do that yeah yes thank you marcel uh so yeah so one question uh we have uh is there a cap to the number of keywords i can enter for the topic yes so that's a great question and um let us follow up on that with the limit to the number of keywords i mean typically you know we'll we'll see upwards of i think you know 15 probably at the most again because the idea is not to cover every single use case or permutation or iteration of a keyword it's just to kind of include the main ones but um let us follow up on the community with that and get a response great thanks and then the other question we had in q a is um for the itsm paper jam example uh can you show the utterances that were created that helped the intent match uh what do we do when no intent is matched even after reducing the threshold yeah and i think um that's more of an nlu tuning exercise right that's a way to um you know we always recommend get a list of test utterances and you want to run through those test utterances and if you're not seeing the results you expect to meaning you're testing an utterance and it's not either matching the intent or it's matching an intent that's incorrect that's an opportunity to maybe add that utterance into the model and retrain it so again that just kind of goes hand in hand with some of the tuning activities for nlu great thanks and then the next question uh with the new nlu enhancements should we adjust the confidence delta property now david i think there's something that you could probably take yeah yeah yes i mean that's a good question thanks and thanks for being such a active uh participant also on the community um yeah so indeed uh actually it's not uh it's because of the the fallback uh capability to keywords that you need to rethink the confidence delta and that is explained in the documentation but i'm in the process of writing up or actually rewriting the article on the delta by adding a piece what happens to um to to when you go to quebec because indeed um that has an impact on that confidence delta so give me a little bit of time i'm expecting that that that uh blog or article to reach let's say published date by end of next week um so stay tuned and and you will learn everything that you need to know to confidently uh make the right decisions there great thanks uh next question we've got oh these things keep shifting on me what is limitation of utterances i heard his 200 characters now it's 200 utterances so there are two other utterances uh max by default uh on an intent so there the there are now a set of properties that are also part of the niu workbench and there you can review all the all of the properties one of them is the number of utterances per intent and the reason why we say that is because to be honest if you have a hard time um basically tuning your intent to match between let's say uh 30 to 60 utterances max and actually it would be even better to do it with with between 15 and 30 then there is maybe something wrong with the uh intent itself it might be too broad for example great thanks um let me see that's that uh we do have a question on what is the batch testing i belie this may be a longer answer than is um suitable for for now i'm gonna if we have to we will we will give an update right with uh with an upcoming academy session and we will i will also make sure that there is a a blog article on the community soon uh that explains all those utilities that will come with the advanced workbench great thanks yeah absolutely we will have definitely another future academy on the against workbench utilities um and then next question from from eric yeah it doesn't seem like i can form a question out of that let me go to yeah i'm not sure if there's any other questions um marcel david looking through the q a uh we're gonna think you want to address from the questions or comments you see yeah the keywords is uh um uh similar for for keywords right you mean uh you could put multiple keywords in in in various languages into the virtual agent um but there is a max for nlu there is no max but you need to have multiple models to basically serve each and every language so i would say if you have if you are supporting multiple languages miu is the way to go regardless so you might not want to rely on fallback to keywords there great um i think that's all i think we'll pause there for questions i just wanted to start to wrap things up a bit um we do this session will be uh has been recorded and will be published to virtual to the community site in a couple of days feel free to again post questions in community and you'll you'll rely on the collective knowledge not only just the three of us or with with paul but also again the broader virtual agent community and uh again thank you everyone for for attending this has been super helpful thank you marcel david and paul for for presenting everything and we'll uh we'll stop here thank you