[Music] all right awesome so we're going to go ahead and get started um I do have a couple of slides to go through um just for some housekeeping so this is our AI Academy today um we do have a great topic for you today and a lot of great experts on with me and very excited about this topic um before we get started let's just go ahead and go through the Safe Harbor notice um you see this on all of ourmes um you know in this session we are going to cover what's in the product today but in the case that we do make some forward-looking statements or we do speak to some road map items um please uh always check with the sa Harbor notice and check with your account team as well before making any purchasing decisions um some resources for you today um so in this session we do have some additional resources alongside uh the live session so we have our generative Ai and intelligence Forum on the community um so we have a quick link that you can use to get to it or if you're in the community and you're looking at the menus you'll see it underneath our product hubs uh and in the product hubs we have a lot of Articles resources blogs and a really active Q&A for you to use so if you do have questions um please visit that product hub for you generative AI questions or now assist questions uh and if you have if you see that you want to answer some um it is an interactive Forum so you can do that there I've also linked our Academy um YouTube playlist so this Academy session will be there um previous Academy sessions that we've had on different topics will be there as well uh and we'll post um this recording to this playlist in just a few days uh and then I've also linked some different articles uh and KBS here and I will go ahead and drop them in the chat for our live attendees um so a lot of the uh Q&A and probably the answers and that we're going to give out today uh will have a lot more detail that you can find uh in these different knowledge articles that are in support um so I will mention you do need a support login to see them um but they are a good resource for you especially for this session uh and further questions that you might have on our models responsible Ai and um data sharing so I'm going go ahead and drop those in um the chat for everyone here so you should have those and then a few more housekeeping items um so theseis are for you um we do try to bring you know fresh ideas contents guidance and understanding of our AI products in the sessions um as mentioned these sessions are recorded and posted publicly to YouTube and we do encourage you especially in this session since it is ask the experts to use the Q&A panel to ask questions uh I do kind of have a note on Q&A especially for this session um just due to the the nature of this topic since we are going into our llms and going into Data sharing and responsible AI um since we are in a broad webinar format we are going to be covering this information at a high level we want to make sure that we cover this information for those who are maybe new to the concepts and new to our now assist products so we're not going to get into technical deep Dives uh on these topics today uh there may be some questions that you have that we can't answer in detail on a webinar format itself so if we can't get to your question or we can't get to it in detail you do have those Avenues as far as the knowledge articles that I've linked or contacting account team too if you need to go a little bit deeper than the answers that we can give out in this session today um so our goals for today um you know it is an asset expert session uh so I'm going to give a quick just kind of brief background of AI here at service now for just a couple minutes and then we're going to get into our expert topics so we have three different topics for you today uh and then we're going to leave some time at the end for some wrap up so um we can go ahead and get started so again if you're just joining the topic today is our now llm service we also will be discussing responsible AI our big code project uh and data sharing here at service now uh I'm joined by um some great experts here today and I'll introduce them as we get to their sections my name is Ash Snider um I'm an outbound product manager here at service now um helping with the llm service uh and different topics that we have so let's go ahead and get into some backgrounds on AI at service now um you know we're going to have a lot of questions and I kind of want to set the context uh kind of of our AI Journey here at service now and the Investments that we've made in our current landscape of our AI products so I know that you know AI um has been synonymous with generative AI in the past two years since the kind of the public launch of chat GPT in that interface uh and I really want to you know just show this slide to kind of note that AI isn't something new to service now I know that we did launch our analysis products and we know that has really been a focus for us lately but we have been investing in Ai and AI products for the past seven years or so uh We've really grown our portfolio with the singular purpose of providing one of the only intelligent uh platforms for in to-end digital transformation so we've made investments in AI organically uh via Acquisitions uh you can see the Acquisitions that we've made uh and the different products and um different features and capabilities that have been brought alongside of them in the past seven years um and you know we've Incorporated this into our platform so our AI Evolution really has been geared to meet our customers needs uh we are investing in AI for the long term um so it isn't just with generative AI it didn't start there and it's definitely not ending there um and you know as we acquire companies and as we develop our own products uh you can rest assured that our AI is delivered you know using the same architecture and the same data model uh and the same security that you get with the service now ploud platform so did kind of want to give you a background on where we've been to kind of give you the context on where we're at and where we're going today um alongside of you know investing in different companies and making uh AI Solutions since 2017 the companies that we've acquired um you know such as element AI have really been Genera contributing to the generative AI space um since 2014 you can kind of see on this slide uh kind of a timeline of where you know service now or companies that service now has acquired um and where we've been really embedded in kind of AI efforts and not just generated AI efforts but this slide's kind of focusing on that um so we've been here for many years um you know again the the talent that we have the teams that we have the researchers that we've had we've been in this space for many years uh even predating the release of gpt3 or what's commonly known uh as chat GPT that really kind of brought generative AI into the for front of public attention uh we've had a lot of service now research L efforts um such as the big code project which we'll talk about in just a little bit uh and we have a lot of Partnerships um and collaborative efforts with entities such as hugging face uh and other um Partnerships out there as well so again um this isn't where we you know we didn't just start in the past two years or so with generative AI we didn't just start with AI we're definitely a leader in open Innovation um with our open models and our open scientific collaboration and Partnerships and then for the kind of the the last slide I want to show you as far as kind of background and and where we're at with our llm so we're going to do speaking to the now llm service um this is kind of our own domain um specific service now large language model that we have um developed in house protecting your data and keeping it within service now uh anchored in best practices and trained on our data and optimized for our customer use cases um so we do provide those llms and the llm service out of the box um but we also provide you the flexibility to bring your own general purpose llm if you have use cases that we currently don't support uh or just need to have the option to do so um So currently we support um general purpose models such as Aur open AI Google AI IBM Watson X and left Alpha for custom use cases so whatever you need to do generative AI wise um our platform will support you and support your use cases um so with that I do want to turn it over to our first expert so I have R Shire Vasan here he is the director of now llms and the now llm Service uh he's going to be our first Speaker today and I do want to kick us off with a question uh and some content that we have before we open it up um the first one being R why did we build now llm service yeah thanks Ashley um happy to be here today and chat with you all about it and I think the slide you just had up talks a little bit about why you want the domain expertise and you want to make models work for the use cases on our platform that are important to our users while maintaining privacy and trust and all of that um so if you can go to the next slide um I think we have right so if you talk about all of the capabilities on our platform that we may want to help with and all of the different specific skills skill being say an outcome in a given context you really want to make sure that you can do some basic level things really really well you want to be able to generate content content is going to be domain specific you want to be able to summarize content and that's a good example of why service now would want to train its own llm um generally llms have infinite memory of everything on the internet and history and they're very creative and you may want that for very specific domain applications but in the service now context on the platform you probably want your llm to be rooting its answer in the context that you have for it you want it to be focused on your case you don't want it to be thinking about every case the system has ever had except for how it's relevant so that's a really good reason for us to tune it and almost reverse the basics of llm in that it's not thinking about everything and trying to be creative it is trying to answer your question with a specific context and so these are all examples of those platform level skills or outcomes you may want to drive and then uh if you go for one more you'll see how those can all apply across our platform across domains so you'll see that summarization may vary in a specific context uh chat and agent summarization is going to be different than say a case summarization and so those are another layer of aspects that you want to consider when training the now llm to make sure that you can address all of these domain specific outcomes and um I think maybe I'll touch upon a little bit of how we get there in the next slide where I have a little bit more detail is that all of this is uh I just talked about really the purpose built piece and the instruction fine-tune piece making sure the data is contextually relevant but in other uh aspects that we want to consider for our customer are balancing performance and cost so we really want to make sure that we're not throwing uh overpowered sledgehammers at small problems right so that balance is really important we want to make sure that from a design perspective that we can provide output so that humans can be in the loop so that we can actually have human preferences relevant to the outcomes and U making sure that optionality still exists making sure it is plug-and playay in a lot of ways uh so that if you choose as a customer to use a different large language model you can and that means that the large language models and the prompts have to work well with the context and the outcomes you're trying to drive so uh I guess with that I I'll hand it back to you actually I know have we have a little bit of content we want to go through before we get to questions and I see a bunch of them in the chat so keep them coming and we'll definitely get to them okay yeah I I did have one if we have time just to kind of answer this one live um since it is pertaining to now llm um so we had question um how is the now LM created uh and I guess the followup onto that is manual steps automation number of people involved I'm not sure how deep we can get into this but maybe if we can kind of high level skim how we how we create our it it's a good question um the basic process of creating the nowm is we've chosen a foundational model uh the evaluation of a foundational model is across many industry standard benchmarks and then we have a whole set of internal benchmarks about how well is it going to work for some of those pieces that we just talked about uh summarization or capabilities we care about uh after we do that we then want to go and fine-tune it there is manual work involved there's automation involved it's both they work hand inand uh a lot of it is data science heavy work to make sure that the model really does improve for the outcomes we care about uh order of magnitude number of people we involved it's hundreds it's not like tens of people it's hundreds of people across the company uh if you think about it and maybe this is an easy transition point to to what Peter is going to talk about we have to both get the data in a trustworthy responsible way make sure we're using it in the right places make sure it's representative of customer outcomes and that's what makes uh I guess the secret sauce of what makes the now llm better than an out-of-the-box foundational model awesome that that is a great segue um I do see we have some more questions coming in so R if you have a moment um I think we have some other experts answering but let's go ahead and move on to our responsible AI section so we do have Peter white here he is our director um of responsible AI here at service now do you want to kick it off with this first question for you Peter why should now when it comes to AI yeah absolutely thank you so much um ultimately everything begins with trust right and if if you go to the next slide actually our our CEO Bill mcder actually says a nicely because trust is the ultimate human currency right um I think we all are based on trust and I think um one thing that is important to not forget while we're talking about a technology it's all about humans it's about humans that are affected it's about humans that are building it and humans actually have value and the middle section here you see like principles and values that we actually really embed in our responsib eye development so for us it's really important to anchor everything that we do everything that rul just mentioned how we build these now Ms anchor them across these four four pillows right it's really important for us to be human centered human centered in the way also that we are um that we're really clear about where we are using AI right that humans are actually in the lube that they know what's going on we at the same time need to be inclusive that we're representing the humans that are using our assistance but also our really really diverse customer set right um so really the representation repes representativeness of data is really really crucial here uh for us as well and at the same time as we're doing all of that as we're developing what Raul just just shared one thing that is really important to us is to be transparent right transparent to you to the customers to the community on what we're doing and how we're doing things um Sean will touch a little bit more on that as we're transparent about like how we've been developing together with the community basically large language models so we'll go into that a little bit more but also our transparency efforts around our products are really at the Forefront because we're releasing model cards that are really um showing a little bit more detail there and at the same time we also have guiding principles that actually U root Us in accountability we really make sure that everything that we do in the company is actually uh has oversight oversight from Executives from externals Etc really make sure that we have basically influence in uh how we're doing things and that we're doing the right thing and that's not only rooted into how we're doing development but also the tools that we provide in the now platform and the governance that we have built in you see that on the right side um touch a little bit on it domain specific models as one specific actor that will help us to govern better the outputs um at the same time really is important that we actually have good data that goes into it um so R touched on that as well a little bit um and and gave a segue here one important thing for us is that we're rooting our models in the best possible data and that could be data that comes from uh third parties open source um uh synthetically created data even and most importantly that actually we have uh customer data sharing programs where customers are offering uh filtered uh data to us uh that we can also purpose for for evaluation and uh testing and and model training but also we have in the platform itself built guard rails and controls that actually help customers um govern generi in a much more um succinct way uh with our gen controller with our now assist admin panel that really allows you to have one choke point basically around how someone and where uh gener eyes used in the platform and in the service now ecosystem so together all of these things build a really good trust layer basically and a circle of trust around everything that we do and everything that we do is rooted basically in in trust and how we just go into the next slide I just want to touch on that basically just high level really quick um practicing responsible AI is not just a oneoff many companies out there they just actually like have a checkpoint somewhere in this software development life cycle and say yep okay we have this checkpoint and we actually are doing respons AI we actually have a peripher of um a lot of um elements that we actually invoking in the software development life cycle all the way from ideation to monitoring and we really make sure that high-risk assessments are conducted that we actually make sure that our security and architecture reviews are done that data assessments are done that everything is uh oversight is provided to the stewards that we train the data uh the models responsibly in uh secure environments that we actually are hosting that we have the quality controls and the test data that is used the teams have test plans that we try as we deploy the product try to be as transparent as possible uh through the model cards and then also at the end ultimately overse provide oversight and oversee the deployment right we want to make sure whatever is deployed within your uh instances and within your production environments actually has the right level of oversight and we actually ensure uh that nothing goes out of bounds that no risk basically is created so end to end we're really ensuring that um the software development life cycle is rooted in responsible principles and that we're not just making it a check checkbox event so with that actually handing it over to to Sean in the next next chapter yeah before we get to Sean I did um we did have some customer or some attendee questions here that really kind of coincide with this topic so I I'm glad that we put it in here um and I wanted to get um you Ro and you Peter to just kind of quickly um discuss you know and I know we kind of touch Bas on this a little bit but why why should customers participate in data sharing um you know and kind of you know how how is that data sharing leverage yeah um one one important one important piece well you want to go first from a development perspective and then I think I can also um highlight a little bit more how yeah sure sure I was just going to say thank you for the opportunity for I guess a small sales pitch here for something that Peter and I have been passionately working on for four years uh yeah I think fundamentally AI is only as good as the data that it has access to to be trained on to be built on to be tested against so that's I think for me the pitch boils down to if you have trust in the processes and the team building the AI for you which I think is where Peter will have a lot to say uh and that that processes are done correctly and right stringently then you are going to end up with a better functional AI for your domain needs and that's the reason why to participate there's more shess that the model can work for your use case for your domain for your specific outcomes which is really hard to assure if you don't have the context yeah that's right exactly and you see here a little bit highlighted we've been spending um quite some time over the last actually I will said four years we've been on this trying to build u customer data programs that customers can trust ultimately uh because for us it's important to validate uh our models on customer data and ideally also then train on it and here you see a little bit this this virtual Circle right we start usually with service now data and then as the now LMS and as the products in the past even before LMS get deployed we see basically how the data is um how the data is shaping how customers are using the product and for those customers that have opted in uh to data sharing um basically and it's an opt optin uh to to do that um they can actually um contribute that data then is filtered and then basically fed back into the loop um so that we can actually um use that data for further improving basically um the model in our development life cycle that I just showed you and one important piece here is that it's it's voluntary uh but it helps a great deal to really make the model better and in particular we've spent I said the last four years building out um high security uh development environments with a lot of the security guard rails and controls in place to actually have customers trust us uh with their data because it's ultimately the most important and valuable thing that that many companies actually own in the digital space it's their data and so therefore it's really important to us to actually um act on that responsibly and correctly and so we've been doing that and building that out over the last years and I want to thank all the customers that have contributed to it all think think all those customers that that maybe like consider like uh uh contributing to it because you actually your return on investment will be much greater uh if uh we have the the chance to actually use your data for for further improving the products that that you love and that you use today well thank you both I think this a really great segue I did see some questions in there that um are really going to be uh we're going to give some insight to in this next section so I want to bring on Sean Hughes uh he's an AI ecosystem director here at service now on our research team uh and Sean um this section I know this is a close one to you you've been working on this for years now um really want to get into uh how was star coder built yeah thank you Ashley so good morning good afternoon everyone apologies for the croaky voice I'm just coming off the back of some uh uh cold so you've heard a you know great overview of you know why now llm how do we build it responsibly and in this section we're going to take you through a very quick high level uh case study of the star coder model that we built in conjunction in partnership with the big code community and hugging face and when we talk about building models from scratch we're really talking about base Foundation models that have not had any prior fine tuning instruction tuning or anything like that and we do this to make sure that that we have the full understanding of and control over the data that goes into training the base model um most open source models that you'll find on hugging face uh hub for example are typically fine-tuned or instruction tuned based off someone else's Foundation model and you know you sort of struggle to understand the where all the data has come from the techniques used and things like that so you know with the investments into um acquiring element Ai and other companies bringing real expertise into the company that can build models from scratch this has enabled us to advance how we approach llms and implementing Into the Now platform so at a high level we took the star coder models that we built in the community we then chose the best models that came out of the project brought them into service now where we then created three fune models based off service now's platform workflow data um as Peter had mentioned in some cases customers had uh opted in to contribute some data that was fully anonymized and uh stripped of anything sensitive to them Etc and with that we were able to create three now llms one that is really good at summarization in the context of customer workflows as well as being able to do code generation and workflow generation so the next slide please we will quickly cover at a very high level you know what it really takes to build a model now if you look at this slide there's a an ey chart in font size six so pretty much nobody can read it but everything in the white boxes over there is taken from fully public information our model cards our search papers um the tools we used and that we open sourced and released to the public through the the big code project GitHub um it's all public information there if you saw on the previous slides we actually had a a QR code and a link to the full case lady so I'm not going to talk through all those boxes but just know we do provide provide full transparency into these models that we built and that we deploy to power the different skills in the platform next slide please so there was a question uh by Hank Yan Hof which is how do we differentiate with competitors all can extract this and get the data right well as you can see from this study by Stanford University they recently published the second edition of the foundation model transparency index and when you look at all the major Foundation model providers that you know you probably all know and have heard about um what you can see each model is represented in its own column you're looking for blue the more blue you have the more transparently developed the model is the more you will be able to understand how the model was built where did the data come from what labor was employed into the model how much compute was used what was the environmental impact from a carbon emissions perspective all of that is what you're looking for when you're trying to select a model so you know service now aims to to be as transparent as we can be um in the star Cod of case you can see we are 100% transparent in every single box except for two and for mitigations and impact when you're doing a building a foundation model you really are at the very start of the model development life cycle you aren't at the point yet where you're doing fine tuning for for key use cases and you aren't at the Point clearly when you're impact import sorry implementing those models uh to be available to end users so when it comes to mitigations and and downstream impact you clearly can't do that at the very start of the life cycle um measures where you drisk the model and you implement mitigations and you manage the impact that will have on users both positive and negative that's what we do later on in the uh implementation phas uh in the platform engineering teams but again as customers are looking to bring their own model um even with big fancy named companies and and really popular open source models just know not all open sources equal most companies will publish an open weight model but they won't provide you the details on the data used to train it they won't provide you details on the lab that went into the foundation model they won't provide you the details on the compute used so you really don't know the full impact of the model next slide please you know and so once you have implemented the model you really want to make sure that the skills that you are powering you've chosen the best model performance wide performance wise balancing the uh the inference latency as well as the cost of operation and you know the cost the end cost to the customer with the star coder models that were fine-tuned for code generation you know service now's own use of the models showed that and we reported this last year that we ourselves saw a 52% increase in developer productivity and speed of innovation and that little animation you see there is actually a a screen recording of the code generation in action you know right within the workflow uh next slide please so to wrap it up uh we had a pleasant surprise within the last two weeks uh we got a notification from a uh well-known uh Tech publication called venturebeat uh they contacted us and said just to let you know you have service now has been selected as a finalist in the Venture beat AI Innovation Awards and the next day we actually were announced as the winner of this and this was in recognition for the best Enterprise implementation of generative AI in software and that was BAS B off our star coder project that led to the now llms which you now see as uh some of the skills like code generation and workflow generation so it's it's great to see the full life cycle come to fruition in under two years thank you thank you Sean um thank you everyone I know we went a few uh minutes over today I appreciate all my experts for being on today and everyone for attending um be on the lookout for future uh AI Academy sessions they are on the community they're posted in our generative Ai and intelligence Hub uh if we didn't get to your question today I know we have some unanswered ones uh what we'll do is take these offline uh and post them to the event that's associated with the academy so you can find them in the the product Hub um just a few days after this session and we'll post the recording to YouTube um but thank you for everyone joining today and thank you again experts have a great rest of your day thank you thank you everyone [Music]