This FAQ is intended for informational purposes and is subject to change. For specific questions on Now Assist products and data handling/security for the Now LLM Service, we encourage you to contact your account team.

**Where can I get an overview of the Now LLM Service?**

We posted a [customer-facing AI Academy](https://www.servicenow.com/community/now-assist-blog/ai-academy-ask-the-experts-now-llm-service-july-23rd-2024/ba-p/3000970) session in July 2024, where we covered our Now LLM Service, answering questions on why our Now LLM Service was built, the benefits of using Now LLMs in our products, and how we train our models. We also cover Responsible AI at ServiceNow and how we incorporate it into our product development in each step of the lifecycle, how data sharing is used to improve our models, and our work on the BigCode project for our StarCoder LLM, alongside attendee questions on each of these topics.

**What are the latest updates for the Now LLM Service?**

In the Xanadu release, we introduced a fine-tuned model to replace the Mixtral-8x7B-Instruct model. The fine-tuned model is called the ServiceNow text-to-text LLM. This model is used for conversational use cases such as question-answering and summarization. You can find updates in the [product documentation](https://docs.servicenow.com/bundle/xanadu-intelligent-experiences/page/administer/now-assist-platform/reference/now-llm-model-updates.html).

This table represents a current mapping of skills and models used as of the Xanadu release, keep in mind these skills are subject to change and this list may change in a future release. You can see which LLM a skill is using in the Now Assist admin console within your instance.

| Now Assist Skill | Model |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| Chat Summarization | ServiceNow text-to-text LLM |
| Sidebar discussion summarization | ServiceNow text-to-text LLM |
| Chat reply recommendation | ServiceNow text-to-text LLM |
| Email recommendation | ServiceNow text-to-text LLM |
| KB Generation | ServiceNow text-to-text LLM |
| Incident, Change request, Case, HR Case, Security Incident, Feedback, Multi-feedback, Work Order task, Claim, Government case, Service Problem Case, Test summarization | ServiceNow text-to-text LLM |
| Resolution notes generation | ServiceNow text-to-text LLM |
| Alert analysis | ServiceNow text-to-text LLM |
| Flow generation | ServiceNow text-to-flow LLM |
| Flow recommendations | ServiceNow flow next-best-action LLM |
| Code generation | ServiceNow text-to-code LLM |
| Playbook generation | Azure OpenAI |
| Playbook recommendations | Azure OpenAI |
| Catalog item generation | Azure OpenAI |
| App generation | Azure OpenAI |
| Analytics generation | ServiceNow text-to-text LLM |
| Project Gen AI Docs, Planning Item Gen AI Docs, EAP Teams Gen AI Docs | ServiceNow text-to-text LLM |
| Now Assist Q&A Genius Results | ServiceNow text-to-text LLM |
| Now Assist Multi-Turn Catalog Ordering | ServiceNow text-to-text LLM |
| Now Assist Topics | ServiceNow text-to-text LLM |

**What is a foundational model?**

A foundational model is a large-scale, pre-trained model that serves as the foundation for building more specific models. These models are designed to be highly versatile and adaptable to a wide range of downstream tasks. Some examples of foundational models we’ve used are:

**Why did we use a foundational or pre-trained model during the Washington DC release for our conversational skills?**

As part of our AI Development Lifecycle, we are constantly evaluating our current production models against candidate open-source foundational models based on the use case, performance (quality and latency), and operational cost. The Mixtral-8x7B-Instruct model was determined to meet our criteria for our model decision strategy and was more performant than the current model in production for the Vancouver release (ServiceNow summarization LLM, ServiceNow search Q&A LLM, etc.). Instead of continuing to use lower-performant models in production while fine-tuning the Mixtral-8x7B-Instruct model, we chose to release the foundation model into production for use while our teams were fine-tuning and aligning the Mixtral-8x7B-Instruct model for the Xanadu release. We may repeat this strategy in the future as more performant models are released in order to deliver the best-performing models for ServiceNow use cases with the lowest operating costs to our customers as soon as possible.

**What is fine-tuning, and why are models fine-tuned?**

During the fine-tuning of our models, we use **Instruction fine-tuning**. Instruction fine-tuning is a technique used to enhance the performance of large language models (LLMs) in following natural language instructions and completing specific tasks. This process involves further training a pre-trained LLM on a dataset of instruction-output pairs, where each input consists of a task or instruction, and the corresponding output demonstrates the desired response. This technique helps bridge the gap between the model's fundamental next-word prediction objective and the practical goal of following user instructions. The model learns to better interpret and execute various types of instructions, enhancing its ability to understand and respond to user prompts. Instruction-tuned models generally perform better than zero-shot and few-shot learning models across multiple tasks due to their ability to understand and execute specific tasks based on detailed instructions.

For background on Zero-shot and Few-shot learning:

* **Zero-shot learning (ZSL):** A machine learning technique that enables models to recognize and classify objects, or data points they have never encountered before during training. Instead of relying on labeled data for every possible category, Zero-shot learning uses semantic information and relationships between known and unknown categories.
* **Few-shot learning (FSL):** A machine learning technique where models are trained to make predictions with only a small amount of labeled data.

**What is alignment, and why do models have to undergo this treatment?**

Alignment refers to the process of ensuring that a model's behavior and outputs are consistent with human values, intentions, and ethical considerations. Aligning a model with human preferences confers many benefits. It improves the models’ ability to follow instructions. Alignment reduces the likelihood of the model producing harmful, biased, or unethical content. The alignment process also encourages the model to prioritize accurate information and avoid generating false or misleading content. It's important to note that while alignment techniques have been applied, no model is perfect. Users should still exercise caution and critical thinking when interacting with the model, as it may occasionally produce unexpected or inappropriate responses.

We align our models on two key principles:

1. **Helpfulness** \- Refers to a model’s ability to assist users by providing relevant, accurate, and informative responses that address their needs or questions in the context of ServiceNow workflows.
2. **Harmlessness** \- Refers to a model’s ability to avoid causing harm, distress, or offense to users such as preventing the model from generating toxic, biased, hateful content, etc., and the generation of information about illegal activities, etc.

**Why does ServiceNow run its own Instruction Fine-Tuning and Alignment?**

The Mixtral-8x7B-Instruct base model has a good understanding of the English language and is trained on open-source datasets that give it the general ability to operate as a conversational agent. Mixtral’s Instruct model may not have had a focus on Service Management Generative AI use cases. To develop a Now LLM that provides the best possible performance on Now Assist skills such as summarization, Q&A, and conversational catalog, we’ve fine-tuned and aligned the model with proprietary data. The advantages of doing so are:

* **Improved performance**: ServiceNow fine-tuned models outperform generic models on ServiceNow use cases, leading to better decision-making and more accurate results for our customers.
* **Meet Enterprise AI standards**: Through instruction fine-tuning and alignment, we ensure that our AI models adhere to a high bar in model safety and are Enterprise ready.
* **Customization**: Instruction fine-tuning allows us to tailor AI models to Service management requirements. This customization enhances the model's performance on domain-specific tasks, improving accuracy and efficiency.

**What data do we use for training?**

We list our training data on our [model cards](https://docs.servicenow.com/csh?version=latest&topicname=now-llm-model-updates) for each specific model, and we encourage customers to refer to our model cards for information on our models. As an example, our [ServiceNow text-to-text LLM model](https://downloads.docs.servicenow.com/resource/enus/infocard/text-to-text-llm.pdf) was trained on open-source datasets and ServiceNow platform-specific text data. The platform data is synthetic, derived from datasets provided by customers with examples from IT, CSM, and HRSD domains and is anonymized before use.

**Will we always use fine-tuned models?**

We may release foundational models trained on ServiceNow datasets into production if they are determined to be more performant and meet our criteria for offering the lowest cost of operation to our customers.

**What is model evaluation?**

Evaluation in large language models (LLMs) development is a crucial step in the process to ensure that the model performs well and meets the desired outcomes. At ServiceNow, we have a defined Model Quality Evaluation workflow that sits within our Model Development lifecycle. During model quality evaluation we test for the following:

1. **Model Response Quality** \- Measure the model’s ability to produce correct and expected responses to prompts. Skill-specific rubrics such as accuracy, hallucination, completeness, helpfulness, and response fluency are used to evaluate.
2. **Model Performance** \- Evaluate the model’s responsiveness and throughput across diverse load and stress scenarios to uncover bottlenecks, discover latency regression between versions, and determine product scalability.

Here's a visual representation of our Now LLM Development Cycle at a high level:

**What is the Now LLM Generic provider listed in Now Assist Skill Kit?**

When using the Now Assist Skill Kit, you’ll have the option to use the Now LLM Generic provider, which uses the ServiceNow text-to-text LLM model. Ensure your use case fits what the model is designed to handle when selecting a provider. For example, if trying to create a prompt to generate code using the Now LLM Generic Provider may not be the best provider for this use case as it facilitates assistive text generation use cases. For more information on using the Now LLM Generic provider, see the [model card](https://downloads.docs.servicenow.com/resource/enus/infocard/text-to-text-llm.pdf) for intended use and model details.

**What languages do our models currently support?**

Our current Now LLM Service models use English. See our [model cards](https://docs.servicenow.com/csh?version=latest&topicname=now-llm-model-updates) for more information.

**What is a model card?**

A model card is like a nutritional label that we see on foods. They are technical documents that provide detailed information about a machine learning model’s operational details. We post up-to-date model cards on ServiceNow models on our product documentation. We post our current model cards in this [product documenation](https://docs.servicenow.com/csh?version=latest&topicname=now-llm-model-updates).

**Why don't I see the Azure OpenAI models that are used in some Now Assist for Creator skills listed in your model cards?**

The Azure OpenAI models are not managed by ServiceNow. For information on Azure OpenAI models, see the [model cards](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models) on Microsoft's website. For information on data processing for Azure OpenAI models, contact your account representative.

**Why can’t I change the model on out-of-the-box Now Assist skills?**

Our models are designed to work with the skills that we ship out of the box. We evaluate our models against ServiceNow use cases and prompts, and our product teams design and test our Now Assist skills to produce the best outcomes using Now LLM Service models. We support Now Assist skills using their shipped model and cannot guarantee the quality and outcomes of our Now Assist skills using other models. In the case that a customer needs to use a different model with one of our Now Assist skills, a custom skill can be created using the Now Assist Skill Kit to use the provider of choice.