Walkthrough guide for AIOPS and Metrics: Getting a custom Metric into the ServiceNow MID Server API
This guide provides a comprehensive walkthrough for integrating custom metrics into the Metric database for a new Configuration Item (CI). In our scenario, we aim to start recording temperature measurements from a datacenter rack equipped with instruments and servers essential for our service. The facilities services offer this measurement via a REST API, which we can query. We will create a CI and begin collecting metrics from it.
Metrics can encompass a variety of data points you wish to monitor, such as the number of users logged in or the number of transactions within the last five minutes. This data is valuable for anomaly detection and can be utilized in the alert process to aid in troubleshooting.
This walkthrough specifically addresses the integration of a custom metric for which a standard integration is not available. For standard metric integrations, please refer to ACC-M and/or the Integration Launchpad in the Service Operation Workspace.
Prerequisites
- Red Hat Linux 9 for a MID Server, 2 vcpu, 4 Gb, 256 Gb ssd.
- ServiceNow Install ITOM Health with the following components:
Overview of the steps involved:
- Step 1 Setup MID Server
- Step 2 Enable Metric intelligence on MID
- Step 3 Create a CI
- Step 4 Send in a Metric and setup a CI binding rule
- Step 5 Setup a cron job
- Step 6 View results
- Troubleshooting
Step 1 Setup MID Server
Create a user for the MID server to use.
Setup a user in the Servicenow instance with mid_server role. This role will expand it the roles listed in the screenshot. Make sure this user is not locked or needs to change its password on first log in. If you make some mistakes in the next commands the user may get lock out and you may need to clear that lock if you try again.
Press “New” on the top right corner. Enter username “miduser1”, right click the menu bar and press “save”. Now you can add a role named “mid_server”, this will expand in more granular roles when saved. Then use “Set Password” and set a secure password. Copy and past this password to a secure location, you’ll need it later. Make sure to uncheck the “Password needs reset”. And please be reminded that a user can get “Lock out” when you mis type the password in a later step. Then you can unlock the user here.
Log in into the Linux VM and create a user for the mid server run run as.
sudo useradd servicenowuser
And set a new password:
sudo passwd servicenowuser
Save the download link for the MID Server
Use the link to download the MID server package to the Linux VM (this link will change for newer MID Server version so use the URL as per the previous step):
Install the Mid Server package:
sudo rpm -ivh --nodeps mid-linux-installer.washingtondc-12-20-2023__patch1-02-28-2024_03-09-2024_0815.linux.x86-64.rpm
Then configure the MID server with this one liner, please change the instance URL usernames and password as you previously configured them.
sudo /opt/servicenow/mid/agent/installer.sh -silent -INSTANCE_URL https://INSTANCE.service-now.com/ -USE_PROXY N -MUTUAL_AUTH N -CERTIFICATE_REVOCATION N -MID_USERNAME miduser1 -MID_PASSWORD "PASSWORD" -MID_NAME MetricsMID -APP_NAME MetricsMIDApp -APP_LONG_NAME LinuxMetricMidServer -NON_ROOT_USER servicenowuser
In order for the servicenowuser user to be able to restarts a Linux service polkit needs to be setup. Use the following instructions:
sudo vi /etc/polkit-1/rules.d/10-midserver.rules
And add the following content:
polkit.addRule(function(action, subject) { if (action.id == "org.freedesktop.systemd1.manage-units") { if (subject.user == "servicenowuser") { return polkit.Result.YES; } }
});
Make the file readable by polkit user 'polkitd' by changing it's ownership
sudo chown polkitd /etc/polkit-1/rules.d/10-midserver.rules
Restart polkit
sudo systemctl restart polkit
After a few minutes the Mid Server should be available for validation
For background information on MID server installation, for example if you want to use another flavour then Red Hat. The polkit part may not apply.
Step 2 Enable Metric intelligence on MID
This step will setup the MID Server to be ready to receive metrics begin pushed in via a REST API call.
The previous step had you locate the MID Server in the instance. Use “Setup Metric Intelligence”:
This will start the Metric Intelligence Extension. Wait for it to get started:
Click on the name “MI Metric Extension MIDServer name” to setup the REST listener:
This will also create a API key:
If you click the MID Web Server API Key you can find the key:
Store the key value. You’ll need it later.
Side note: Another method to find the API key later is here:
And look for mid_webserver_api_key
Go to the MID Server page again and locate the started Mid Web Server:
After a while the MID Web Server has started, a port number can be found here (make a note of it):
Connections can be setup securely which if accessed remotely is a must for security reasons. Since we are going to approach the port locally, we’ll not do this for simplicity for now.
The MID Server is now ready to receive REST calls.
The steps above are documented here:
Step 3 Create a CI
We will create a CI on which we register our Metrics. In our example we will be measuring the temperature on a rack in a data center. We will create a rack CI.
Class manager
Click on Open Hierarchy and locate Equipment Rack and click on it:
Click on CI List and New
Note the cmdb table name:
Enter “cmdb_ci_container_rack.list”
And mark this as a favorite so we can easily get back to it later:
Click on Rack1 and we will be adding some related lists so we can see the Metrics easily later:
Add “Metric Time Series Models” and “Metric To CI Mappings” to the Selected list and press save.
So the CI related list view will look like:
Now we have created a CI and setup its views so we can ingest metrics.
Note: If you pick a CI class with no identification rule you must create it in the CI class manager.
Step 4 Send in a Metric and setup a CI binding rule
Since we don’t have a real temperature measuring device, we are going to use the weather temperature in Amsterdam as our proxy source for temperature metrics.
On the Linux MID server create a script metric_single.sh with the follow content:
vi metric_single.sh
Change the apikey as per the one you created when setting up the mid server in a previous step.
!/bin/bashCITY="Amsterdam"# Fetch the weather data in JSON formatRESPONSE=$(curl -s "http://wttr.in/${CITY}?format=j1")# Check if the response contains valid JSON dataif echo "$RESPONSE" | jq -e . > /dev/null 2>&1; then# Extract the current temperature from the JSON responseTEMPERATURE=$(echo "$RESPONSE" | jq -r '.current_condition[0].temp_C')echo "The current temperature in Amsterdam is ${TEMPERATURE}°C."elseecho "Failed to get the temperature data."ficurrent_epoch=$(date +%s%3N)url="http://localhost:8097/api/mid/sa/metrics"apikey="8e7626923bc3ce10507e9daf55e45a12"json_data='[{"metric_type": "rack_temperature", "node": "Rack1", "value": '"${TEMPERATURE}"', "timestamp": '"${current_epoch}"', "ci2metric_id": { "node": "Rack1" } ,"unit": "celcius", "source": "EnvMonitor"}]'
curl -v "${url}" -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: Key ${apikey}" -d "${json_data}"
Note the key components of the JSON payloadMetric_type = rack_temperatureNode = Rack1
Source = EnvMonitor
https://www.servicenow.com/community/itom-blog/walkthrough-guide-for-aiops-and-metrics-getting-a-custom-metric/ba-p/2982298