Within the event management module of ServiceNow there has long been the ability to ingest CloudWatch alarms from AWS. In this post I'm going to explain how to achieve this same outcome, but using a MID server to route the data. Why is this needed? Some customer instances of ServiceNow are configured to only allow access from specific IP ranges so that external users are not able to access their system. In such configurations, accepting events directly to the instance from a public cloud provider like AWS is not possible. Special thanks to my colleague Vinh Tran for assistance with some of these items.

The components involved in this solution are 1) an AWS SNS topic and subscription, 2) an AWS Lambda application, 3) a ServiceNow Listener Transform Script, and 4) a ServiceNow event rule. Obviously a MID server is also needed here, but I will not be detailing the setup here.

AWS Configuration

Let's start with the AWS components. You will first need to create a Lambda application. You can wait on specifying triggers until the next step.

AWS Lambda Application

The environment variables are important as they are used in the Python code of the application. The password is anticipated to be base64 encoded. The instance URL needs to be in the form of http://{IP address of your MID server}:{port}/api/mid/em/inbound_event?Transform={name of your transform script}. We will see the {port} configuration later in this post. The "transformHeader" is the same as the transform script you specify in the URL.

Here is the code of the Lambda app:

import json
import logging
import os
import urllib3
import base64

logger = logging.getLogger()
logger.setLevel(logging.INFO)

INSTANCE = os.environ['instance']
HEADER = os.environ['transformHeader']
SNUSER = os.environ['user']
SNPWD = os.environ['pwd']

MYPWD = base64.b64decode(SNPWD).decode('utf-8')

def lambda_handler(event, context):
    logger.info("Event: " + str(event))
    #message = event
    message = json.loads(event['Records'][0]['Sns']['Message'])
    logger.info("Message: " + str(message))

    # Create the http session
    http = urllib3.PoolManager()

    # Set the ServiceNow instance or MID server endpoint to receive events
    url = INSTANCE

    # Eg. User name="admin"
    user = SNUSER

    # base64 password
    pwd = MYPWD

    auth = user + ":" + pwd

    # Set proper headers
    headers = {"Content-Type":"application/json","Accept":"application/json","Transform":HEADER}
    headers.update(urllib3.util.make_headers(basic_auth = auth))

    # Encode the payload
    encoded_data = json.dumps(message).encode('utf-8')

    # Do the HTTP request
    r = http.request('POST', url, body=encoded_data, headers=headers)

    # Log the output of the request
    logger.info("Data: " + str(r.data))
    logger.info("Status: " + str(r.status))

The Lambda app is expecting to receive CloudWatch alarms from a SNS topic, so you make the app be a subscriber to a SNS topic where you have configured alarms to be sent.

AWS SNS Topic Subscription

That is about it on the AWS configuration. Setting up the subscription will add a trigger to the Lambda app.

MID Server Additional Configuration

Let's review what needs to be done to ensure the MID server can accept the alarms. First, you need to ensure you have defined a MID Web Server Context (MID Server -> Extensions -> MID Web Server).

MID Web Server Context

Be sure to specify a port that is not already in use on the MID server. This is the port you specify as part of the URL provided to the Lambda app above. The user and password specified on this record are also the ones provided to the Lambda app so it can authenticate to the MID server web listener.

The final MID server setup step is to create a MID WebService Event Listener Context as shown below, and associate it with the MID web server context you just created above (Event Management -> Event Listener (Push) -> MID WebService Event Listener).

MID WebService Event Listener Context

It is also important to verify the MID server OS (presumably Windows) is allowing inbound traffic on the port you specified for the MID web server above. In lieu of turning of the entire firewall, you might test disabling the policies while doing initial configuration or if traffic doesn't seem to reach the endpoint you're specifying (run the following command to disable "netsh advfirewall set allprofiles state off").

MID Server Transform Script

One of the final steps is to now create a transform script that will be used by the MID web server to process the incoming CloudWatch alarms.

Listener Transform Script

While the code in the script is not terribly long, it would be quite a bit to consume in the length of this post. As such, I have posted an update set that will create the MID script include and this listener script record. The "Header value" on this record is the same as used in the Lambda app.

The final step is to create an event rule or rules to process the alarms. While there are some out of the box rules for AWS CloudWatch, I created an update set that will process specifically for EC2 instance alarms. This can server as a good starting point should you need to create others based on how this process is bringing in data to the event table. The update set is in the same repo as the MID transform script update set.

This same process could be replicated for other cloud providers where it's not feasible to obtain all of the possible IP address from all the data centers they have around the world. Should you need to scale this setup due to a large volume of CloudWatch alarms, consider putting multiple MID servers behind a load balancer (easy in the public cloud environments) and have the Lambda app send data to the load balancer.