Sending alerts when script processing fails

This article describes how to configure the Integration Agent to send an alert to the xMatters administrator if an integration encounters a script processing error.

Overview

Although the Integration Agent is equipped with fault-tolerant features that will handle failed connections to xMatters, other conditions - such as script execution failures - can prevent notification requests from reaching xMatters.

For example, if the management system sends a malformed XML message to the Integration Agent, the Integration Agent will not be able to parse the XML and the notification will fail. Similarly, if the Integration Agent sends a REST POST to xMatters with incomplete datea, the post will fail with a DATA_VALIDATION_ERROR, and there will be no notification.

Generally, a failure that occurs during script execution will cause an exception to occur. An exception is an anomalous condition encountered during execution of a program that causes the program to be interrupted. When an exception occurs during execution of an integration script, a message is written to the Integration Agent log; for example:

java.lang.Exception: There are data validation errors in the form.

These messages are usually followed by a stack trace: a list of software modules that were active when the exception occurred. The stack trace includes the name of the integration script and the line number of the code that failed to execute.

Unless the exception is caught, or accommodated by the script, the Integration Agent will stop processing the notification request and the incident data will be discarded.

Exception handling

Once you have identified the failing line of code (as described below), you can wrap it within instructions that define the action to take if an exception occurs. This type of software construct is referred to as an exception handler.

An exception handler can alert an administrator whenever an exception prevents an xMatters notification from being sent. For instance, if the Integration Agent receives malformed data from the management system, an administrator may want to be informed so that they can take remedial action.

Alerting mechanisms

There are several ways to send an alert to an administrator when an exception occurs. 

The first option is for the failing integration script to construct a special alert and send it to the administrator. The issue with this approach, though it would seem to be the easiest at first glance, is that many integrations are complex, and adding new alerts to them can be complicated and time-consuming.

Another option is for the failing integration script to invoke a different integration service that is specifically configured to send this type of alert. The Integration Agent includes a "Generic" integration with built-in fields such as "person_or_group_id", "subject", and "message" that make it convenient and well-suited for sending a simple alert to a specific user or group.

Enabling the Generic integration

The Generic integration service is automatically included with all Integration Agents and available from all xMatters instances, and requires only a couple of simple steps to enable it.

To enable the Generic integration:

  1. On the Integration Agent server, open the <IAHOME>\conf\IAConfig.xml file in a text editor.
  2. Locate the <service-config> section, and add the following line to the list of paths:
    <path>generic/generic.xml</path>
  3. Save and close the file.
  4. Log in to your xMatters instance, navigate to the Developer tab, and then click Event Domains.
  5. On the Event Domains page, click generic.
  6. On the Event Domain details page, scroll down to the Integration Services section, and click the Add New link.
  7. Type generic in the Name field, and then click Save.
    • You can leave the Path field empty.

Invoking the Generic integration

You can invoke the Generic integration with a single line of code, but it is recommended that you make a function that accepts information about the error (such as the exception message) to populate the required fields and then send the alert. This allows the alerting code to be reused if you encounter exceptions in more than one part of the failing integration.

To make a function:

  1. Open the failing integration's main JavaScript file in a text editor and locate the importClass section near the top of the file.
  2. If it does not already exist, add the following line to the importClass section:
importClass(Packages.com.alarmpoint.integrationagent.apxml.APXMLMessageImpl);
  1. Locate an existing function definition within the same file and insert the following code before it: 
function sendErrorNotification(errorString)
{
	/*
		sendErrorNotification(): use the Generic integration to send an alert to an administrator,
		eg when an exception occurs.
		
		Input: errorString - must be a string object or a JSON object, ie, containing information about the failure.
		Note that other object types (eg, APXML objects) may cause this function to fail, so they should be converted
		to string before this function is called.
	*/
	
	var recip = "person or group name";  // must be a notifiable userID or groupID in xMatters.
	var integrationName = "";  // Put the name of the failing integration here and it will be included in the notification.
	
	// convert the errorString data (which may be a JSON object or a Java string) to a JS string
	var errStr = (typeof(errorString)=="object"? errorString: ""+errorString  );
	errStr = JSON.stringify(errStr);
	log.info("Entering sendErrorNotification(): errorString is " + errStr );
	var apxmlMessage = new APXMLMessageImpl();
	apxmlMessage.setToken("person_or_group_id", recip);
	apxmlMessage.setToken("subject", "An error was encountered by the " + integrationName + " integration while processing an incident.");

	apxmlMessage.setToken("message", errStr );

	log.info("Sending the following APXML data message to the Generic integration: " + apxmlMessage );
	ServiceAPI.sendAPXML(apxmlMessage, "generic", "generic");
}
  1. Save and close the file.

Add the exception handler

The next step is to locate the script name and line number of the instruction that caused the exception. To find this information, check the Integration Agent log for the failing integration:

  • In some cases, the exception message will include this information; for example:
ERROR - Integration Service (applications, hpomi-3-0) was unable to process a request due to an unexpected exception:
org.mozilla.javascript.EcmaError: TypeError: The element type "affects_business_service_list" must be terminated by the matching end-tag "</affects_business_service_list>". (D:\xMatters\integrationagent-5.1.7\integrationservices\hpomi-3-0\hpomi.js#91)
  • In other cases, the information will  be in the stack trace following the exception message. To find it, search for ".js:". The result will be similar to the following:
at org.mozilla.javascript.gen.D__xMatters_integrationagent_5_1_7_integrationservices_hpomi_3_0_hpomi_js_20._c_apia_http_1(D:\xMatters\integrationagent-5.1.7\integrationservices\hpomi-3-0\hpomi.js:91)

The above examples indicate that the instruction that caused the exception is in line 91 of the hpomi.js file: 

var response = new XML( new WSUtil().formatStringForE4X( restService.get(eventId)+"" ) );

To add your exception handler, prepend the line with:

try
{

And append it with:

}
catch (ex)
{
    sendErrorNotification( ex );
    throw ex;
}

The result should be similar to the following:

try
{
	var response = new XML( new WSUtil().formatStringForE4X( restService.get(eventId)+"" ) );
}
catch (ex)
{
	sendErrorNotification( ex );
	throw ex;
}

 

Save your files, restart the Integration Agent, and then try to reproduce the issue. The code changes should send an alert to the designated user or group via the Generic integration. You can then locate details about the failed notification in the Integration Agent log file by inspecting the messages preceding the "Entering sendErrorNotification()" information message.

For example, if the administrator receives an alert with information about a missing "affects_business_service_list" tag, they can review the Integration Agent log to determine what was actually sent by the management system. (In the above example, the management system was providing incomplete or corrupted XML data which caused an exception when sent to an XML parser.)

 

xMatters reference

Information created and supplied by Jeremy Brown. (DTN-5332)

 

 

 

 

 

 

Have more questions? Submit a request

1 Comments

  • 0
    Avatar
    Alistair Simpson

    Excellent!!

Please sign in to leave a comment.
Powered by Zendesk