xMatters running job in ServiceNow seems to be stuck

Problem

A scheduled job (for example, 'xM Incident Action - C') was in a running state for over 24 hours that resulted in events not being processed between xMatters and ServiceNow. Both times, ServiceNow had to restart the node the job was running on to get the job in a ready state so events could be processed again.

ServiceNow determined there's a script include (xMattersDataHelper) that was waiting on a response from xMatters at line 171. Either xMatters did not respond with a timeout or there was a communication issue that resulted in the response not being received, and the job hung waiting for this response.

What can be done to catch when a response is not sent back? Each time this happens, our on call teams are not being alerted when significant P1/P2 come in.

Environment

• All versions of xMatters
• ServiceNow integration 5.1.2
• ServiceNow London version

Resolution 

The following workaround in the xMattersDataHelper script include should help this situation. In the script, find the following line (around line 172):

var response = restMessage.execute(); 

Before this line, add:

restMessage.setHttpTimeout(<time_in_ms>);

For time_in_ms, enter a timeout length in milliseconds. We do not recommend exceeding a minute (in other words, 60000). However, the setting depends on your environment.

See this document from ServiceNow on setting connection timeouts.

If you're using version 5.4 of our ServiceNow integration, this line has already been added.

Cause

ServiceNow has difficulty continuing a script if it's waiting for a connection or response from a thread to come back. The best thing we can do is allow that request to expire and return a response so the script can continue running.