Updating an integration to delete incidents synchronously

Issue overview

Many integrations allow the management system to send a request to xMatters to delete pending notifications for a specified incident ID. For example, if an incident is reassigned or its priority is raised, the management system will send a message to the integration agent requesting that pending notifications be cancelled for that incident ID.

Older integrations often use the DEL integration service to request these cancellations. This mechanism is acceptable in most instances if no new notifications are to be requested; however, if the DEL is followed by an ADD, then new notifications may not be created. This failure occurs if the DEL and the ADD are not sent to xMatters in the correct order, in which case the DEL negates the ADD.

The underlying problem is that messages for different integrations are sent to xMatters via individual outbound queues belonging to the integration agent. For example, the BMC BPPM integration has its own outbound queues, and so does the DEL integration. Messages that are added to a given queue are guaranteed to be sent to xMatters in the order in which they were received, but there is no such guarantee if the messages are in different queues.

For this reason, we recommend that an integration not use the DEL integration service. Instead, notification requests (ADDs) and cancellations (DELs) can be sent to the same integration service, ensuring that they will be delivered to xMatters in the correct order. This is accomplished by specifying the appropriate method (ADD or DEL) in the integration script while composing the APXML request to xMatters.

In the sections below, you'll find the following:

While doing the work described below, you may need to refer to the integration agent guide for detailed information about the concepts involved.


This article uses version 1.1.2 of the BMC BPPM integration throughout as an example of the processes required. Your integration may differ slightly - or significantly - based on the trigger and initiation processes involved.

Configure the management system

Each integration uses its own mechanism to initiate an injection into the integration agent: some use a batch file to call the APClient.bin, while others call it directly. Other integrations are capable of injecting directly into the integration agent directly, without using the APClient.bin function.


As in many integrations, the BPPM integration relies upon the management system using a batch script to inject ADDs and DELs into the integration agent. The integration does this by using batch script files to invoke the xMatters-supplied command-line tool APClient.bin.

To configure the management system, append an extra parameter to the xm_send_notification.bat script and set it to "false". The DEL script will be replaced with a copy (essentially) of the ADD script, with the following attributes:

  • the DEL script has the same number of parameters as the ADD script
  • the DEL script invokes the same integration service
  • all of the parameters can be empty, except for the incident ID and final parameter, which will be set to "true" 


"C:\Program Files\xmatters\Integrationagent\bin\apclient.bin" --map-data bmcbppm %1 %mc_ueid%
false ...[parameters omitted]... "%msg%"


"C:\Program Files\xmatters\Integrationagent\bin\APClient.bin.exe" --map-data del "%mc_ueid%"

Note that these two scripts both invoke APClient.bin in the same way, but they provide different integration names after the "--map-data" argument. This indicates that the ADDs are sent to the bmcbppm integration service, and the DELs are sent to the del integration service.

Append an extra parameter to the ADD script (xm_send_notification.bat, in this example) indicating whether the integration should send a DEL to xMatters.

Updated xm_send_notification.bat

"C:\Program Files\xmatters\integrationagent\bin\apclient.bin" --map-data bmcbppm20 %1 %mc_ueid% "false" ... "%msg%" "false"


Replace the contents of the "del" batch script so that it has the same number of arguments as the "add" script, and the final parameter is "true":

Updated xm_delete_notification.bat

"C:\Program Files\xmatters\integrationagent\bin\apclient.bin" --map-data bmcbppm %1 %mc_ueid% "" ... "" "true"

Note: In this specific integration, the %mc_ueid% parameter contains the incident ID that designates the notifications to be deleted.

Configure the integration scripts

This stage of the conversion is more complex than the previous one, and requires the following steps:

Configure the integration's XML definition file

The XML file defines aspects of the integration service that are used by the integration agent at startup time. Among these are the parameters that the integration service expects when a message is injected from the management system.


To configure the XML definition file, we add a parameter that matches the one appended in step 1 above. Using bmcbppm.xml as an example, we see a list of parameters that match the ones in the xm_send_notification.bat shown above:

parameters in bmcbppm.xml

<parameter type=>recipients</parameter>
<parameter type="string">incident_id</parameter>
...[parameters omitted]...
<parameter type="string">message</parameter>

We will add a parameter to contain the true/false information indicating whether the integration should send a DEL to xMatters:

<parameter type="string">recipients</parameter>
<parameter type="string">incident_id</parameter>
...[parameters omitted]...
<parameter type="string">message</parameter>
<parameter type="string">delete_existing</parameter>

This parameter creates a token called "delete_existing" that will be included in the APXML message created by the integration service. The integration script uses this token to determine whether an ADD or a DEL request is to be sent to xMatters.

Provide code to process the delete_existing token

Next, we will update the integration service script containing the actual instructions for processing injections from the management system. If the integration uses APClient.bin, then the processing instructions will be in a function called apia_input (APXML), which accepts an APXML message from the integration agent's inbound queues and typically returns an APXML message to the integration agent's outbound queues. In its most basic form, this function consists of the single instruction "return apxml;".

If the integration does not use APClient.bin for incident injections, a different function handles the incoming injection. For example, apia_http handles injections sent via direct request to the integration agent's HTTP endpoint. 


The BMC BPPM integration uses apia_input, although it uses a variable to send a copy of the APXML request to the outbound queues, rather than sending the original message.

The objective of this step is to add code that will test the delete_existing token in the incoming APXML message. We will create a variable called should_delete to contain the result of this test.

We will test the delete_existing token before any others, because other tokens may be null if this is a DEL injection:

// test whether the "delete_existing" token is true or false in the incoming APXML
var should_delete = apxml.getValue("delete_existing")
var msEpochTimestampSec = apxml.getValue("date_reception");

Provide code to send DEL requests to xMatters

Now that we have a variable to tell us whether the incoming APXML message contains an ADD or a DEL request, we can modify the integration script to prepare and send the appropriate outgoing message.

In the simplest case, we will want to send a request to xMatters only to cancel pending notifications for a specified incident ID.


Because our inbound APXML includes the incident ID, we need only set the setMethod and setSubclasss tokens in the outbound APXML message:


To prevent this code from affecting all injections (including ADDs) we will need to put a conditional block around the existing code. The result will be similar to the following:

Revised apia_input to detect delete_existing flag and send del request to xMatters

function apia_input(apxml) {
    var result = null;
    // test whether the "delete_existing" token is true or false in the incoming APXML
    var should_delete = apxml.getValue("delete_existing");

    if (should_delete)
        return (apxml);
        var msEpochTimestampSec = apxml.getValue("date_reception");
        var msEpochTimestampMilli = Long.parseLong(msEpochTimestampSec) * 1000;

        apxml.setToken("bppm_epoch_timestamp_ms", Long.toString(msEpochTimestampMilli), APXMLToken.Type.NUMERIC);

        // Forward enriched APXML to xMatters, subject to deduplication
        var webService = new xMattersWS();
        var recentlyReceived = EventDeduplicator.getInstance().recentOccurrenceCount(apxml, DEDUPLICATOR_FILTER);

        if (recentlyReceived <= 1)
            result = apxml;
            log.warn("***Deduplicator Suppressed Notification ***: An event with tokens " + webService.convertTokensToString(apxml) + " has been injected into the " + 
            apxml.getValue("agent_client_id") + " event domain " + recentlyReceived + " times within the configured suppression period.  It has been suppressed." );
        return result;

Note that this will only delete any pending notifications for the supplied incident ID, and no notifications at all will result from the DEL injections from the management system. ADD injections will be handled exactly the same as previously.

If this is not the desired behaviour (in other words, if the integration is expected to send notifications even when a DEL injection is received from the management system), then it is possible to respond to the DEL injection by sending a pair of requests to xMatters: a DEL and then a subsequent ADD. In other words, a single injection from the management system results in two request messages to xMatters.

This functionality is provided by the integration agent utilities software package, which is already included with many integrations. The utilities package is an optional add-on to the integration agent which provides capabilities such as a deduplicator filter to prevent event storms. It also provides the capability to send a DEL request to xMatters, either alone or in conjunction with a subsequent ADD request.

You can tell whether an integration includes the utilities package by looking for a reference to "xmattersws.js" in the "load" statements at the top of the main integration script. For example, in bmcbppm.js we see:


The BMC BPPM integration already includes the utilities package, but does not use it to enqueue messages for delivery to xMatters. (The utilities package was initially added to this integration purely so that event deduplication could be performed.) The utilities package can be added to any integration agent-based integration, but the process is outside the scope of this document.

To modify an integration so that it responds to a DEL injection by deleting old notifications and then creating new ones, use the utilities package's submitAPXML() method instead of the return APXML instruction shown previously. For example:

var sentSuccessfully = submitAPXML(apxml, shouldDelete);

This code will test the value of its second parameter and, if it is true, will send a DEL request message to xMatters before sending an ADD. It will ensure that the incident_id and apxml_process_group tokens are the same in the ADD and the DEL request, thereby guaranteeing that the messages are sent to xMatters in the correct order.

Ensuring synchronous deletions

In this document, the term "synchronous deletions" signifies that a DEL request and a subsequent ADD request will be sent from the integration agent to xMatters in the order in which they were received from the management system.

The integration agent's queueing mechanism ensures that messages will be sent synchronously as long as three conditions are met:

  • the request messages have the same integration name
  • they have the same apia_priority token
  • they have the same apia_process_group token

The instructions provided so far will ensure that messages have the same integration name, but they will not address the need for the same apia_priority token or the same apia_process_group token.

It is possible to set all messages to have "normal" priority (the options are "normal" and "high").


When added to the BPPM integration's apia_input function, this instruction will ensure that all APXML messages sent to xMatters will have the same apia_priority.

apxml.setToken("apia_priority", "normal");

Similarly, it is simple to ensure that all DEL and ADD messages pertaining to a given incident ID have the same apia_process_group token. To do so, assign this token the same value as the incident_id token:

apxml.setToken( "apia_process_group", apxml.getToken("incident_id") );

Many older integrations were not designed to use these tokens, and so there is no possibility of conflict between the above code and the designed behaviour of the integration. However, it is possible that an integration will have existing code that uses the apxml_priority token to force the message into the integration's "high" or "normal" outbound queue.

The actual benefit of forcing a message into the "high" outbound queue is generally negligible, however, especially when compared to the cost of failed notifications if a DEL and an ADD reach xMatters out of order. Therefore it is recommended that you force all notifications to use the same apxml_priority token unless you know of a compelling reason for doing otherwise.

Further information

See the xMatters integration agent guide for more detailed information about the processes described in this article.

xMatters internal reference: DTN-4101, DTN-3922

Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk