Configuring the Integration Agent to deduplicate events

Deduplication allows an integration service to prevent "event storms," which can occur when a management system sends multiple notification requests pertaining to a single incident. By detecting and blocking duplicate notification requests, the Integration Agent can prevent event storms from affecting xMatters users.

The Integration Agent's deduplicator has three components:

• A "filter" which specifies the conditions that will cause incident requests to be discarded.
• Script instructions that match incident requests to the filter conditions.
• Deduplication capability in the Integration Agent, which is invoked by the script instructions.

The filter and script instructions must be understood and used properly for the deduplicator to perform correctly.

Deduplication Filters

Deduplication filters are specified in <IAHome>\conf\deduplicator-filter.xml. In order for an integration service to perform deduplication, that integration service must specify a filter that exists in deduplicator-filter.xml. The filter will typically have the same name as the integration service. For example, the "sample-plan" integration might have a filter that is specified as follows:

<<filter name="sample-plan">
	<predicates>
		<predicate>building</predicate>
		<predicate>city</predicate>
	</predicates>
	<suppression_period>60</suppression_period>
	<window_size>10</window_size>
</filter>

In addition to its name, a filter has three components:

• Predicates: Each predicate must correspond to a property* that is present in the data that will be forwarded to xMatters by the Integration Agent. Based on the values of these predicates, each message will be categorized as "unique" or "duplicate" by the Integration Agent. See below for examples and explanation.
• Suppression Period: After each unique notification request is received by the Integration Agent, a record of its predicate values is stored in memory for this number of seconds, and any subsequent notification request is discarded if all its predicates match the stored values. When the suppression period expires for those predicate values, the Integration Agent "forgets" that combination of predicates.
• Window Size: If this number of unique notification requests is received within the suppression period, then the oldest one is "forgotten", meaning that a second such request will be allowed (even if the suppression period has not yet expired).

*Deduplication works with newer JSON-based integrations as well as the older APXML based ones, but behind the scenes the event properties are translated to APXML for the purposes of deduplication.

Here are a couple of helpful examples using the "sample-plan" integration service and filter:

Example 1: Requests are deduplicated until the suppression period expires

• At 10:10:10, the sample-plan integration service receives a notification request with properties:

   "building": "Building A",
   "city": "Victoria"

  This request is processed, the Event request is delivered to xMatters, and the deduplicator stores "10:10:10", "Building A", "Victoria" in memory.

• At 10:10:11, a second request is received that has the same three properties. The second request is suppressed.
• At 10:11:10, the 60-second suppression period expires and the "10:10:10", "Building A", "Victoria" record is purged from the deduplicator's memory.
• At 10:11:11, another request is received with tokens:

  "building": "Building A",
   "city": "Victoria"

  This request is processed, the Event request is delivered to xMatters, and the deduplicator stores "10:11:11", "Building A", "Victoria" in memory.

Example 2: Requests are deduplicated until the window size is exceeded

• At 10:10:10, the sample-plan integration service receives a notification request with properties:

  "building": "Building A",
   "city": "Victoria"

  This request is processed, and the deduplicator stores "10:10:10", "Building A", "Victoria" in memory.

• At 10:10:11, nine more requests are received, all of which are different from each other as well as from the original request. All the requests are processed and a time-stamped record of each is stored in memory.
• At 10:10:12, one more unique request is received. A time-stamped record of it is stored in memory. Since eleven unique records have now been received, the window size is exceeded, causing the first record ("10:10:10", "Building A", "Victoria") to be purged from the deduplicator's memory.
• At 10:10:13, the sample-plan integration service receives another notification request with properties:

  "building": "Building A",
   "city": "Victoria"

  This request is processed, the Event request is delivered to xMatters, and the deduplicator stores "10:10:13", "Building A", "Victoria" in memory.

Script Instructions

The following script instructions are required for the deduplicator to perform correctly:

DEDUPLICATION_FILTER_NAME = "sample-plan";

• This instruction creates a constant whose value matches the name of the deduplication filter.
• It is typically found in the integration's configuration.js script file (if there is one); otherwise it should be at the top of the main integration script (e.g., sample-plan.js) immediately after the "importPackage", "importClass", and "load" instructions.
• The constant is used by the utility functions in the XMUtil script package that ships with the Integration Agent.

load("lib/integrationservices/javascript/event.js");

• This instruction is typically found at the top of an integration service's main script, e.g., sample-plan.js
• event.js contains instructions to load the other utility script files that ship with the Integration Agent, including xmutil.js
• If you don't want your integration to load event.js, you can load xmutil.js directly (i.e., by replacing "event.js" with "xmutil.js" in the above instruction). However, due to dependencies, if you load xmutil.js directly, you will have to load log.js first. See the code in the "Example" section below for details.

if (XMUtil.deduplicator.isDuplicate(event.properties)) {
	XMUtil.deduplicate(event.properties);
	return;
}

• After the integration script has prepared an Event request to be sent to xMatters, the above code determines whether the request should be deduplicated or delivered.
• The argument can be either an APXML message (typically called "apxml"), or the properties component of an Event object (e.g., "event.properties") as shown above. If it receives an event object, the xmutil deduplicator will convert it internally to APXML.

XMUtil.deduplicator.incrementCount(event.properties);

• If the notification data is an APXML message, then this code should immediately precede the instruction that sends the notification request to xMatters.
• If the notification data is an event object, then this code should immediately follow the instruction that sends the notification request to xMatters.
• The reason for this distinction is:
  • Event objects are sent from the Integration Agent to xMatters via an HTTP POST (typically "XMIO.post"), which may fail and throw an exception. In this case, the Integration Agent will retry the HTTP POST, but if the XMUtil.deduplicator.incrementCount instruction has already been executed, then the retry will be suppressed by the deduplicator.
  • APXML messages are sent to xMatters via the "return" statement at the end of an apia_input() or apia_http() function. The XMUtil.deduplicator.incrementCount instruction will never be executed if it follows the "return" statement.

Example: Adding deduplication to the generic integration

As of Integration Agent 5.2.0, all of the included sample integrations do include deduplication.  The "sample-plan" integration provides a clear example with comments, in sample-plan.js.

The "generic" integration script provides an uncomplicated example that does not include deduplication. To add deduplication to the generic integration, make the following changes in generic.js:

• Near the top of the file, replace:

  importClass(Packages.com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider);
  
  /* Created by Adam Dahlquist, adahlquist@xmatters.com
  

  with:

  importClass(Packages.com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider);
  
  // Declare the name of the deduplicator filter
  DEDUPLICATION_FILTER_NAME = "generic";
  
  // Load the Integration Agent script packages that include the deduplicator code
  load("lib/integrationservices/javascript/log.js");
  load("lib/integrationservices/javascript/xmutil.js");
  load("lib/integrationservices/javascript/apxml.js");
  load("lib/integrationservices/javascript/xmio.js");
  
  /* Created by Adam Dahlquist, adahlquist@xmatters.com
  

• Replace the apia_input function with the following code:

  function apia_input(apxml)
  {
  	IALOG.info("Apia_input received the following APXML message: \n" + apxml);
  
  	// Event processing code goes here, ie, before the deduplication code
  
  	// Check whether the processed event message is a duplicate, and discard it if so
  	if (XMUtil.deduplicator.isDuplicate(apxml)) {
  	XMUtil.deduplicate(apxml);
  	return;
  	}
  
  	// Increment the deduplicator.  Note that this instruction goes AFTER the call to XMIO.post in REST-based integrations.
  	XMUtil.deduplicator.incrementCount(apxml);
  
  	// Send the message to xMatters and terminate apia_input
  	return apxml;
  }
  

• You will also need to add a filter to <IAHome>\conf\deduplicator-filter.xml. You can use the example filter from this KBA, exactly as it appears above.
• Save the files and restart the Integration Agent.

A couple of things to note:

• If the filter includes a predicate that does not match any property in the notification request, then that notification request will be ignored by the deduplicator and will never be suppressed.
• Some early integration scripts used the deduplicator instructions incorrectly, so if the above information contradicts the released code, then the released code is probably at fault.

 xMatters reference: DTN-6286 Originally by Jeremy Brown