Most xMatters integrations are written to be single-threaded: they process one incident injection at a time, and subsequent incidents remain in the integration's inbound queue until they can be processed.
However, if increased throughput is required, the integration agent has the ability to process more than one incident at a time - this capability is referred as "concurrent processing," or "multi-threading". A client might need to exploit this capability if the number of incident injections from the management system is very large, or if incident processing involves time-consuming operations such as requests to external web services.
For an integration to be multi-threaded, the integration scripts must be configured appropriately. This article describes the process of adding multi-threading capability to an existing integration service.
- xMatters integration agent 4.1 patch 5 or higher.
- "Del" injections from the management system must be processed by the same integration service as "Add" injections. If the integration uses the "DEL" integration service to process "Del" injections, then the integration must be updated according to the instructions in https://support.xmatters.com/hc/en-us/articles/203632939.
Note: If the management system does not issue "Del" messages to the integration, then the integration is ready to be configured for multi-threading.
Configuring the integration service
Assign a value to the apia_process_group
The "apia_process_group" token is used to identify messages which must be processed sequentially. For example, many management systems send a Del message to the integration agent when a ticket is reassigned or re-prioritized, and then an Add is sent immediately afterward:
- The Del message is used to tell xMatters to terminate any pending notifications for the specified incident.
- The Add message causes a new round of notifications to be initiated.
Clearly, these two messages must be delivered to xMatters in the correct order, or the Del will cause the new notifications to be immediately terminated.
Unrelated incident messages can be safely sent to xMatters without the need to preserve order. This behavior is preferred in a multi-threaded integration because it allows messages to be delivered to xMatters without having to wait for earlier messages which might have been blocked during some processing step. To enforce this behavior, the apia_process_group is used to associate related messages with each other so they will be processed in the correct order.
The apia_process_group is also used to distinguish unrelated messages so that they can be processed concurrently.
To assign an appropriate value to the apia_process_group token:
Configure the management system to assign the incident ID to the apia_process_group token. The details of this step will vary according to the specific management system and the way it sends incidents to the Integration Agent. If the management system uses apclient.bin to send incidents to the IA:
- In the command string that invokes apclient.bin, find the parameter that represents the incident ID and duplicate it. For example, in the BMC BPPM integration, a script invokes apclient.bin using the %mc_ueid% environment variable to represent the incident ID:
"C:\Program Files\xmatters\Integrationagent\bin\apclient.bin" --map-data bmcbppm20 %1 %mc_ueid% "false" "%severity%" ...
Modify this script to include a second instance of the incident ID as follows:
"C:\Program Files\xmatters\Integrationagent\bin\apclient.bin" --map-data bmcbppm20 %1 %mc_ueid% %mc_ueid% "false" "%severity%" ...
- Modify the integration's XML configuration script to associate the new parameter with the apia_process_group token. For example, in the BPPM integration's configuration file "bmcbppm.xml" replace:
<parameter type="string">recipients</parameter> <parameter type="string">incident_id</parameter> <parameter type="string">fyi</parameter>
<parameter type="string">recipients</parameter> <parameter type="string">incident_id</parameter> <parameter type="string">apia_process_group</parameter> <parameter type="string">fyi</parameter>
To determine how to modify your integration, refer to the specific xMatters integration guide to find out how your management system indicates the incident ID, and then add a second instance of this parameter to the apclient invocation and the integration configuration file as shown in the example above.
If your integration uses a mechanism other than apclient.bin to send incidents to the Integration Agent, then the Integration Agent's inbound queues are probably not being used, and so multiple threads cannot be managed by the apia_process_group token.
To summarize, the desired outcomes are:
- The management system will send all adds and dels to the same integration service (i.e., dels will not be sent to the "del" integration service).
- All injections (whether "add" or "del") from the management system will have an apia_process_group token that contains the incident_id before the injection reaches the Integration Agent's inbound queue.
Ensure that all events have the same priority
If an Add and a Del injection have been assigned different priorities by the management system, then they will not be sent to the same outbound queue and may reach xMatters out of order.
To prevent this from happening, open the integration's XML configuration file and add a constant that will overwrite any priority tokens to ensure that all injections are processed in the correct order.
If the XML configuration file does not already have a "constants" section, add one with the following contents:
<constants> <constant name="apia_priority" type="string" overwrite="true">normal</constant> </constants>
If the file already has a constants section, then just add the above "apia_priority" line to it.
Refer to the xMatters Integration Agent documentation for further information about the apia_priority token.
Configure the the concurrency setting for the integration service
The concurrency setting is configured in the integration's XML definition file, located within the IntegrationServices folder. For example, in the CA Spectrum integration, the file is located in the integrationservices\caspectrum folder and is called caspectrum.xml
The concurrency setting is governed by code similar to the following:
<concurrency> <normal-priority-thread-count>3</normal-priority-thread-count> <high-priority-thread-count>3</high-priority-thread-count> </concurrency>
This code can be added, if it isn't already present. Refer to the Ping integration definition file for an example.
In this case, the integration service will be able to process three separate incident injections concurrently (assuming that all injections have the default "normal" apia_priority). The "High" apia_priority queue, despite its name, does not provide any preferential treatment; it serves only to segregate particular messages from other messages. This segregation is to be avoided if Add and Del messages are to be processed in the correct order.
Test and tune
Once the integration service has been configured as described above, restart the integration agent and test basic functionality (for example, ensure that notifications are still being delivered by xMatters.)
Validation and tuning can be done rigorously or informally. For example, if the integration performs a time-consuming operation such as making a web service request before forwarding the incident information to xMatters, and the operation is known to take an average of five seconds, then you can inject three incidents to the integration simultaneously and then check the messages in the integration agent log file (or the xMatters Event Log.) If the incidents were forwarded to xMatters at roughly the same time, then concurrency is working correctly; if they are delivered five seconds apart, then some troubleshooting may be required.
To tune the concurrency setting informally:
Use the command "iadmin get-status" while the management system is sending incidents to the integration agent. In the output from the iadmin script, look for the message "Normal priority inbound APXML queue size:" (note that there is one such message for each integration service, so be sure you are looking at the correct one.) If the number of messages in the inbound queue becomes large, increase the concurrency setting.
To tune the concurrency setting more rigorously:
Enable logging of the individual queues by adding the following lines to <IAHOME>\conf\log4j.xml:
<logger name="org.apache.activemq.broker"> <level value="DEBUG"/> </logger>
You will now be able to monitor the size of the inbound queues in the integration agent log file as messages are accepted from the management system and processed by the integration agent. Ideally, there will always be an available thread waiting to process a message when it is added, so the number of messages in the inbound queue should always be very small. Occasional spikes may result when the management system injects a large number of incidents, but the queued messages should be processed promptly. If the number of messages in the queue continues to grow over seconds or minutes, then the concurrency setting should be increased.
You can isolate the relevant messages by opening a command window in the <IAHOME>\log folder and issuing a command similar to the following (replace "test.test" with the name of your integration service):
find "inbound.test.test.normal" alarmpoint.txt | find "[BrokerService[localhost] Task]"
By monitoring these messages over a suitable period, you can gain an accurate understanding of the integration agent's ability to process injected incidents in a timely manner, and by updating the concurrency setting you can adjust the configuration if necessary.
Filtering and screening
If the integration's inbound queues remain large even when the integration is multi-threaded, then individual messages may be taking a long time to process. This commonly occurs when the integration must communicate with the management system via web service request.
An obvious source of incoming messages is the management system. Management systems sometimes inject large volumes of spurious messages (such as those which do not result in desirable notifications). These messages can generally be suppressed by the integration agent's Portable Filtering and Suppression module. If it's feasible, prevent spurious messages from reaching the integration agent by configuring the management system appropriately.
A less obvious source of incoming messages is the xMatters server. For example, if xMatters is configured to send an annotation to the management system whenever a notification is delivered, then the annotations will be handled by the same incoming queues as the injections from the management system, and each annotation will consume a thread during processing. Annotations are commonly delivered to the management system via web service request, and the corresponding thread will be unavailable for other purposes until the management system provides a response. Therefore, it is advisable to configure xMatters to not send delivery annotations to the management system, since deliveries can be tracked via the xMatters user interface, if necessary.
DOC-4179 Originally created by Jeremy Brown