Slow rate of outbound messages from integration agent to xMatters

Issue Overview

Some customers have reported that the integration agent will occasionally submit only one or two outbound messages to xMatters during each message exchange cycle. This happens even when the outbound queues contain many more messages.

Issue Details

The outbound messages are not lost, and will eventually be submitted to xMatters, but if the inbound message rate continues to exceed the artificially low outbound rate, the outbound queues will grow even larger and event submission to xMatters will be delayed.

Symptoms

To diagnose this issue, enable debug logging of the integration agent's message exchange by uncommentng the relevant section of the <IAHOME>/conf/log4j.xml file:

  <!-- APXML Message Exchanger -->  
<!--  
  <logger name="com.alarmpoint.integrationagent.messaging">  
    <level value="DEBUG"/>  
  </logger>  
-->  

With debug logging enabled, the number of messages submitted by the integration agent during each message exchange cycle will be reported in the integration agent logs as follows:

2013-02-01 08:11:11,927 [MessageExchange-1] INFO - Up to 50 messages will be exchanged with AlarmPoint Web Server http://www.company.com:8888/api/services/AlarmPointWebService.

2013-02-01 08:11:11,927 [MessageExchange-1] INFO - Submitting 2 APXML messages.

Cause

The integration agent's message exchange logic contains a timing/race condition which can prevent it from detecting that additional messages are available in the outbound queues. The presence of this issue is highly dependent on the system's current performance and work load; because the issue is timing related, it is not present on every system.

Resolution

This issue will be addressed in a future patch of the integration agent, but the following workaround will resolve this issue without any undesirable side effects.

To apply the workaround for this issue:

  1. Open the <IAHOME>/conf/IAConfig.xml in a text editor and locate the following section:
    <!--  
     | Factory for creating connections to embedded ActiveMQ broker.  
     | The redelivery process is implemented by application code, so disable on the connection.  
     +-->  
    <bean id="activeMqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" scope="singleton" depends-on="brokerService">  
        <!--property name="brokerURL" value="vm://localhost?jms.redeliveryPolicy.maximumRedeliveries=-1&amp;jms.prefetchPolicy.queuePrefetch=1"/-->  
        <property name="brokerURL" value="failover:(tcp://localhost:61618)?jms.redeliveryPolicy.maximumRedeliveries=-1&amp;jms.prefetchPolicy.queuePrefetch=1"/>  
        <!--property name="brokerURL" value="failover:(tcp://host1:61616,tcp://host2:61617,tcp://host3:61618)?jms.redeliveryPolicy.maximumRedeliveries=-1&amp;jms.prefetchPolicy.queuePrefetch=1"/-->  
    </bean>  
  1. Copy-and-paste the above section below itself and make the following changes to the copy:
    • Rename id="activeMqConnectionFactory" toid="activeMqConnectionFactoryOutbound"
    • Remove &jms.prefetchPolicy.queuePrefetch=1 from the brokerUrl.
  2. Your IAConfig.xml file should now resemble the following:
    <!--  
     | Factory for creating connections to embedded ActiveMQ broker.  
     | The redelivery process is implemented by application code, so disable on the connection.  
     +-->  
    <bean id="activeMqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" scope="singleton" depends-on="brokerService">  
        <!--property name="brokerURL" value="vm://localhost?jms.redeliveryPolicy.maximumRedeliveries=-1&amp;jms.prefetchPolicy.queuePrefetch=1"/-->  
        <property name="brokerURL" value="failover:(tcp://localhost:61618)?jms.redeliveryPolicy.maximumRedeliveries=-1&amp;jms.prefetchPolicy.queuePrefetch=1"/>  
        <!--property name="brokerURL" value="failover:(tcp://host1:61616,tcp://host2:61617,tcp://host3:61618)?jms.redeliveryPolicy.maximumRedeliveries=-1&amp;jms.prefetchPolicy.queuePrefetch=1"/-->  
    </bean>  
  
    <bean id="activeMqConnectionFactoryOutbound" class="org.apache.activemq.ActiveMQConnectionFactory" scope="singleton" depends-on="brokerService">  
        <property name="brokerURL" value="failover:(tcp://localhost:61618)?jms.redeliveryPolicy.maximumRedeliveries=-1"/>  
    </bean> 
  1. Locate the following section:
    <!--  
     | Create the object responsible for accessing ActiveMQ.  
     +-->  
  <bean id="queueManager" class="com.alarmpoint.integrationagent.jms.activemq.ActiveMQQueueManagerImpl" scope="singleton">  
        <property name="brokerService"><ref local="brokerService"/></property>  
        <property name="connectionFactory"><ref local="activeMqConnectionFactory"/></property>  
    </bean>  
  1. Change the connectionFactory reference so it refers to the new activeMqConnectionFactoryOutbound:
    <!--  
     | Create the object responsible for accessing ActiveMQ.    
   +-->    
  <bean id="queueManager" class="com.alarmpoint.integrationagent.jms.activemq.ActiveMQQueueManagerImpl" scope="singleton">     
     <property name="brokerService"><ref local="brokerService"/>property>      
    <property name="connectionFactory"><ref local="activeMqConnectionFactoryOutbound"/>property>   
   bean>  
  1. Save and close the file.
  2. Restart the integration agent.

xMatters Reference

SUP-6825, SUP-6894, DTN-3197, JDN-3821

Originally created by Don Clark

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk