How do I monitor the current load on an xMatters installation?

Does xMatters print out any information on load/current performance in a log? For example, how do I find data such as the current delay in processing events, the number of events pending processing, or any other indication of whether xMatters is backed up on processing live notifications?

You can monitor the current load on xMatters and determine the state of live notifications by running queries against the xMatters database.

Live notification states

The following list identifies all of the possible states of live notifications within the xMatters system:

  • CREATED - The live notification has just been created.
  • SUSPENDED - A notification has been suspended due to an out of coverage situation.
  • SERVICEABLE - Represents a state where a notification is awaiting to be processed by a Device Engine.
  • RETRY - Represents a state where a notification has been attempted but wasn't able to be delivered. A internal event will occur sometime in the future to requeue this notification for delivery.
  • PROCESSING - Represents a state where a notification is in the state of being processed by a voice device engine.
  • FAILURE - The live notification was not able to be delivered and will no longer be attempted.
  • QUERY_PROVIDER - This state is similar to the SERVICEABLE state but notifications in this state are awaiting for device engines to query a provider for delivery status not send out a notification.
  • DELIVERED_PROVIDER - Represents a state where a notification has been successfully delivered to a provider.
  • QUEUED_PROVIDER - Represents a state where a notification has been successfully delivered to a provider but is currently queued at the provider awaiting deliver to a device. (Two-way paging protocol specific).
  • DELIVERED_DEVICE - Notifications in this state have been successfully delivered to a device.(Two-way paging protocol specific).
  • VIEWED_DEVICE - Notifications in this state have been successfully delivered to a device and the notification has been viewed. (Two-way paging protocol specific).
  • RECEIVED_RESPONSE - A response was received from the user.
  • DUPLICATE - Notifications that are duplicates for this source notification and device. These notifications will never be delivered.
  • DELIVERED - Notifications in this state have been marked as DELIVERED. This means that the notification is no longer pushed out to the user but the user can still view the message via the webui and phone.
  • DELINKED - Notifications in this state are at the end of their lifecycle. There is no transistion from this state.
  • DISPATCHING - Represents a state where a notification is being dispatched to a device engine. Notifications are expected to transistion from this state within a configured threshold (5 mins). Otherwise they will be moved back to SERVICEABLE to requeue them.

You can run database queries to locate and list all of the notifications in the above states to gain an understanding of xMatters processing levels.

Example Queries

With the exception of number 5 below, the following examples illustrate how to run queries to determine load performance on an Oracle or DB2 database only (for SQL Server installations, the queries would require alternate coding):

1. Active events per Node

SELECT NODES.NAME, NODES.HOST_NAME, NODES.STATUS, NODES.ACTIVITY_STATUS, NODE_EVENTS.NUM_EVENTS  
FROM NODES,  
(SELECT NODE_ID, COUNT(NODE_ID) NUM_EVENTS FROM EVS  
 WHERE EVS.STATUS='ACTIVE'  
GROUP BY NODE_ID) NODE_EVENTS  
WHERE NODES.NODE_ID = NODE_EVENTS.NODE_ID  
ORDER BY NUM_EVENTS DESC;  

2. Active events with active live notifications per Node

SELECT NODE_ID, COUNT(NODE_ID) NUM_EVS_WITH_LIVE_NTFN FROM LIVE_NTFN, NTFN, EVS  
WHERE LIVE_NTFN.STATE IN ('CREATED','SERVICEABLE','RETRY','PROCESSING',  
 'QUERY_PROVIDER','DELIVERED_PROVIDER',  
 'QUEUED_PROVIDER''DELIVERED_DEVICE',  
 'VIEWED_DEVICE''RECEIVED_RESPONSE''DISPATCHING')  
 AND LIVE_NTFN.NTFN_ID = NTFN.NTFN_ID  
 AND NTFN.EV_ID = EVS.EV_ID  
GROUP BY NODE_ID  
ORDER BY NUM_EVS_WITH_LIVE_NTFN DESC;  

3. Number of live notifications in each possible state

SELECT STATE, COUNT(STATE) NUM_IN_STATE FROM LIVE_NTFN  
GROUP BY STATE  
ORDER BY NUM_IN_STATE DESC;  

4. Nodes with live notifications count divided by state

SELECT NODES.NAME, STATE, COUNT(STATE) NUM_IN_STATE  
FROM LIVE_NTFN, NTFN, EVS, NODES  
WHERE LIVE_NTFN.NTFN_ID = NTFN.NTFN_ID  
 AND EVS.EV_ID = NTFN.EV_ID  
 AND NODES.NODE_ID = EVS.NODE_ID  
 AND (NODES.DELETED_ID is NULL OR NODES.DELETED_ID = -1)  
GROUP BY NODES.NAME, STATE  
ORDER BY NODES.NAME, NUM_IN_STATE DESC;  

5. All events: creation time, total seconds to deliver last notification

Note: This query may return a high number of results.

Oracle & DB2

select E.EV_ID, E.WHEN_CREATED, cast((max(cast(APE.WHEN_CREATED as date))-cast(E.WHEN_CREATED as date))*24*60*60 as NUMERIC) duration,  
max(APE.WHEN_CREATED), count(APE.AUDIT_EV_ID)  
from AUDIT_AP_EVS AAE, AUDIT_PROC_EVS APE, AUDIT_NTFN_EVS ANE, EVS E, NTFN N, LIVE_NTFN LN  
where AAE.AP_AUDIT_EV_TYPE like 'LIVE_NOTIFICATION_PROVIDER_DELIVERED'  
 and APE.AUDIT_EV_ID = AAE.AUDIT_EV_ID  
 and APE.AUDIT_EV_ID = ANE.AUDIT_EV_ID  
 and N.NTFN_ID = ANE.NTFN_ID  
 and N.NTFN_ID = LN.NTFN_ID  
 and APE.EV_ID = E.EV_ID  
 group by E.EV_ID, E.WHEN_CREATED  
 order by E.EV_ID desc;  

SQL Server 

select E.EV_ID, E.WHEN_CREATED, (DATEDIFF(MINUTE, E.WHEN_CREATED,APE.WHEN_CREATED)) duration,  
max(APE.WHEN_CREATED), count(APE.AUDIT_EV_ID)  
from AUDIT_AP_EVS AAE, AUDIT_PROC_EVS APE, AUDIT_NTFN_EVS ANE, EVS E, NTFN N, LIVE_NTFN LN  
where AAE.AP_AUDIT_EV_TYPE like 'LIVE_NOTIFICATION_PROVIDER_DELIVERED'  
 and APE.AUDIT_EV_ID = AAE.AUDIT_EV_ID  
 and APE.AUDIT_EV_ID = ANE.AUDIT_EV_ID  
 and N.NTFN_ID = ANE.NTFN_ID  
 and N.NTFN_ID = LN.NTFN_ID  
 and APE.EV_ID = E.EV_ID  
 group by E.EV_ID, E.WHEN_CREATED, APE.WHEN_CREATED  
 order by E.EV_ID desc;  

xMatters Reference

DTN-1451, SUP-2097, JDN-1118

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk