Does xMatters print out any information on load/current performance in a log? For example, how do I find data such as the current delay in processing events, the number of events pending processing, or any other indication of whether xMatters is backed up on processing live notifications?
You can monitor the current load on xMatters and determine the state of live notifications by running queries against the xMatters database.
Live notification states
The following list identifies all of the possible states of live notifications within the xMatters system:
- CREATED - The live notification has just been created.
- SUSPENDED - A notification has been suspended due to an out of coverage situation.
- SERVICEABLE - Represents a state where a notification is awaiting to be processed by a Device Engine.
- RETRY - Represents a state where a notification has been attempted but wasn't able to be delivered. A internal event will occur sometime in the future to requeue this notification for delivery.
- PROCESSING - Represents a state where a notification is in the state of being processed by a voice device engine.
- FAILURE - The live notification was not able to be delivered and will no longer be attempted.
- QUERY_PROVIDER - This state is similar to the SERVICEABLE state but notifications in this state are awaiting for device engines to query a provider for delivery status not send out a notification.
- DELIVERED_PROVIDER - Represents a state where a notification has been successfully delivered to a provider.
- QUEUED_PROVIDER - Represents a state where a notification has been successfully delivered to a provider but is currently queued at the provider awaiting deliver to a device. (Two-way paging protocol specific).
- DELIVERED_DEVICE - Notifications in this state have been successfully delivered to a device.(Two-way paging protocol specific).
- VIEWED_DEVICE - Notifications in this state have been successfully delivered to a device and the notification has been viewed. (Two-way paging protocol specific).
- RECEIVED_RESPONSE - A response was received from the user.
- DUPLICATE - Notifications that are duplicates for this source notification and device. These notifications will never be delivered.
- DELIVERED - Notifications in this state have been marked as DELIVERED. This means that the notification is no longer pushed out to the user but the user can still view the message via the webui and phone.
- DELINKED - Notifications in this state are at the end of their lifecycle. There is no transistion from this state.
- DISPATCHING - Represents a state where a notification is being dispatched to a device engine. Notifications are expected to transistion from this state within a configured threshold (5 mins). Otherwise they will be moved back to SERVICEABLE to requeue them.
You can run database queries to locate and list all of the notifications in the above states to gain an understanding of xMatters processing levels.
With the exception of number 5 below, the following examples illustrate how to run queries to determine load performance on an Oracle or DB2 database only (for SQL Server installations, the queries would require alternate coding):
1. Active events per Node
SELECT NODES.NAME, NODES.HOST_NAME, NODES.STATUS, NODES.ACTIVITY_STATUS, NODE_EVENTS.NUM_EVENTS FROM NODES, (SELECT NODE_ID, COUNT(NODE_ID) NUM_EVENTS FROM EVS WHERE EVS.STATUS='ACTIVE' GROUP BY NODE_ID) NODE_EVENTS WHERE NODES.NODE_ID = NODE_EVENTS.NODE_ID ORDER BY NUM_EVENTS DESC;
2. Active events with active live notifications per Node
SELECT NODE_ID, COUNT(NODE_ID) NUM_EVS_WITH_LIVE_NTFN FROM LIVE_NTFN, NTFN, EVS WHERE LIVE_NTFN.STATE IN ('CREATED','SERVICEABLE','RETRY','PROCESSING', 'QUERY_PROVIDER','DELIVERED_PROVIDER', 'QUEUED_PROVIDER', 'DELIVERED_DEVICE', 'VIEWED_DEVICE', 'RECEIVED_RESPONSE', 'DISPATCHING') AND LIVE_NTFN.NTFN_ID = NTFN.NTFN_ID AND NTFN.EV_ID = EVS.EV_ID GROUP BY NODE_ID ORDER BY NUM_EVS_WITH_LIVE_NTFN DESC;
3. Number of live notifications in each possible state
SELECT STATE, COUNT(STATE) NUM_IN_STATE FROM LIVE_NTFN GROUP BY STATE ORDER BY NUM_IN_STATE DESC;
4. Nodes with live notifications count divided by state
SELECT NODES.NAME, STATE, COUNT(STATE) NUM_IN_STATE FROM LIVE_NTFN, NTFN, EVS, NODES WHERE LIVE_NTFN.NTFN_ID = NTFN.NTFN_ID AND EVS.EV_ID = NTFN.EV_ID AND NODES.NODE_ID = EVS.NODE_ID AND (NODES.DELETED_ID is NULL OR NODES.DELETED_ID = -1) GROUP BY NODES.NAME, STATE ORDER BY NODES.NAME, NUM_IN_STATE DESC;
5. All events: creation time, total seconds to deliver last notification
Note: This query may return a high number of results.
Oracle & DB2
select E.EV_ID, E.WHEN_CREATED, cast((max(cast(APE.WHEN_CREATED as date))-cast(E.WHEN_CREATED as date))*24*60*60 as NUMERIC) duration, max(APE.WHEN_CREATED), count(APE.AUDIT_EV_ID) from AUDIT_AP_EVS AAE, AUDIT_PROC_EVS APE, AUDIT_NTFN_EVS ANE, EVS E, NTFN N, LIVE_NTFN LN where AAE.AP_AUDIT_EV_TYPE like 'LIVE_NOTIFICATION_PROVIDER_DELIVERED' and APE.AUDIT_EV_ID = AAE.AUDIT_EV_ID and APE.AUDIT_EV_ID = ANE.AUDIT_EV_ID and N.NTFN_ID = ANE.NTFN_ID and N.NTFN_ID = LN.NTFN_ID and APE.EV_ID = E.EV_ID group by E.EV_ID, E.WHEN_CREATED order by E.EV_ID desc;
select E.EV_ID, E.WHEN_CREATED, (DATEDIFF(MINUTE, E.WHEN_CREATED,APE.WHEN_CREATED)) duration, max(APE.WHEN_CREATED), count(APE.AUDIT_EV_ID) from AUDIT_AP_EVS AAE, AUDIT_PROC_EVS APE, AUDIT_NTFN_EVS ANE, EVS E, NTFN N, LIVE_NTFN LN where AAE.AP_AUDIT_EV_TYPE like 'LIVE_NOTIFICATION_PROVIDER_DELIVERED' and APE.AUDIT_EV_ID = AAE.AUDIT_EV_ID and APE.AUDIT_EV_ID = ANE.AUDIT_EV_ID and N.NTFN_ID = ANE.NTFN_ID and N.NTFN_ID = LN.NTFN_ID and APE.EV_ID = E.EV_ID group by E.EV_ID, E.WHEN_CREATED, APE.WHEN_CREATED order by E.EV_ID desc;
DTN-1451, SUP-2097, JDN-1118