Second look at callbacks - through the integration agent (GIG part 7)

NOTE: This article uses the older integration agent and any new integrations (or learning) should be directed to the newer xMatters Agent. Check out the Special Agent 6 article for the updated version of this article. 


Well, here we are again. In our last article, we introduced the integration agent and got it all set up on the new Smart Ovens™ to notify Master Chef when the alarm has gone off. And remember back in part 5 we introduced callbacks?

Well, in this article, we’re going to use the response feature of xMatters along with callbacks to allow Master Chef to set the temperature or turn the oven off in the rare case that he gets pulled away from the bakery…. He’s a busy guy and life happens. Read on for details.

Oh, and I think I’ve finally worked out a better editor, so hopefully there will be some consistency in formatting this time.

In our first look at callbacks we introduced responses, so check back there if you need a refresher. I added a few responses to our Timer Finished form that will take actions on our oven based on what Master Chef needs to do.

As long as we are poking around in the UI, let’s add the “Integration Service”. An Integration Service makes xMatters On-Demand aware of which integration agents serve what integrations. This helps xMatters know where to send responses when they come in. So cruise over to the Event Domains page on the Developer tab.

There is a lot of stuff here that is used for legacy purposes, but the piece we are interested in is the “applications” event domain. This is the “channel” on which all incoming REST integrations come in on and you might remember this as the “domain” value we set in the ovens.xml configuration file on the integration agent side. Click the applications link and scroll down to the bottom to Integration Services.

Click Add New and in the Name field, type “ovens”. This is the name of the service we set just below the domain in the ovens.xml file. These must match up as they help xMatters and the integration agent figure out what each is talking about. I highly recommend adding a descriptive description… you will thank yourself (or your co-workers will!) 6 months down the line when trying to figure out what the heck this thing is.

Save that, and then go back on the Event Domains page - you may see the Status for applications as "Shut Down". If your integration agent is running, refresh the page and you should see it switch to "Active". Hooray!


This means the integration agent successfully registered the ovens integration with xMatters and that responses can be passed back to the integration agent. If you don’t see Active displayed after a couple of refreshes, check that the domain and name in the ovens.xml file matches the event domain and integration service name in the integration services page. Also, the IAHOME/log/AlarmPoint.txt file will contain any errors that are thrown.


Next, we’ll switch gears and configure the integration agent side.


The integration agent is a powerful little utility that can serve as a middleware for translating the square peg coming from a management system (such as our Smart Ovens™) to the round hole that xMatters On-Demand will accept. But it can also act as a way to pass information from the outside world into our protected internal network. Woah! Black magic you say!


Everyone knows that routers and wi-fi networks are great at letting requests out, but they by default clamp down and don’t let anything into the network from the outside. There are a lot of nasties out there and we wouldn’t want to let them in. This layer of security is provided by the firewall, either by the ISP or the router, or somewhere in between. For our little bakery it might not be a big deal to poke a hole in the network and let the outside world (such as xMatters On-Demand) reach our little integration agent. However, anyone who has tried to make inbound network requests to their Enterprise IT departments will appreciate how much fun it can be in the real world.


This is where the black magic comes in. Instead of making holes in our firewall, we can tell the integration agent to poll xMatters for stuff. So instead of On-Demand calling up the integration agent to say stuff needs to happen, the integration agent can check in and ask xMatters if stuff happened. There is a small delay but it is rarely noticeable in my experience.


This black magic is controlled by a setting in the IAHOME/conf/IAConfig.xml file called external-service-request. direct mode means xMatters On-Demand will initiate the conversation (push to the integration agent), while indirect means the integration agent will initiate the conversation (pull from xMatters).


So fire up your favorite text editor on the Smart Oven™ and update the external-service-request mode to be indirect. Like so:

  <external-service-request mode="indirect" />

The next step is to tell the ovens integration that we want to include some callbacks when it creates the event initially. Crack open the IAHOME/integrationservices/ovens/configuration.js and look for the CALLBACKS variable. We set this to an empty array previously which tells xMatters we don’t care about knowing about stuff that happens to the event. But now we do! So, update that variable to include "response" in the array. Like so:

// -------------------------------------------------
// Callbacks requested for this integration service.
// -------------------------------------------------
CALLBACKS = ["response"];

Note that there are two other types of callbacks we could have, “status” and “deliverystatus” but in our case the SmartOvens™ don’t care about if the notifications were delivered or the status of the events, just what Master Chef wants to do about it. In other integrations we might want to take action such as add a message to work notes in an Incident (such as what the ServiceNow to xMatters integration can do) that a notification was delivered to so and so on some and some device at such and such time.


Ok, cool. We’ve got the integration service registered with xMatters On-Demand and our integration agent configured to tell xMatters about the callbacks, the last piece is to deal with the responses as they come in. So roll up your sleeves and crack open the ovens.js file.


Since our SmartOvens™ can only accept instructions via command line, we’ll need to add a function to let the integration agent make command line calls. The “ping” integration service that comes with the integration agent has a version, but I’ve found the one one below more robust and more useful. Feel free to use either, but add the execute and copyToString functions to this file. For readability, I’ve moved these to the execute.txt file attached.


In our previous post we copied several functions, notably the apia_event function which handled the incoming message from the oven and one of which was the apia_callback function that we kind of ignored. I’ll give you one guess as to what this function does…. Right now, all it does is print a couple of pieces of information from the message from xMatters:

function apia_callback(msg)
   var str = "Received message from xMatters:\n";
   str += "Incident: " + msg.incident_id;
   str += "\nEvent Id: " + msg.eventidentifier;
   str += "\nCallback Type: " + msg.xmatters_callback_type;;


Since msg is a JSON object, we can reference the fields inside it by name. I like to see the entire payload sent back as it is very useful for debugging purposes. So let’s add a line to dump the entire payload to the log and we’ll add our execute function. I’ve taken the liberty to peruse the oven documentation for how to change the temperature via command line and it turns out there is a script called /etc/smartovenstm/ that accepts one parameter - the temperature to change the oven to. Just what the baker ordered! This represents our interface from the integration agent to the oven and where the hand-off from our xMatters realm to the oven occurs. Here is the updated function:

function apia_callback(msg)
   var str = "Received message from xMatters:\n";
   str += "Incident: " + msg.incident_id;
   str += "\nEvent Id: " + msg.eventidentifier;
   str += "\nCallback Type: " + msg.xmatters_callback_type;
   str += "\nRAW: \n" + JSON.stringify( msg );;

   // Check to see if this is a response and not
   // a deliveryStatus or status type callback. 
   if( msg.xmatters_callback_type == 'response' ) {
      cmds = ["/etc/smartovenstm/", "150"];
      IALOG.debug( "cmd: " + cmds );
      var output = execute( cmds ); 'SmartOvenTM result: ' + output );


Oh, and one other thing we need to do is import a couple of java libraries. Add these two lines to the top of the ovens.js file. These are for the StringWriter and ProcessBuilder java objects that we use in our execute and copyToStringfunctions.



On to testing! Save that file and restart the integration agent. Note that any changes to files in the integration agent require a restart. Alternatively, you can issue the reload <service> command, but in the past there have been issues with this and just to be safe, I recycle the whole thing. In larger deployments, this integration agent might serve several integrations that we might not want to impact, so you can reload one or all the services and keep the integration agent daemon or service running.


Anyhoo, after the integration agent comes back up, inject another event with our APClient.bin command:

[ovens@theoven1 bin]$ ./APClient.bin --map-data 'applications|ovens' '[{"targetName":"chef"}]' "oven1" "10:34 AM" "11:04 AM"
<?xml version="1.0" encoding="UTF-8"?>
<transaction id="1"><header><method>Agent</method><subclass>OK</subclass></header><data/></transaction>
[ovens@theoven1 bin]$ 


Just a reminder, since we are still building the integration, we are just simulating events. At some point, we’d tell the oven to make these commands, but being able to simulate stuff from the management system makes for debugging the flow of information from one place to another much easier.
Once we get the email, we can respond and see what our integration agent logs say.

Clicking the link gives me the nice and friendly message from xMatters

And inspecting the IAHOME/log/AlarmPoint.txt file shows success!!

2015-02-11 08:17:45,435 [applications|ovens-1] INFO - Received message from xMatters:
Incident: INCIDENT_ID-901650
Event Id: 901650
Callback Type: response
{"apia_priority":"normal","eventidentifier":"901650","date":"15-02-11 08:17:32.581","agent_application_id":"ip-172-31-28-212/","response":"150","apia_process_group":"apia_default_group","apia_source":"alarmpoint: ","device":"Work Email","agent_client_id":"applications|ovens","additionalTokens":{},"annotation":"null","incident_id":"INCIDENT_ID-901650","recipient":"chef","xmatters_callback_type":"response"}
2015-02-11 08:17:45,436 [applications|ovens-1] DEBUG - cmd: /etc/smartovenstm/,150
2015-02-11 08:17:45,453 [applications|ovens-1] INFO - SmartOvenTM result: temp changed to 150


Note that we see the “RAW” payload in JSON format, complete with all the items we need. Then, because I have DEBUG enabled on the integration agent, I see the command sent to the oven on the line with DEBUG - cmd: .... And after that, I see the output from the oven with the updated temperature.


So this completes our closed loop integration. We've instructed the ovens to fire events into xMatters, then we process that information an notify the appropriate people (that was part 6), then when a user responds, we can take action on the system that initiated the action. 


File Description
execute.txt execute and copyToString functions. To be copied into ovens.js REB communication plan with responses


Integration service with all changes made in part 6 and part 7 (this post)


Click here for the next installment of the GIG!

Have more questions? Submit a request


  • 0
    Cameron Stewart

    Travis, this is another great article in this series - thanks!

Please sign in to leave a comment.
Powered by Zendesk