1. xMatters
  2. Community
  3. Integrations & Toolchains

Prometheus integration problem

Pauric Coyle
Not Yet Reviewed

Im trying to integrate prometheus with xmatters but for some reason I cant seem to get alertmanager to fire alerts to xmatters. im following "https://github.com/xmatters/xm-labs-prometheus". attached is the alertmanager.yml and the prometheus.yml. I need to get all rules to fire to xmatters that i have in place, I have used the url of "https://pfs.xmatters.com/reapi/2015-04-01/forms/7a1d743c-c4a5-4588-b314-0146cb203804/triggers" in alert manager to trigger my alert but im not sure if this is the correct one. 

0

Comments

13 comments
Date Votes

Please sign in to leave a comment.

  • Daniel Thomas

    Hi Pauric

    Review the URL you are trying to use as that is incorrect. The section from the help page is here :

    1. Open the alertmanager.yml file and navigate to the receivers section. The location of the file and the section will depend on the details of the installation.
    2. Add a new receiver. The name of the receiver will be the recipients of the event. The webhook url is found in the Inbound from Alertmanager step in your xMatters workflow. For example, to target the Database group:
    - name: 'Database'
  webhook_configs:
    - url: 'https://acme.xmatters.com/api/integration/1/functions/UUID/triggers?apiKey=KEY'

    You want the inbound integration url, details on finding that can be found here : https://help.xmatters.com/ondemand/xmodwelcome/integrationbuilder/generate-urls.htm

    Let me know if you continue to have issues finding the correct URL.

    Thanks

     

     

     

     

     

     

     

     

     

    0
  • Pauric Coyle

    • Hi Thomas,
      1. my recievers section is as follows :
      receivers:
      - name: 'Database' webhook_configs:
      - url: 'https://acme.xmatters.com/api/integration/1/functions/UUID/triggers?apiKey=KEY'

      The problem here is that the url seems to be incorrect, what section do i find this url on is it under the config on the server or where exactly in xmatters do I find it ?
      if i click the link you sent me it brings me directly to the home page, is this api key connected to a user directly or the entire xmatters app ?

    0
  • Pauric Coyle

    I cant find the api KEY 

    0
  • Pauric Coyle

    in inbound for alert manager 

    0
  • Pauric Coyle

    Thanks Don I have the api key now but not recieveing any alerts via xmatters, I have not but in the following part : 

     routes:
  - match_re:
      service: ^(octoapp)$
    receiver: Database

    as I want alertmanager to fire alerts for the current rules we have in place, can you assist me on this   

    0
  • Travis Depuy

    Hey Pauric,

       That routes entry you have will match on a service that is called exactly "octoapp". What is the name of the service you want to alert on? You will need to adjust the regular expression to match your service. Have a look at the route element in the alertmanager config here. 

    Happy Thursday!

    0
  • Pauric Coyle

    Hi Travis, 

    Thanks very much for your reply, its not a specific service i wish to alert on its all the services that are define in alert manager that i wish to alert with, for example there are rules set in Prometheus that alert manager will trigger, i wish to have these rules trigger for xmatters. I will also check out the flow designer now, Thanks again

    0
  • Travis Depuy

    Hmm, Yea, that shows the xMatters side was never triggered. 

    So, first question. Do you see an alert in Alertmanager? That indicates the alert rules are successfully firing. If you don't see an alert, then go back to the prometheus.yaml file and investigate your alert rules and whatever gauges or metrics that are tied to your services have the expected values. 

    If you do see an alert in Alertmanager, then the alert rules are properly set, but the receiver or routes are incorrect, so you will want to check those out. 

    The installation guide on github has some examples, but here are a couple extra examples in case they help. This is monitoring a service called "octoapp", specifically the "some_guage" guage and if it is >20 then trigger alertmanager. (Also fires at 55, I'm not sure why...). The "recipient: "Prometheus Alerts"" references the "receiver" in the alertmanager.yml.

    alertmanager.yml:

    global:
      resolve_timeout: 5m

    route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 2m 
      receiver: 'Prometheus Alerts'
    receivers:
    - name: 'Prometheus Alerts'
      webhook_configs:
              - url: 'HTTP_TRIGGER_URL_HERE'

    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname', 'dev', 'instance']

     

    alert.rules.yml

    groups:
    - name: alert.rules
      rules:
      - alert: octo_alert
        expr: some_gauge > 30
        for: 20s
        labels:
          service: octoapp
          severity: page_octo
        annotations:
          description: The description goes here
          summary: The summary goes here
          recipient: "Prometheus Alerts"
      - alert: octo_alert
        expr: some_gauge > 55 
        for: 20s
        labels:
          service: octoapp
          severity: page_octo
          instance: alert2
        annotations:
          description: asdf
          summary: "summary goes {{ $labels.instance }}" 
          recipient: "Prometheus Alerts"

     

    I hope that helps? I know a bit about the alerting rules in Prometheus, but the rest is rather a mystery. 

    Let us know how it goes!

    0
  • Travis Depuy

    Hey Paul,

       So I would recommend editing your post as there are some API keys in there you don't want to expose. Same with the slack webhooks. 

    The screen shots of the activity stream show a "Content-Type" header of "x-www-form-urlencoded" instead of "application/json" which is why it is erroring. But it sounds like this was just a curl test, so we can ignore them. 

    The formatting on your alertmanager.yaml file got mangled. I tried to put it through the prettify yaml, but it fails on one of the "send_resolved" entries. I don't really know much about yaml, so I'm not sure where to help here. If alertmanager takes it with no errors, then maybe it's just an issue with the copy/paste. 

    What ever you have in the "name" in your alertmanager.yaml you will need to reference in your alert rules section. So I would say, replace "Appdynamics Machine Agent" with "xMatters Alerts", then make sure "xMatters Alerts" is referenced in each of your alert rule files for each alert you want to fire to xMatters. 

    In my alert.rules file, I'm using "Prometheus Alerts", which you can see referenced in the "receiver" value in my alert.rules.yaml file:

      - alert: octo_alert
        expr: some_gauge > 55 
        for: 20s
        labels:
          service: octoapp
          severity: page_octo
          instance: alert2
        annotations:
          description: asdf
          summary: "summary goes {{ $labels.instance }}" 
          recipient: "Prometheus Alerts"

     

    So I would take a look at your rule files and verify they are set the way you want. Also, check with the alertmanager documentation, here for details on the formats of these files. 

    I'm swamped this week, but if you don't get it resolved by next week I can see about doing a call. 

    Happy Monday!

     

    0
  • Pauric Coyle

    Hey Travis good to hear from you again. 

    I have edited the api keys didn't realise I left them in there, they are now removed. 

    yes correct this was just a curl test to confirm connectivity from my instance to xmatters which has worked as we cna see in the previous activity stream. 

    The .yml file is working fine it is alerting the correct endpoint for example teams and slack the issue is with the following part of alertmanager.yml : 

    - name : 'Appdynamics Machine Agent'
    webhook_configs:
    - url: 'https://pfs.xmatters.com/api/integration/1/functions/360a51c3-a7f3-4bf8-8e65-f70b06d9f3f1/triggers?apiKey=881a366a-17cb1bff7' (erased part of the api key for privacy purposed)

    I have now changed the file to : 

    - name : 'Xmatters Alerts'
    webhook_configs:
    - url: 'https://pfs.xmatters.com/api/integration/1/functions/360a51c3-a7f3-4bf8-8e65-f70b06d9f3f1/triggers?apiKey=881a366a-17cb-47e7-9b44-a0c76bb' (I have erased part of api keys)

    In relation to the rule files they are set in alert manager and are define in Prometheus each rule file is dependent on a application for example there is a rule file from windows and rule file for linux but insde them rule file they are not writting to Teams or slack this is done on the alertmanager.yml where it is refernecing slack and teams. so what i need to do somehow is reference these rule files to point to xmatters some how. 

     

     

    0
  • Pauric Coyle

    Hey Guys can i arrange a call in relation to this?

    0
  • Travis Depuy

    Hey Pauric. I reached out via email. 

    0
  • Eric Maxwell

    Hey Pauric I read through your posts and from what I have gleaned of your configuration I think I might know the issue.

    Basically, it looks like in your alertmanager.yml config you have defined the xMatters webhook in a different receiver than the Slack and Email.  This receiver (Xmatters Alerts) is not ever referenced in your route section.

    From what I can tell you have two options.

    1. Move the xMatters webhook url into the webhook_configs of the pfs-hooks receiver.  I grabbed what you have posted and I believe the below should work for you, but the main point is moving the xMatters webhook URL into the pfs-hooks webhook_configs.  Make sure to update the xMatters URL with yours. The yaml formatting maybe a little off after posting in the forum.

      global:
        resolve_timeout: 5m
        slack_api_url: 'dfjs'

      route:
        group_by: ['alertname']
        group_wait: 20s
        group_interval: 5m
        repeat_interval: 12h
        receiver: pfs-hook

      receivers:
        - name: pfs-hook
          email_configs:
          - to: 'alerts@om'
            send_resolved: true
            from: 'i'
            smarthost: 'mail.smtp2go.com:25'
          webhook_configs:
          - url: 'http:/st:2000/alertmanager'
            send_resolved: true
          - url: 'http://localhosttmanager'
            send_resolved: true
          - url: 'https://<your_instance_name>.xmatters.com/api/integration/1/functions/<endpoint_guid>/triggers?apiKey=<api_key>'
            send_resolved: true
          slack_configs:
          - api_url: 'httpi6ZLMRJZRAnCM'
            channel: '#rackspace-alerts'
            send_resolved: true
            text: |-
            {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
            *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
            {{ end }}
    2.  If you want the alerts to xMatters to follow a different criteria than Email and Slack then you will need to expand your route configuration and reference the Xmatters Alerts receiver.
    0

Didn't find what you were looking for?

New post