We used the instructions on GitHub https://github.com/xmatters/xm-labs-sentry to set up the Sentry integration. I am not familiar with Sentry at all and do not have access to it, so am relying on our Sentry admin. She has created several test alerts and most of them have been received by xMatters. I can see the payloads. A couple of alerts were not received. I do not know exactly what she is doing and do not know why some alerts are not received. For the ones that make it to xMatters, the payloads do not have all of the fields the xMatters script is looking for. One alert had a huge payload. The header did not include the name 'sentry-hook-resource' so there was nothing in the output fields. Another alert did include 'sentry-hook-resource' but the value was 'issue'. The script is looking for 'event_alert'. The payload was much smaller and did not include most of the fields the script is looking for.
So is this a problem with the Sentry set up? Is she not creating the right type of alert? Does the xMatters script need to be modified because it is outdated? She said Sentry has changed a bit from the time the documentation was written, and their webhook is now considered legacy.
Comments
Please sign in to leave a comment.
Hi Julia, some of the links to the Sentry documentation in the GitHub instructions are outdated - I'll see about getting them updated.
My first question is what types of events in Sentry do you want to trigger alerts in xMatters? There are a number of payloads Sentry can send, including, event_alert, issue, and metric_alert. You're right that the Inbound from Sentry trigger only supports event alerts (though we do have another trigger for metric alerts).
The alert with a huge payload sounds like an event_alert, though I don't know why it's not sending the 'sentry-hook-resource' header if that's the case. Looking at the structure of the event_alert payload in the Sentry docs - https://docs.sentry.io/product/integrations/integration-platform/webhooks/#payload-1 - does that look like what you're receiving?
Can you confirm if your Sentry admin has set up an Internal Integration - https://docs.sentry.io/product/integrations/integration-platform/#webhooks - and that the integration is added to any alert rules that should trigger a notification in xMatters?
However, it sounds like you're also getting an 'Issue' payload, which isn't supported by the github integration out of the box. If you want to alert on issues, you'd need to create a copy of the Inbound from Sentry trigger and edit the outputs and script to provide the information you need in the xMatters alert. For example, removing the URLs (since they're not available in the issue payload) and changing
to
And replacing 'timestamp' with 'firstSeen' or 'lastSeen', etc.
Thank you for your response. The large payload I received does not look like the payload in the documentation. It looks like what I pasted below. I have forwarded your response to our Sentry admin so she can verify her setup.
{"id": "2553751323",
"project": "hl-covid-service",
"project_name": "hl-covid-service",
"project_slug": "hl-covid-service",
"logger": null,
"level": "error",
"culprit": "raven.scripts.runner in main",
"message": "This is an example Python exception",
"url": "https://sentry.io/organizations/salt-lake-county/issues/2553751323/?referrer=webhooks_plugin",
"triggering_rules": [],
"event": {
"event_id": "be036aa378fe4022802116a46ba51f9a",
"level": "error",
"version": "5",
"type": "default",
"logentry": {
"formatted": "This is an example Python exception",
"message": null,
"params": null
},
"logger": "",
"modules": {
"my.package": "1.0.0"
},
"platform": "python",
"timestamp": 1628092579.967,
"received": 1628092639.969074,
"environment": "prod",
"user": {
"id": "1",
"email": "sentry@example.com",
"ip_address": "127.0.0.1",
"username": "sentry",
"name": "Sentry",
"geo": {
"country_code": "AU",
"city": "Melbourne",
"region": "VIC"
}
},
"request": {
"url": "http://example.com/foo",
"method": "GET",
"data": {
"hello": "world"
},
"query_string": [
[
"foo",
"bar"
]
],
"cookies": [
[
"foo",
"bar"
],
[
"biz",
"baz"
]
],
"headers": [
[
"Content-Type",
"application/json"
],
[
"Referer",
"http://example.com"
],
[
"User-Agent",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36"
]
],
"env": {
"ENV": "prod"
},
"inferred_content_type": "application/json",
"fragment": null
},
"contexts": {
"browser": {
"name": "Chrome",
"version": "28.0.1500",
"type": "browser"
},
"client_os": {
"name": "Windows",
"version": "8",
"type": "os"
}
},
"stacktrace": {
"frames": [
{
"function": "build_msg",
"module": "raven.base",
"filename": "raven/base.py",
"abs_path": "/home/ubuntu/.virtualenvs/getsentry/src/raven/raven/base.py",
"lineno": 303,
"pre_context": [
" frames = stack",
"",
" data.update({",
" 'sentry.interfaces.Stacktrace': {",
" 'frames': get_stack_info(frames,"
],
"context_line": " transformer=self.transform)",
"post_context": [
" },",
" })",
"",
" if 'sentry.interfaces.Stacktrace' in data:",
" if self.include_paths:"
],
"in_app": false,
"vars": {
"'culprit'": null,
"'data'": {
"'message'": "u'This is a test message generated using ``raven test``'",
"'sentry.interfaces.Message'": {
"'message'": "u'This is a test message generated using ``raven test``'",
"'params'": []
}
},
"'date'": "datetime.datetime(2013, 8, 13, 3, 8, 24, 880386)",
"'event_id'": "'54a322436e1b47b88e239b78998ae742'",
"'event_type'": "'raven.events.Message'",
"'extra'": {
"'go_deeper'": [
[
"{\"'bar'\":[\"'baz'\"],\"'foo'\":\"'bar'\"}"
]
],
"'loadavg'": [
0.37255859375,
0.5341796875,
0.62939453125
],
"'user'": "'dcramer'"
},
"'frames'": "<generator object iter_stack_frames at 0x107bcc3c0>",
"'handler'": "<raven.events.Message object at 0x107bd0890>",
"'k'": "'sentry.interfaces.Message'",
"'kwargs'": {
"'level'": 20,
"'message'": "'This is a test message generated using ``raven test``'"
},
"'public_key'": null,
"'result'": {
"'message'": "u'This is a test message generated using ``raven test``'",
"'sentry.interfaces.Message'": {
"'message'": "u'This is a test message generated using ``raven test``'",
"'params'": []
}
},
"'self'": "<raven.base.Client object at 0x107bb8210>",
"'stack'": true,
"'tags'": null,
"'time_spent'": null,
"'v'": {
"'message'": "u'This is a test message generated using ``raven test``'",
"'params'": []
}
},
"colno": null,
"data": null,
"errors": null,
"raw_function": null,
"image_addr": null,
"instruction_addr": null,
"addr_mode": null,
"package": null,
"platform": null,
"symbol": null,
"symbol_addr": null,
"trust": null,
"snapshot": null
},
{
"function": "capture",
"module": "raven.base",
"filename": "raven/base.py",
"abs_path": "/home/ubuntu/.virtualenvs/getsentry/src/raven/raven/base.py",
"lineno": 459,
"pre_context": [
" if not self.is_enabled():",
" return",
"",
" data = self.build_msg(",
" event_type, data, date, time_spent, extra, stack, tags=tags,"
],
"context_line": " **kwargs)",
"post_context": [
"",
" self.send(**data)",
"",
" return (data.get('event_id'),)",
""
],
"in_app": false,
"vars": {
"'data'": null,
"'date'": null,
"'event_type'": "'raven.events.Message'",
"'extra'": {
"'go_deeper'": [
[
"{\"'bar'\":[\"'baz'\"],\"'foo'\":\"'bar'\"}"
]
],
"'loadavg'": [
0.37255859375,
0.5341796875,
0.62939453125
],
"'user'": "'dcramer'"
},
"'kwargs'": {
"'level'": 20,
"'message'": "'This is a test message generated using ``raven test``'"
},
"'self'": "<raven.base.Client object at 0x107bb8210>",
"'stack'": true,
"'tags'": null,
"'time_spent'": null
},
"colno": null,
"data": null,
"errors": null,
"raw_function": null,
"image_addr": null,
"instruction_addr": null,
"addr_mode": null,
"package": null,
"platform": null,
"symbol": null,
"symbol_addr": null,
"trust": null,
"snapshot": null
},
{
"function": "captureMessage",
"module": "raven.base",
"filename": "raven/base.py",
"abs_path": "/home/ubuntu/.virtualenvs/getsentry/src/raven/raven/base.py",
"lineno": 577,
"pre_context": [
" \"\"\"",
" Creates an event from ``message``.",
"",
" >>> client.captureMessage('My event just happened!')",
" \"\"\""
],
"context_line": " return self.capture('raven.events.Message', message=message, **kwargs)",
"post_context": [
"",
" def captureException(self, exc_info=None, **kwargs):",
" \"\"\"",
" Creates an event from an exception.",
""
],
"in_app": false,
"vars": {
"'kwargs'": {
"'data'": null,
"'extra'": {
"'go_deeper'": [
"[{\"'bar'\":[\"'baz'\"],\"'foo'\":\"'bar'\"}]"
],
"'loadavg'": [
0.37255859375,
0.5341796875,
0.62939453125
],
"'user'": "'dcramer'"
},
"'level'": 20,
"'stack'": true,
"'tags'": null
},
"'message'": "'This is a test message generated using ``raven test``'",
"'self'": "<raven.base.Client object at 0x107bb8210>"
},
"colno": null,
"data": null,
"errors": null,
"raw_function": null,
"image_addr": null,
"instruction_addr": null,
"addr_mode": null,
"package": null,
"platform": null,
"symbol": null,
"symbol_addr": null,
"trust": null,
"snapshot": null
},
{
"function": "send_test_message",
"module": "raven.scripts.runner",
"filename": "raven/scripts/runner.py",
"abs_path": "/home/ubuntu/.virtualenvs/getsentry/src/raven/raven/scripts/runner.py",
"lineno": 77,
"pre_context": [
" level=logging.INFO,",
" stack=True,",
" tags=options.get('tags', {}),",
" extra={",
" 'user': get_uid(),"
],
"context_line": " 'loadavg': get_loadavg(),",
"post_context": [
" },",
" ))",
"",
" if client.state.did_fail():",
" print('error!')"
],
"in_app": false,
"vars": {
"'client'": "<raven.base.Client object at 0x107bb8210>",
"'data'": null,
"'k'": "'secret_key'",
"'options'": {
"'data'": null,
"'tags'": null
}
},
"colno": null,
"data": null,
"errors": null,
"raw_function": null,
"image_addr": null,
"instruction_addr": null,
"addr_mode": null,
"package": null,
"platform": null,
"symbol": null,
"symbol_addr": null,
"trust": null,
"snapshot": null
},
{
"function": "main",
"module": "raven.scripts.runner",
"filename": "raven/scripts/runner.py",
"abs_path": "/home/ubuntu/.virtualenvs/getsentry/src/raven/raven/scripts/runner.py",
"lineno": 112,
"pre_context": [
" print(\"Using DSN configuration:\")",
" print(\" \", dsn)",
" print()",
"",
" client = Client(dsn, include_paths=['raven'])"
],
"context_line": " send_test_message(client, opts.__dict__)",
"in_app": false,
"vars": {
"'args'": [
"'test'",
"'https://ebc35f33e151401f9deac549978bda11:f3403f81e12e4c24942d505f086b2cad@sentry.io/1'"
],
"'client'": "<raven.base.Client object at 0x107bb8210>",
"'dsn'": "'https://ebc35f33e151401f9deac549978bda11:f3403f81e12e4c24942d505f086b2cad@sentry.io/1'",
"'opts'": "<Values at 0x107ba3b00: {'data': None, 'tags': None}>",
"'parser'": "<optparse.OptionParser instance at 0x107ba3368>",
"'root'": "<logging.Logger object at 0x107ba5b10>"
},
"colno": null,
"data": null,
"errors": null,
"raw_function": null,
"image_addr": null,
"instruction_addr": null,
"addr_mode": null,
"package": null,
"platform": null,
"post_context": null,
"symbol": null,
"symbol_addr": null,
"trust": null,
"snapshot": null
}
]
},
"tags": [
[
"browser",
"Chrome 28.0.1500"
],
[
"browser.name",
"Chrome"
],
[
"client_os",
"Windows 8"
],
[
"client_os.name",
"Windows"
],
[
"environment",
"prod"
],
[
"level",
"error"
],
[
"sentry:user",
"id:1"
],
[
"server_name",
"web01.example.org"
],
[
"url",
"http://example.com/foo"
]
],
"extra": {
"emptyList": [],
"emptyMap": {},
"length": 10837790,
"results": [
1,
2,
3,
4,
5
],
"session": {
"foo": "bar"
},
"unauthorized": false,
"url": "http://example.org/foo/bar/"
},
"fingerprint": [
"{{ default }}"
],
"hashes": [
"3a2b45089d0211943e5a6645fb4cea3f"
],
"culprit": "raven.scripts.runner in main",
"metadata": {
"title": "This is an example Python exception"
},
"title": "This is an example Python exception",
"location": null,
"_ref": 5602212,
"_ref_version": 2,
"_metrics": {
"bytes.stored.event": 8038
},
"id": "be036aa378fe4022802116a46ba51f9a"
}
}
I think she was using the wrong token for that payload. She sent me a new token. But still some alerts are not being received. When verifying the setup, she noticed 2 check boxes in the webhooks section. She can check the issue box but not the error box. It says "Your organization does not have access to the error subscription resource." Would this cause event_alert to not be received? Why would we not have access to that resource?
Digging into their documentation, I think I can at least see why you're getting the message around the "error subscription resource". It sounds like a licensing issue. On the Webhooks page, in the Error section near the bottom, it mentions that's only available for Business plans and above. I'll see if I can get you some more information on why it's not sending event alerts.
Our Sentry admin thinks they need to be on the business plan in order for this integration to work but that doesn't make sense to me. The script is looking for “event_alert” in the Sentry-Hook-Resource field. This is different from “error” (the type of alert available for business plans) so I don’t think upgrading is necessary. I don’t know why the integration only includes event_alert unless that should be available for all types of plans. There must be some other reason why event_alert is not getting created and sent to xMatters.
Anyway, they are on the team plan so metric alerts should work. I found this documentation about metric alerts https://help.xmatters.com/ondemand/xmodwelcome/flowdesigner/sentry-steps.htm so I will work on that type of alert and hope it's what they want.