Grafana webhooks for custom alerts

• Grafana alerts can be dispatched to many in-built notification channels
• But in some cases, Grafana might not support the required destination to deliver alert notifications
• To support such custom alerts, webhooks can be developed to deliver alerts to desired destination

How Grafana webhook notifier works

• A webhook can be created as a contact point in Grafana and an alert rule can be configured to use the webhook to deliver notifications
• A webhook sends the alert information as a JSON to a configured URL over HTTP
• A sample alert data from Grafana webhook is shown below

{
    "receiver": "test webhook",
    "status": "firing",
    "alerts": [
        {
            "status": "firing",
            "labels": {
                "alertname": "test",
                "grafana_folder": "test"
            },
            "annotations": {},
            "startsAt": "2024-10-14T16:36:10+05:30",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "<http://localhost:3000/alerting/grafana/de0fg38xxe70gd/view?orgId=1>",
            "fingerprint": "75329ee086bfb97f",
            "silenceURL": "<http://localhost:3000/alerting/silence/new?alertmanager=grafana&matcher=alertname%3Dtest&matcher=grafana_folder%3Dtest&orgId=1>",
            "dashboardURL": "",
            "panelURL": "",
            "values": {
                "B": 38.19475553557968,
                "C": 1
            },
            "valueString": "[ var='B' labels={__name__=A-series} value=38.19475553557968 ], [ var='C' labels={__name__=A-series} value=1 ]"
        }
    ],
    "groupLabels": {
        "alertname": "test",
        "grafana_folder": "test"
    },
    "commonLabels": {
        "alertname": "test",
        "grafana_folder": "test"
    },
    "commonAnnotations": {},
    "externalURL": "<http://localhost:3000/>",
    "version": "1",
    "groupKey": "{}/{__grafana_autogenerated__=\\"true\\"}/{__grafana_receiver__=\\"test webhook\\"}:{alertname=\\"test\\", grafana_folder=\\"test\\"}",
    "truncatedAlerts": 0,
    "orgId": 1,
    "title": "[FIRING:1] test test ",
    "state": "alerting",
    "message": "**Firing**\\n\\nValue: B=38.19475553557968, C=1\\nLabels:\\n - alertname = test\\n - grafana_folder = test\\nAnnotations:\\nSource: <http://localhost:3000/alerting/grafana/de0fg38xxe70gd/view?orgId=1\\nSilence:> <http://localhost:3000/alerting/silence/new?alertmanager=grafana&matcher=alertname%3Dtest&matcher=grafana_folder%3Dtest&orgId=1\\n>"
}

Create a webhook in Grafana

• As shown in the above image, webhook can be crated in the Alerting→Contact Points section
• The URL of API server and other required settings are required to be mentioned during webhook creation
• Provision for basic authentication, authorization header, custom message title and message are also present

Sample API server

• An API server can be developed to process alert information sent from Grafana webhook
• Grafana webhook will call the API server to deliver alert data
• The API server can use the alert information from JSON body of HTTP request and deliver the alert to a custom alert destination like SMS gateway, custom database etc.
• A sample API server developed in python Flask for delivering alert data from webhooks to alert destinations like SMS gateway, database, message queue etc.

from flask import Flask, request
import json

# create a flask server
app = Flask(__name__)

@app.route('/alert', methods=["POST"])
def alert():
    alertData = request.json

    """
    # dump alert data to a file for debugging purposes
    out_file = open("test.json", "w")
    json.dump(alertData,out_file,indent=4)
    out_file.close()
    """
    # use alert data to deliver notification
    alertMsg = alertData["message"]
    # send alert data to desired destination like an sms gateway, database etc.
    print(alertMsg)
    return 'ok'

@app.route('/')
def index():
    return 'Hello, World!'

# __name__ will be __main__ only if this file is the entry point
if __name__ == '__main__':
    # run the server on this ip and port 50100
    app.run(host='0.0.0.0', port=50100, debug=True)

Video

Video on this post can be seen here

References

• Official documentation on Grafana webhooks - https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/