Grafana Alerting. Sound + Polyglot Voice with AWS Polly

4 min readDec 1, 2020

Once you go. There’s no going back.

Source: https://syshead.files.wordpress.com/2015/09/unzopvn.jpg

Grafana has long been the de facto goto in the IT industry when it comes to creating easy to use, intuitive and beautiful dashboards that also supports a wide range of datasources from custom and third-party vendors.

Recently I’ve come across an interesting question in which an user wanted to add a sound alerting to his Grafana setup. Interestingly enough I’ve decided to give it a go on this idea. There has actually been some discussion about his topic here and here. The latter being the one that most inspired this project.

So. What’s the catch? The goal is for each alert to have a sound+ international voice describing the issue as depicted in the diagram below:

From the diagram one can notice that we need at least two tabs, a server to receive, store and retrieve the alerts and AWS Polly to render each alert message.

From here on. We’re going to deep dive into every step required to accomplish this:

1. Grafana alert push webhook
2. Backend build
3. Frontend build
4. AWS Polly

Grafana alert push webhook

So firs we setup a notification channel of type webhook.

And this is what we receive after hitting the “Send Test” button.

For the purpose of this project we’ll only need the “message” but one can see there’s plenty of other information to add more intelligence to one’s project.

Backend Build

For backend we’d need two routes:

/recv : Receive and store alerts from Grafana
/send : Return alert list from our frontend

As our backend is fairly simple one can use the backend stack one’s most familiar with. I chose Falcon web framework written in python as it’s resulting code looks clear and concise.

# Run command
gunicorn app:api --workers=2

Frontend Build

For our frontend part we just need to regularly fetch out backend for new alerts. If there are any new alerts, play the radar sound and then the voice messages.

AWS Polly

For the final part. We’d first need a way to detect which language the message is about and then play it as AWS Polly offers a number of languages just off-the-shelf.

Language detection

For language detection there’s a pretty neat python package called langdetect that suits our needs.

from langdetect import detect_langs


cases = [
    "O servidor B está com consumo CPU acima de 70% por mais de 5 minutos",
    "The server C is above 70% CPU for more than 10 minutes",
    "A query do servidor Z está demorando muito",
    "A query do servidor Z está tomando timeout",
    "服务器 A CPU 利用率高",
    "informaciones captadas automáticamente por SismoWeb y acciona el disparo de alertas por e-mail o SMS siempre",
    "Envío de alertas por email o SMS en caso de alarmas o alertas de diagnósticos",
]

for case in cases:
    print(case)
    print(detect_langs(case))
    print("====")
    print()

The output looks accurate:

O servidor B está com consumo CPU acima de 70% por mais de 5 minutos
[pt:0.9999977577732495]
====

The server C is above 70% CPU for more than 10 minutes
[en:0.999995532419058]
====

A query do servidor Z está demorando muito
[pt:0.999995959469694]
====

A query do servidor Z está tomando timeout
[pt:0.9999971518611472]
====

服务器 A CPU 利用率高
[zh-cn:0.9999980430575592]
====

informaciones captadas automáticamente por SismoWeb y acciona el disparo de alertas por e-mail o SMS siempre
[es:0.9999955938227942]
====

Envío de alertas por email o SMS en caso de alarmas o alertas de diagnósticos
[es:0.9999952138292143]
====

Voice Play Endpoint

To let AWS Polly play the audio one can proxy it through AWS API Gateway.

To use this function one can issue a GET request to:

https://api-gateway-endpoints/<resource>?lang=en-US&message=Problem detected

Wrapping up

So now we have our setup with one tab (our frontend) receiving the alerts proxied by our backend. After the frontend fetches the alert list it then plays an alerting sound followed by the message voices.