Grafana Alerting. Sound + Polyglot Voice with AWS Polly
Once you go. There’s no going back.
Grafana has long been the de facto goto in the IT industry when it comes to creating easy to use, intuitive and beautiful dashboards that also supports a wide range of datasources from custom and third-party vendors.
Recently I’ve come across an interesting question in which an user wanted to add a sound alerting to his Grafana setup. Interestingly enough I’ve decided to give it a go on this idea. There has actually been some discussion about his topic here and here. The latter being the one that most inspired this project.
So. What’s the catch? The goal is for each alert to have a sound+ international voice describing the issue as depicted in the diagram below:
From the diagram one can notice that we need at least two tabs, a server to receive, store and retrieve the alerts and AWS Polly to render each alert message.
From here on. We’re going to deep dive into every step required to accomplish this:
1. Grafana alert push webhook
2. Backend build
3. Frontend build
4. AWS Polly
Grafana alert push webhook
So firs we setup a notification channel of type webhook.
And this is what we receive after hitting the “Send Test” button.
For the purpose of this project we’ll only need the “message” but one can see there’s plenty of other information to add more intelligence to one’s project.
For backend we’d need two routes:
- /recv : Receive and store alerts from Grafana
- /send : Return alert list from our frontend
As our backend is fairly simple one can use the backend stack one’s most familiar with. I chose Falcon web framework written in python as it’s resulting code looks clear and concise.
# Run command
gunicorn app:api --workers=2
For our frontend part we just need to regularly fetch out backend for new alerts. If there are any new alerts, play the radar sound and then the voice messages.
For the final part. We’d first need a way to detect which language the message is about and then play it as AWS Polly offers a number of languages just off-the-shelf.
For language detection there’s a pretty neat python package called langdetect that suits our needs.
from langdetect import detect_langs
cases = [
"O servidor B está com consumo CPU acima de 70% por mais de 5 minutos",
"The server C is above 70% CPU for more than 10 minutes",
"A query do servidor Z está demorando muito",
"A query do servidor Z está tomando timeout",
"服务器 A CPU 利用率高",
"informaciones captadas automáticamente por SismoWeb y acciona el disparo de alertas por e-mail o SMS siempre",
"Envío de alertas por email o SMS en caso de alarmas o alertas de diagnósticos",
for case in cases:
The output looks accurate:
O servidor B está com consumo CPU acima de 70% por mais de 5 minutos
The server C is above 70% CPU for more than 10 minutes
A query do servidor Z está demorando muito
A query do servidor Z está tomando timeout
服务器 A CPU 利用率高
informaciones captadas automáticamente por SismoWeb y acciona el disparo de alertas por e-mail o SMS siempre
Envío de alertas por email o SMS en caso de alarmas o alertas de diagnósticos
Voice Play Endpoint
To let AWS Polly play the audio one can proxy it through AWS API Gateway.
To use this function one can issue a GET request to:
So now we have our setup with one tab (our frontend) receiving the alerts proxied by our backend. After the frontend fetches the alert list it then plays an alerting sound followed by the message voices.