lupyuen.org/articles/prometheus.html

809 lines
No EOL
40 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>Monitor IoT Devices in The Things Network with Prometheus and Grafana</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="Monitor IoT Devices in The Things Network with Prometheus and Grafana"
data-rh="true">
<meta property="og:description"
content="How we monitor our IoT Sensor Devices connected to The Things Network... With Prometheus Time Series Database and Grafana Dashboards"
data-rh="true">
<meta property="og:image"
content="https://lupyuen.github.io/images/prometheus-title.jpg">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical" href="https://lupyuen.org/articles/prometheus.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">Monitor IoT Devices in The Things Network with Prometheus and Grafana</h1>
<nav id="rustdoc"><ul>
<li><a href="#payload-formatter" title="Payload Formatter">1 Payload Formatter</a><ul>
<li><a href="#checkpoint-alpha" title="Checkpoint Alpha">1.1 Checkpoint Alpha</a><ul></ul></li></ul></li>
<li><a href="#mqtt-gateway-for-prometheus" title="MQTT Gateway for Prometheus">2 MQTT Gateway for Prometheus</a><ul>
<li><a href="#prometheus-metrics" title="Prometheus Metrics">2.1 Prometheus Metrics</a><ul></ul></li>
<li><a href="#start-mqtt-gateway" title="Start MQTT Gateway">2.2 Start MQTT Gateway</a><ul></ul></li>
<li><a href="#checkpoint-bravo" title="Checkpoint Bravo">2.3 Checkpoint Bravo</a><ul></ul></li></ul></li>
<li><a href="#prometheus-time-series-database" title="Prometheus Time Series Database">3 Prometheus Time Series Database</a><ul>
<li><a href="#checkpoint-charlie" title="Checkpoint Charlie">3.1 Checkpoint Charlie</a><ul></ul></li></ul></li>
<li><a href="#grafana-dashboard" title="Grafana Dashboard">4 Grafana Dashboard</a><ul>
<li><a href="#checkpoint-delta" title="Checkpoint Delta">4.1 Checkpoint Delta</a><ul></ul></li></ul></li>
<li><a href="#transform-and-filter-sensor-data" title="Transform and Filter Sensor Data">5 Transform and Filter Sensor Data</a><ul>
<li><a href="#auto-dashboard-refresh" title="Auto Dashboard Refresh">5.1 Auto Dashboard Refresh</a><ul></ul></li></ul></li>
<li><a href="#mqtt-with-tls-encryption" title="MQTT with TLS Encryption">6 MQTT with TLS Encryption</a><ul>
<li><a href="#checkpoint-echo" title="Checkpoint Echo">6.1 Checkpoint Echo</a><ul></ul></li></ul></li>
<li><a href="#sensor-data-alerts" title="Sensor Data Alerts">7 Sensor Data Alerts</a><ul></ul></li>
<li><a href="#whats-next" title="Whats Next">8 Whats Next</a><ul></ul></li>
<li><a href="#notes" title="Notes">9 Notes</a><ul></ul></li>
<li><a href="#appendix-configure-the-things-network-mqtt" title="Appendix: Configure The Things Network MQTT">10 Appendix: Configure The Things Network MQTT</a><ul></ul></li>
<li><a href="#appendix-install-grafana" title="Appendix: Install Grafana">11 Appendix: Install Grafana</a><ul></ul></li></ul></nav><p>📝 <em>21 Oct 2021</em></p>
<p>Suppose we have some <strong>IoT Devices</strong> that transmit <strong>Sensor Data</strong> (via LoRa and LoRaWAN) to <strong>The Things Network</strong></p>
<p><a href="https://lupyuen.github.io/articles/ttn">(Thats the free-to-use public global wireless network for IoT Devices)</a></p>
<blockquote>
<p><img src="https://lupyuen.github.io/images/grafana-flow3.jpg" alt="IoT Devices transmitting Sensor Data to The Things Network" /></p>
</blockquote>
<p><em>How shall we monitor the Sensor Data transmitted by the IoT Devices?</em></p>
<p>Today we shall monitor IoT Sensor Data by connecting open source <strong>Prometheus and Grafana</strong> to The Things Network…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-title.jpg" alt="Monitoring IoT Devices in The Things Network with Prometheus and Grafana" /></p>
<ol>
<li>
<p><strong>The Things Network</strong> pushes our <strong>Sensor Data over MQTT</strong> in real time</p>
</li>
<li>
<p>Our <strong>MQTT Gateway</strong> consumes the Sensor Data…</p>
</li>
<li>
<p>And publishes the Sensor Data to our <strong>Prometheus Time Series Database</strong></p>
</li>
<li>
<p>Which gets rendered as a <strong>Grafana Dashboard</strong> like this…</p>
</li>
</ol>
<p><img src="https://lupyuen.github.io/images/prometheus-grafana4.png" alt="Monitoring Devices on The Things Network with Prometheus and Grafana" /></p>
<p><em>Why Prometheus and Grafana?</em></p>
<p>Prometheus works great for <strong>storing and querying</strong> IoT Sensor Data.</p>
<p>And Grafana works well with Prometheus for <strong>visualising IoT Sensor Data</strong>.</p>
<p><img src="https://lupyuen.github.io/images/ttn-title.jpg" alt="PineDio Stack BL604 RISC-V Board (foreground) talking to The Things Network via RAKWireless RAK7248 LoRaWAN Gateway (background)" /></p>
<p>In a while we shall demo this Prometheus + Grafana Setup with <strong>PineDio Stack BL604 RISC-V Board</strong> (pic above)</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/ttn"><strong>“The Things Network on PineDio Stack BL604 RISC-V Board”</strong></a></li>
</ul>
<p>But it should work for <strong>any LoRaWAN Device</strong> connected to The Things Network… Assuming that we have configured a suitable <strong>Payload Formatter</strong> in The Things Network.</p>
<p>(Read on to learn how)</p>
<p><img src="https://lupyuen.github.io/images/payload-title.jpg" alt="CBOR Payload Formatter for The Things Network" /></p>
<h1 id="payload-formatter"><a class="doc-anchor" href="#payload-formatter">§</a>1 Payload Formatter</h1>
<p><em>Whats a Payload Formatter in The Things Network?</em></p>
<p>A <strong>Payload Formatter</strong> is JavaScript Code that we configure in The Things Network to <strong>decode the Sensor Data</strong> in the LoRaWAN Message Payload.</p>
<p>For PineDio Stack our Sensor Data is encoded with CBOR (Concise Binary Object Representation), so we use this <strong>CBOR Payload Formatter</strong></p>
<ul>
<li><a href="https://lupyuen.github.io/articles/payload"><strong>“CBOR Payload Formatter for The Things Network”</strong></a></li>
</ul>
<p><em>Is it mandatory to use a Payload Formatter?</em></p>
<p>Yes, our MQTT Gateway will work only if we <strong>configure a suitable Payload Formatter</strong> that will decode our Sensor Data.</p>
<p><a href="https://lupyuen.github.io/articles/payload#whats-a-payload-formatter">(More about Payload Formatters)</a></p>
<p><em>What if we cant find a suitable Payload Formatter?</em></p>
<p>We can make one together! <a href="https://www.reddit.com/r/TheThingsNetwork/comments/qafzu4/cbor_payload_formatter_for_the_things_network/?utm_source=share&amp;utm_medium=web2x&amp;context=3">Post a comment here</a></p>
<h2 id="checkpoint-alpha"><a class="doc-anchor" href="#checkpoint-alpha">§</a>1.1 Checkpoint Alpha</h2>
<p>Lets verify that our <strong>Payload Formatter works OK</strong> for decoding our Sensor Data…</p>
<ol>
<li>
<p>Start the <strong>LoRaWAN Firmware</strong> on our LoRaWAN Device (PineDio Stack).</p>
<p>Transmit some Sensor Data every minute…</p>
<p><a href="https://lupyuen.github.io/articles/tsen#run-the-lorawan-firmware"><strong>“Run the LoRaWAN Firmware”</strong></a></p>
</li>
<li>
<p>Log on to <strong>The Things Network Console</strong></p>
</li>
<li>
<p>Click <strong>Applications → (Your Application) → Live Data</strong></p>
</li>
<li>
<p>Our <strong>Decoded Sensor Data</strong> should appear in the Live Data Table like so…</p>
<div class="example-wrap"><pre class="language-json"><code>Payload: { l: 4000, t: 4669 }</code></pre></div>
<p><img src="https://lupyuen.github.io/images/payload-ttn3.png" alt="Decoded Sensor Data in the Live Data Table" /></p>
</li>
<li>
<p>Click on a message in the <strong>Live Data Table</strong>.</p>
<p>We should see the <strong>decoded_payload</strong> field containing our Decoded Sensor Data…</p>
<div class="example-wrap"><pre class="language-json"><code>{
...
&quot;uplink_message&quot;: {
...
&quot;decoded_payload&quot;: {
&quot;l&quot;: 4000,
&quot;t&quot;: 4656
} </code></pre></div>
<p>These are the <strong>Light Sensor</strong> (“<code>l</code>”) and <strong>Temperature Sensor</strong> (“<code>t</code>”) values transmitted by our LoRaWAN Device (PineDio Stack).</p>
<p><a href="https://lupyuen.github.io/articles/cbor#floating-point-numbers">(Our Temperature Values are scaled up 100 times… <code>4656</code> means <code>46.56</code> ºC)</a></p>
</li>
</ol>
<p>Also verify that the <strong>MQTT Server works OK</strong> at The Things Network…</p>
<ol>
<li>
<p>Start our <strong>LoRaWAN Firmware</strong> and transmit Sensor Data every minute</p>
<p><a href="https://lupyuen.github.io/articles/tsen#run-the-lorawan-firmware">(Like this)</a></p>
</li>
<li>
<p>Copy the <strong>MQTT Public Address, Username and Password</strong> from The Things Network…</p>
<p><a href="https://lupyuen.github.io/articles/prometheus#appendix-configure-the-things-network-mqtt"><strong>“Configure The Things Network MQTT”</strong></a></p>
</li>
<li>
<p>Install the <strong>command-line tools for MQTT</strong></p>
<p><a href="https://mosquitto.org/download/"><strong>“Download Eclipse Mosquitto”</strong></a></p>
</li>
<li>
<p>Enter this at the command line…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Change au1.cloud.thethings.network to our
## MQTT Public Address (without the port number)
## Change YOUR_USERNAME to our MQTT Username
## Change YOUR_PASSWORD to our MQTT Password
## For Linux and macOS:
mosquitto_sub \
-h au1.cloud.thethings.network \
-t &quot;#&quot; \
-u &quot;YOUR_USERNAME&quot; \
-P &quot;YOUR_PASSWORD&quot; \
-d
## For Windows:
&quot;c:\Program Files\Mosquitto\mosquitto_sub&quot; ^
-h au1.cloud.thethings.network ^
-t &quot;#&quot; ^
-u &quot;YOUR_USERNAME&quot; ^
-P &quot;YOUR_PASSWORD&quot; ^
-d</code></pre></div></li>
<li>
<p>We should see the <strong>Uplink Messages transmitted</strong> by our LoRaWAN Device…</p>
<div class="example-wrap"><pre class="language-json"><code>{
...
&quot;uplink_message&quot;: {
...
&quot;decoded_payload&quot;: {
&quot;l&quot;: 4000,
&quot;t&quot;: 4656
} </code></pre></div>
<p>Including <strong>decoded_payload</strong> and the Decoded Sensor Data.</p>
<p><a href="https://github.com/lupyuen/cbor-the-things-network#mqtt-log">(See the complete message)</a></p>
</li>
</ol>
<p><img src="https://lupyuen.github.io/images/prometheus-flow2.jpg" alt="MQTT Gateway for Prometheus" /></p>
<h1 id="mqtt-gateway-for-prometheus"><a class="doc-anchor" href="#mqtt-gateway-for-prometheus">§</a>2 MQTT Gateway for Prometheus</h1>
<p>Now we connect our MQTT Gateway to The Things Network…</p>
<ul>
<li><a href="https://github.com/hikhvar/mqtt2prometheus"><strong>MQTT2Prometheus MQTT Gateway</strong></a></li>
</ul>
<p>Our MQTT Gateway shall…</p>
<ul>
<li>
<p><strong>Subscribe to all MQTT Topics</strong> published on The Things Network</p>
<p>(Including the Uplink Messages transmitted by our device)</p>
</li>
<li>
<p><strong>Ingest the Decoded Sensor Data</strong> from the Uplink Messages</p>
<p>(As Prometheus Metrics)</p>
</li>
</ul>
<p>Follow these steps to <strong>configure our MQTT Gateway</strong></p>
<ol>
<li>
<p>Download the <strong>MQTT Gateway Configuration File</strong></p>
<p><a href="https://github.com/lupyuen/prometheus-the-things-network/blob/main/ttn-mqtt.yaml"><strong>ttn-mqtt.yaml</strong></a></p>
</li>
<li>
<p>Edit the file. Fill in the <strong>MQTT Public Address, Username and Password</strong> for The Things Network <a href="https://lupyuen.github.io/articles/prometheus#appendix-configure-the-things-network-mqtt">(from here)</a></p>
<div class="example-wrap"><pre class="language-yaml"><code>## Change au1.cloud.thethings.network to our MQTT Public Address
server: tcp://au1.cloud.thethings.network:1883
## Change luppy-application@ttn to our MQTT Username
user: luppy-application@ttn
## Change YOUR_API_KEY to our MQTT Password
password: YOUR_API_KEY</code></pre></div></li>
<li>
<p>Note that were subscribing to <strong>all MQTT Topics</strong></p>
<div class="example-wrap"><pre class="language-yaml"><code>## Topic path to subscribe to. &quot;#&quot; means All Topics.
topic_path: &quot;#&quot;</code></pre></div></li>
<li>
<p>Our MQTT Gateway shall extract the <strong>Device ID</strong> from the MQTT Topic Path…</p>
<div class="example-wrap"><pre class="language-yaml"><code>## Extract the device ID (eui-YOUR_DEVICE_EUI)
## from the topic path, which looks like...
## v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up
device_id_regex: &quot;(.*/)?devices/(?P&lt;deviceid&gt;.*)/.*&quot;</code></pre></div>
<p>(Which will be helpful for filtering our Sensor Data by Device ID in Grafana)</p>
<p><img src="https://lupyuen.github.io/images/prometheus-config4.png" alt="MQTT Configuration File" /></p>
</li>
</ol>
<h2 id="prometheus-metrics"><a class="doc-anchor" href="#prometheus-metrics">§</a>2.1 Prometheus Metrics</h2>
<p><em>Whats a Prometheus Metric?</em></p>
<p>A Metric is an item of <strong>Monitoring Data</strong> thats collected and reported by Prometheus.</p>
<p>(Think of Metrics like CPU Usage and RAM Utilisation… Prometheus was originally created for monitoring servers)</p>
<p>In this article we shall use “Sensor Data” and “Metric” interchangeably, since Prometheus treats our <strong>Sensor Data as Metrics</strong>.</p>
<p>Lets define the Sensor Data / Metrics that will be <strong>ingested by our MQTT Gateway</strong></p>
<ol>
<li>
<p>Edit <a href="https://github.com/lupyuen/prometheus-the-things-network/blob/main/ttn-mqtt.yaml"><strong>ttn-mqtt.yaml</strong></a></p>
<p>Look for the <strong>metrics</strong> section…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-config5.png" alt="Prometheus Metrics for MQTT Gateway" /></p>
</li>
<li>
<p>We define the <strong>Metric for Temperature</strong> like so…</p>
<div class="example-wrap"><pre class="language-yaml"><code>## Temperature Metric
## Name of the metric in prometheus
- prom_name: t
## JSON Path of the metric in our MQTT JSON message
mqtt_name: &quot;uplink_message.decoded_payload.t&quot;
## Prometheus help text for this metric
help: &quot;Temperature&quot;
## Prometheus type for this metric.
## Valid values are: &quot;gauge&quot; and &quot;counter&quot;
type: gauge
## Map of string to string for constant labels.
## The labels will be attached to every Prometheus metric.
const_labels:
sensor_type: t </code></pre></div></li>
<li>
<p>This tells MQTT Gateway: Our LoRaWAN Device (PineDio Stack) transmits a <strong>Temperature Value</strong> (named “t”) at this JSON Path…</p>
<div class="example-wrap"><pre class="language-javascript"><code>uplink_message.decoded_payload.t</code></pre></div>
<p>Which matches our <strong>JSON Message Format</strong> from Checkpoint Alpha…</p>
<div class="example-wrap"><pre class="language-json"><code>{
...
&quot;uplink_message&quot;: {
...
&quot;decoded_payload&quot;: {
&quot;l&quot;: 4000,
&quot;t&quot;: 4656
} </code></pre></div>
<p><a href="https://lupyuen.github.io/articles/cbor#floating-point-numbers">(Our Temperature Values are scaled up 100 times… <code>4656</code> means <code>46.56</code> ºC)</a></p>
<p><img src="https://lupyuen.github.io/images/prometheus-config7.jpg" alt="Prometheus Metrics for MQTT Gateway" /></p>
</li>
<li>
<p>We define other Metrics the same way, like this <strong>Light Level Metric</strong> thats transmitted by PineDio Stack…</p>
<div class="example-wrap"><pre class="language-yaml"><code>## Light Level Metric
## Name of the metric in prometheus
- prom_name: l
## JSON Path of the metric in our MQTT JSON message
mqtt_name: &quot;uplink_message.decoded_payload.l&quot;
## Prometheus help text for this metric
help: &quot;Light Level&quot;
## Prometheus type for this metric.
## Valid values are: &quot;gauge&quot; and &quot;counter&quot;
type: gauge
## Map of string to string for constant labels.
## The labels will be attached to every Prometheus metric.
const_labels:
sensor_type: l</code></pre></div></li>
</ol>
<h2 id="start-mqtt-gateway"><a class="doc-anchor" href="#start-mqtt-gateway">§</a>2.2 Start MQTT Gateway</h2>
<p>Were ready to <strong>start our MQTT Gateway</strong>!</p>
<p>Follow these steps to download and run <strong>MQTT2Prometheus</strong></p>
<ol>
<li>
<p>Install the <strong>latest version of Go</strong></p>
<p><a href="https://golang.org"><strong><code>golang.org</code></strong></a></p>
</li>
<li>
<p>Enter this at the command line…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Download mqtt2prometheus
go get github.com/hikhvar/mqtt2prometheus
## For Linux and macOS:
cd $GOPATH/src/github.com/hikhvar/mqtt2prometheus
## For Windows:
cd %GOPATH%\src\github.com\hikhvar\mqtt2prometheus
## Build mqtt2prometheus
go build ./cmd
## Run mqtt2prometheus.
## Change &quot;ttn-mqtt.yaml&quot; to the full path of our
## MQTT Gateway Configuration File.
go run ./cmd -log-level debug -config ttn-mqtt.yaml</code></pre></div></li>
<li>
<p>For Windows: Click <strong>“Private Network: Allow Access”</strong> when prompted</p>
<p>(Thats because our MQTT Gateway starts a HTTP Server at port 9641)</p>
</li>
<li>
<p>We should see our MQTT Gateway <strong>ingesting Sensor Data</strong> from The Things Network…</p>
<div class="example-wrap"><pre class="language-text"><code>mqttclient/mqttClient.go:20
Connected to MQTT Broker
mqttclient/mqttClient.go:21
Will subscribe to topic &quot;#&quot;
web/tls_config.go:191
&quot;TLS is disabled.&quot;, &quot;http2&quot;: false
metrics/ingest.go:42
Got message
&quot;topic&quot;: &quot;v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up&quot;, &quot;payload&quot;:
{
...
&quot;uplink_message&quot;: {
&quot;decoded_payload&quot;: {
&quot;l&quot;: 4000,
&quot;t&quot;: 5017
}</code></pre></div></li>
<li>
<p>MQTT Gateway is now listening for <strong>HTTP Requests at port 9641</strong>!</p>
</li>
</ol>
<h2 id="checkpoint-bravo"><a class="doc-anchor" href="#checkpoint-bravo">§</a>2.3 Checkpoint Bravo</h2>
<p>Lets check the <strong>Sensor Data ingested</strong> by our MQTT Gateway…</p>
<ol>
<li>
<p>Start our <strong>LoRaWAN Firmware</strong> and transmit Sensor Data every minute</p>
<p><a href="https://lupyuen.github.io/articles/tsen#run-the-lorawan-firmware">(Like this)</a></p>
</li>
<li>
<p>Enter this at the command-line…</p>
<div class="example-wrap"><pre class="language-bash"><code>curl -v http://localhost:9641/metrics</code></pre></div></li>
<li>
<p>We should see our Sensor Data (Temperature and Light Level) ingested as <strong>Prometheus Metrics</strong></p>
<div class="example-wrap"><pre class="language-text"><code>## HELP l Light Level
## TYPE l gauge
l{sensor=&quot;eui-YOUR_DEVICE_EUI&quot;,
sensor_type=&quot;l&quot;,
topic=&quot;v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up&quot;
} 4000 1634364863274
...
## HELP t Temperature
## TYPE t gauge
t{sensor=&quot;eui-YOUR_DEVICE_EUI&quot;,
sensor_type=&quot;t&quot;,
topic=&quot;v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up&quot;
} 5056 1634364863274</code></pre></div>
<p>This says that the Light Level is <code>4000</code> and the Temperature is <code>50.56</code> ºC, recorded at the Timestamp of <code>1634364863274</code>.</p>
</li>
<li>
<p>Also watch for <strong>received_messages</strong></p>
<div class="example-wrap"><pre class="language-text"><code>## HELP received_messages received messages per topic and status
## TYPE received_messages counter
received_messages{status=&quot;success&quot;,
topic=&quot;v3/luppy-application@ttn/devices/eui-YOUR_DEVICE_EUI/up&quot;
} 3</code></pre></div>
<p>This says that our MQTT Gateway has successfully processed 3 messages from The Things Network.</p>
<p>Lets move on to Prometheus…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-flow3.jpg" alt="Prometheus Time Series Database" /></p>
</li>
</ol>
<h1 id="prometheus-time-series-database"><a class="doc-anchor" href="#prometheus-time-series-database">§</a>3 Prometheus Time Series Database</h1>
<p><em>How do we push the Sensor Data / Metrics from MQTT Gateway to Prometheus?</em></p>
<p>Prometheus collects Metrics by <strong>scraping them over HTTP</strong></p>
<p>(Much like the <strong>curl</strong> command from Checkpoint Bravo)</p>
<p>And stores the Metrics in its <strong>Time Series Database</strong>.</p>
<p>(Which is super efficient for querying sensor values that vary over time)</p>
<p>Lets <strong>configure and start Prometheus</strong> to scrape the Metrics from our MQTT Gateway…</p>
<ol>
<li>
<p>Download and unzip <strong>Prometheus</strong></p>
<p><a href="https://prometheus.io/download/"><strong>“Prometheus Download”</strong></a></p>
</li>
<li>
<p>In the unzipped folder, edit the Prometheus Configuration File…</p>
<p><a href="https://github.com/lupyuen/prometheus-the-things-network#configure-prometheus"><strong>prometheus.yml</strong></a></p>
<p><img src="https://lupyuen.github.io/images/prometheus-config6.png" alt="Prometheus Configuration for MQTT Gateway" /></p>
</li>
<li>
<p>Under the <strong><code>scrape_configs</code></strong> section, add the following…</p>
<div class="example-wrap"><pre class="language-yaml"><code>## Scrape configuration containing the endpoints to scrape
scrape_configs:
...
## Scrape The Things Network Metrics from MQTT2Prometheus
- job_name: &quot;ttn&quot;
## Metrics will be scraped from MQTT2Prometheus
## at http://localhost:9641/metrics
static_configs:
- targets: [&quot;localhost:9641&quot;]</code></pre></div></li>
<li>
<p>Note that Prometheus will scrape the Metrics from MQTT Gateway <strong>every 15 seconds</strong></p>
<div class="example-wrap"><pre class="language-yaml"><code>## Global Configuration
global:
## Set the scrape interval to every 15 seconds
scrape_interval: 15s</code></pre></div></li>
<li>
<p>Start the <strong>Prometheus Server</strong></p>
<div class="example-wrap"><pre class="language-bash"><code>## Change this to the unzipped path of Prometheus
cd prometheus
## For Linux and macOS:
./prometheus
## For Windows:
prometheus.exe</code></pre></div></li>
<li>
<p>We should see…</p>
<div class="example-wrap"><pre class="language-text"><code>main.go:400
&quot;No time or size retention was set so using the default time retention&quot; duration=15d
main.go:438
&quot;Starting Prometheus&quot; version=&quot;(version=2.30.3, branch=HEAD, revision=f29caccc42557f6a8ec30ea9b3c8c089391bd5df)&quot;
web.go:541
&quot;Start listening for connections&quot; address=0.0.0.0:9090
main.go:852
&quot;TSDB started&quot;
main.go:794
&quot;Server is ready to receive web requests.&quot;</code></pre></div></li>
<li>
<p>Prometheus is now listening for <strong>HTTP Requests at port 9090</strong>!</p>
</li>
</ol>
<h2 id="checkpoint-charlie"><a class="doc-anchor" href="#checkpoint-charlie">§</a>3.1 Checkpoint Charlie</h2>
<p>Lets check the <strong>Metrics scraped by Prometheus</strong> from MQTT Gateway…</p>
<ol>
<li>
<p>Start our <strong>LoRaWAN Firmware</strong> and transmit Sensor Data every minute</p>
<p><a href="https://lupyuen.github.io/articles/tsen#run-the-lorawan-firmware">(Like this)</a></p>
</li>
<li>
<p>Browse to our <strong>Prometheus Server</strong></p>
<div class="example-wrap"><pre class="language-uri"><code>http://localhost:9090</code></pre></div></li>
<li>
<p>Enter the <strong>name of our Metric</strong> (like for Temperature)…</p>
<div class="example-wrap"><pre class="language-text"><code>t</code></pre></div>
<p>Like this…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-metric2.png" alt="Checking the Metrics scraped by Prometheus from MQTT Gateway" /></p>
</li>
<li>
<p>Click <strong>“Execute”</strong> and <strong>“Graph”</strong></p>
<p>Our Metric appears in the graph.</p>
<p>(See pic above)</p>
</li>
</ol>
<p>Were ready for our final step… Connecting Prometheus to Grafana!</p>
<p><img src="https://lupyuen.github.io/images/prometheus-flow4.jpg" alt="Grafana Dashboard for Prometheus" /></p>
<h1 id="grafana-dashboard"><a class="doc-anchor" href="#grafana-dashboard">§</a>4 Grafana Dashboard</h1>
<p>Finally we <strong>install and configure Grafana</strong> to pull the Metrics from Prometheus (over HTTP) for rendering in a Grafana Dashboard…</p>
<ol>
<li>
<p>Follow the steps below to <strong>download and install Grafana</strong></p>
<p><a href="https://lupyuen.github.io/articles/prometheus#appendix-install-grafana"><strong>“Install Grafana”</strong></a></p>
</li>
<li>
<p>Browse to our <strong>Grafana Server</strong></p>
<p><strong><code>http://localhost:3000</code></strong></p>
<p><strong>Username:</strong> admin</p>
<p><strong>Password:</strong> admin</p>
</li>
<li>
<p>In the left menu bar, click…</p>
<p><strong>Configuration</strong><strong>Data Sources</strong></p>
<p>Click <strong>“Add Data Source”</strong></p>
<p><img src="https://lupyuen.github.io/images/grafana-datasource4.png" alt="Add Data Source" /></p>
</li>
<li>
<p>Look for <strong>“Prometheus”</strong> and click <strong>“Select”</strong></p>
<p><img src="https://lupyuen.github.io/images/prometheus-datasource.png" alt="Prometheus Data Source for Grafana" /></p>
</li>
<li>
<p>Set the <strong>HTTP URL</strong> to…</p>
<div class="example-wrap"><pre class="language-uri"><code>http://localhost:9090</code></pre></div>
<p><img src="https://lupyuen.github.io/images/prometheus-grafana5.png" alt="Grafana Data Source for Prometheus" /></p>
</li>
<li>
<p>Click <strong>“Save &amp; Test”</strong></p>
</li>
</ol>
<h2 id="checkpoint-delta"><a class="doc-anchor" href="#checkpoint-delta">§</a>4.1 Checkpoint Delta</h2>
<p>For our final checkpoint lets render our Sensor Data in a <strong>Grafana Dashboard</strong>!</p>
<ol>
<li>
<p>Start our <strong>LoRaWAN Firmware</strong> and transmit Sensor Data every minute</p>
<p><a href="https://lupyuen.github.io/articles/tsen#run-the-lorawan-firmware">(Like this)</a></p>
</li>
<li>
<p>In Grafana, click <strong>“Add Panel”</strong> (top right)</p>
<p>Click <strong>“Add An Empty Panel”</strong></p>
<p><img src="https://lupyuen.github.io/images/grafana-dashboard3.png" alt="Add Panel" /></p>
</li>
<li>
<p>Set the <strong>Data Source</strong> to <strong>“Prometheus”</strong></p>
<p>Under <strong>Metric Browser</strong>: Enter the name of our Metric (like for Temperature)…</p>
<div class="example-wrap"><pre class="language-text"><code>t</code></pre></div>
<p>Like this…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-grafana6.png" alt="Grafana Panel for Prometheus" /></p>
</li>
<li>
<p>Click the <strong>“Save”</strong> button (top right)</p>
</li>
<li>
<p>Our <strong>Sensor Data from The Things Network</strong> appears in the Grafana Dashboard!</p>
<p><img src="https://lupyuen.github.io/images/prometheus-grafana4.png" alt="Monitoring Devices on The Things Network with Prometheus and Grafana" /></p>
<p>(Remember: Our Temperature Values are scaled up 100 times)</p>
</li>
</ol>
<h1 id="transform-and-filter-sensor-data"><a class="doc-anchor" href="#transform-and-filter-sensor-data">§</a>5 Transform and Filter Sensor Data</h1>
<p><em>Can we tweak the display of Sensor Data in Grafana?</em></p>
<p>Grafana lets us <strong>transform and filter</strong> the Sensor Data for our Dashboard.</p>
<p>First we show the <strong>Raw Sensor Data</strong> as a table…</p>
<ol>
<li>
<p>Click <strong>“Panel Title”</strong> and <strong>“Edit”</strong></p>
</li>
<li>
<p>Click <strong>“Table View”</strong> (at top)</p>
<p><img src="https://lupyuen.github.io/images/prometheus-table.png" alt="Table View for Grafana Panel" /></p>
</li>
<li>
<p>Not quite what we expected… Everything gets lumped into a <strong>single column</strong>!</p>
<p>Lets <strong>split our Time Series Data</strong> into separate columns.</p>
</li>
<li>
<p>Click <strong>“Transform”</strong> Tab (at bottom)</p>
<p>Click <strong>“Add Transformation”</strong></p>
<p>Select <strong>“Labels To Fields”</strong></p>
</li>
<li>
<p>We should see this…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-transform.png" alt="Labels to Fields for Grafana Panel" /></p>
<p>Much better! Our <strong>Device ID</strong> (“sensor”), <strong>Sensor Type</strong> (“t”) and <strong>Value</strong> are now in separate columns.</p>
</li>
<li>
<p>If were rendering <strong>Multiple Devices or Sensor Types</strong>, we should set the <strong>Value Field Name</strong></p>
<p><a href="https://lupyuen.github.io/articles/prometheus#notes">(See this)</a></p>
</li>
</ol>
<p>Next we <strong>filter the Sensor Data</strong> that will be rendered in our Dashboard…</p>
<ol>
<li>
<p>Click <strong>“Transform”</strong> Tab (at bottom)</p>
<p>Click <strong>“Add Transformation”</strong></p>
<p>Select <strong>“Filter Data By Values”</strong></p>
</li>
<li>
<p>Click <strong>“Add Condition”</strong> and set the Condition…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-filter.png" alt="Grafana Panel with Filter" /></p>
</li>
<li>
<p>The above filter matches the <strong>Device ID</strong> with the Regular Expression…</p>
<div class="example-wrap"><pre class="language-text"><code>eui-70b3.*</code></pre></div>
<p>Which means that only Device IDs starting with <strong>“eui-70b3”</strong> will be rendered.</p>
</li>
<li>
<p>When were done, click the <strong>“Apply”</strong> button (top right)</p>
</li>
</ol>
<h2 id="auto-dashboard-refresh"><a class="doc-anchor" href="#auto-dashboard-refresh">§</a>5.1 Auto Dashboard Refresh</h2>
<p><em>Our Grafana Dashboard doesnt refresh automatically for real-time Sensor Data?</em></p>
<p>No worries! This neat trick will <strong>auto-refresh our Grafana Dashboard</strong> to render real-time Sensor Data…</p>
<ol>
<li>
<p>In our Grafana Dashboard, click the <strong>“Settings”</strong> button (top right)</p>
<p><img src="https://lupyuen.github.io/images/prometheus-refresh3.png" alt="Dashboard Settings" /></p>
</li>
<li>
<p>Under <strong>“Time Options”</strong>, uncheck <strong>“Hide Time Picker”</strong></p>
<p><img src="https://lupyuen.github.io/images/prometheus-refresh2.png" alt="Hide Time Picker" /></p>
</li>
<li>
<p>Click <strong>“Save Dashboard”</strong></p>
</li>
<li>
<p>Click the <strong>“Refresh Interval”</strong> (top right)</p>
<p>Select <strong>“5 Seconds”</strong></p>
<p><img src="https://lupyuen.github.io/images/prometheus-refresh4.png" alt="Refresh Interval" /></p>
<p>Now our Grafana Dashboard auto-refreshes every 5 seconds!</p>
</li>
</ol>
<h1 id="mqtt-with-tls-encryption"><a class="doc-anchor" href="#mqtt-with-tls-encryption">§</a>6 MQTT with TLS Encryption</h1>
<p>Theres a <strong>Security Risk</strong> in our configuration of MQTT Gateway…</p>
<p>Our <strong>MQTT Password is transmitted as clear text</strong> from our computer to The Things Network!</p>
<p>To secure our MQTT Password with <strong>TLS Encryption</strong>, follow the instructions here…</p>
<ul>
<li><a href="https://github.com/lupyuen/prometheus-the-things-network#mqtt-with-tls"><strong>“MQTT with TLS Encryption”</strong></a></li>
</ul>
<p><img src="https://lupyuen.github.io/images/prometheus-tls.jpg" alt="MQTT with TLS Encryption" /></p>
<h2 id="checkpoint-echo"><a class="doc-anchor" href="#checkpoint-echo">§</a>6.1 Checkpoint Echo</h2>
<p><em>What if we have problems enabling TLS Encryption for MQTT?</em></p>
<p>Run <a href="https://www.wireshark.org/"><strong>Wireshark</strong></a> on our computer and trace the <strong>TLS Certificates</strong> that are presented by The Things Network and by our computer.</p>
<p>The certificates should appear like this…</p>
<p><img src="https://lupyuen.github.io/images/prometheus-wireshark2.png" alt="Tracing MQTT with TLS Encryption" /></p>
<h1 id="sensor-data-alerts"><a class="doc-anchor" href="#sensor-data-alerts">§</a>7 Sensor Data Alerts</h1>
<p><em>Can we create Alerts for monitoring our Sensor Data?</em></p>
<p><em>Like when the Temperature gets too hot?</em></p>
<p>Yes thats possible with <strong>Prometheus Alert Manager</strong>!</p>
<blockquote>
<p>“The Alertmanager handles alerts sent by client applications such as the Prometheus server.”</p>
</blockquote>
<blockquote>
<p>“It takes care of <strong>deduplicating, grouping, and routing</strong> them to the correct receiver integration such as <strong>email, PagerDuty, or OpsGenie</strong>.”</p>
</blockquote>
<blockquote>
<p>“It also takes care of <strong>silencing and inhibition</strong> of alerts.”</p>
</blockquote>
<p>More details here…</p>
<ul>
<li><a href="https://prometheus.io/docs/alerting/latest/alertmanager/"><strong>Prometheus Alert Manager</strong></a></li>
</ul>
<p>Drop me a note if youre keen to learn about Prometheus Alerts!</p>
<p><img src="https://lupyuen.github.io/images/prometheus-arch.png" alt="Prometheus Architecture" /></p>
<p><a href="https://prometheus.io/docs/introduction/overview/">(Prometheus Architecture)</a></p>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>8 Whats Next</h1>
<p>I had fun integrating The Things Network with Prometheus and Grafana… Its something I always wanted to do. I hope you enjoyed it too!</p>
<p>In the next article Ill head back to <a href="https://lupyuen.github.io/articles/pinedio2"><strong>PineDio Stack BL604</strong></a> and run more IoT Experiments with LoRaWAN and The Things Network.</p>
<p>(Thankfully we now have a proper platform for Sensor Data visualisation and analysis: Prometheus + Grafana!)</p>
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> for supporting my work! This article wouldnt have been possible without your support.</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor">Sponsor me a coffee</a></p>
</li>
<li>
<p><a href="https://www.reddit.com/r/TheThingsNetwork/comments/qclqxg/monitor_iot_devices_in_the_things_network_with/">Discuss this article on Reddit</a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/book">Read “The RISC-V BL602 / BL604 Book”</a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io">Check out my articles</a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml">RSS Feed</a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/prometheus.md"><code>lupyuen.github.io/src/prometheus.md</code></a></p>
<h1 id="notes"><a class="doc-anchor" href="#notes">§</a>9 Notes</h1>
<ol>
<li>
<p>If were rendering <strong>Multiple Sensor Types</strong> in a Grafana Panel (like Temperature and Light Level)…</p>
<p>Set <strong>Labels To Fields</strong><strong>Value Field Name</strong> to <strong>“sensor_type”</strong></p>
<p><img src="https://lupyuen.github.io/images/prometheus-transform2.png" alt="Rendering multiple Sensor Types" /></p>
<p>This fixes the graph to plot one line per Sensor Type.</p>
</li>
<li>
<p>If were rendering <strong>Multiple Devices</strong> in a Grafana Panel…</p>
<p>Set <strong>Labels To Fields</strong><strong>Value Field Name</strong> to <strong>“sensor”</strong></p>
<p><img src="https://lupyuen.github.io/images/prometheus-transform3.png" alt="Rendering multiple Sensor Devices" /></p>
<p>This fixes the graph to plot one line per Device.</p>
</li>
<li>
<p>Why not use an SQL Database (like MySQL) instead of Prometheus?</p>
<p>Well an SQL Database might not scale up for high volumes of Sensor Data because…</p>
<ul>
<li>
<p>SQL Inserts can be costly, due to the frequent indexing. If we run SQL Select queries too often, we might have contention in the SQL Index.</p>
</li>
<li>
<p>SQL doesnt work well with time-based queries like “Whats the average daily temperature every day for the past year”. It gets tedious to code such queries (nested SQL) and they dont perform well.</p>
</li>
</ul>
<p>Thats why we use a Time Series Database like Prometheus. And when we use a Time Series Database, we need a tool like Grafana that can visualise the Time Series Data.</p>
</li>
<li>
<p>This article is the expanded version of <a href="https://twitter.com/MisterTechBlog/status/1450262680795713538">this Twitter Thread</a></p>
</li>
</ol>
<h1 id="appendix-configure-the-things-network-mqtt"><a class="doc-anchor" href="#appendix-configure-the-things-network-mqtt">§</a>10 Appendix: Configure The Things Network MQTT</h1>
<p>Follow these steps to <strong>enable the MQTT Server</strong> in The Things Network…</p>
<ol>
<li>
<p>Log on to <strong>The Things Network Console</strong></p>
</li>
<li>
<p>Click <strong>Applications</strong><em>(Your Application)</em><strong>Integrations</strong><strong>MQTT</strong></p>
<p><img src="https://lupyuen.github.io/images/grafana-ttn.png" alt="Configure The Things Network MQTT Server" /></p>
</li>
<li>
<p>Click <strong>“Generate New API Key”</strong> and copy the values for…</p>
<ul>
<li>
<p><strong>Public Address</strong></p>
</li>
<li>
<p><strong>Username</strong></p>
</li>
<li>
<p><strong>Password</strong></p>
</li>
</ul>
<p>(This is the only time we can see the password. Dont forget to copy it!)</p>
</li>
</ol>
<h1 id="appendix-install-grafana"><a class="doc-anchor" href="#appendix-install-grafana">§</a>11 Appendix: Install Grafana</h1>
<p>Follow these steps to <strong>install Grafana</strong> on Linux, macOS and Windows…</p>
<ol>
<li>
<p>Browse to <a href="https://grafana.com/oss/grafana/"><strong>grafana.com/oss/grafana</strong></a></p>
<p>Click <strong>“Get Grafana → Self-Managed → Download Grafana”</strong></p>
</li>
<li>
<p>For <strong>“Edition”</strong> select <strong>“OSS”</strong></p>
</li>
<li>
<p>Click Linux, macOS, Windows, Arm or Docker</p>
<p>(Grafana for Linux works on WSL too)</p>
</li>
<li>
<p>Follow the instructions to download and install Grafana</p>
</li>
<li>
<p>For Linux and macOS: Start the Grafana Server</p>
<div class="example-wrap"><pre class="language-bash"><code>## For Ubuntu and WSL
sudo service grafana-server restart
sudo service grafana-server status
## For macOS
brew services start grafana</code></pre></div></li>
<li>
<p>To test Grafana, browse to</p>
<p><strong><code>http://localhost:3000</code></strong></p>
<p><strong>Username:</strong> admin</p>
<p><strong>Password:</strong> admin</p>
</li>
</ol>
<p><img src="https://lupyuen.github.io/images/prometheus-refresh5.png" alt="Grafana rendering PineDio Stacks Internal Temperature over a one-hour period, thanks to Prometheus and The Things Network" /></p>
<p><em>Grafana rendering PineDio Stacks Internal Temperature over a one-hour period, thanks to Prometheus and The Things Network</em></p>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>