Software Components
We utilise Docker and Docker Compose to provide all necessary LoRaBridge software components as a multi-container Docker application. The components needed for bridges and gateways vary and are listed below.
Bridge
Butix, based on works by Lucasbosch and Cmykey, Raspberry Pi 3 illustration, modified, CC BY-SA 4.0
As shown here, a bridge needs a ZigBee dongle for connecting ZigBee devices, a LoRaWAN hat for establishing a link to the gateway as well as optionally a network connection.
Zigbee2MQTT
Zigbee2MQTT exchanges device data with the ZigBee dongle as it handles join events, status updates and so on. It keeps a list of devices and publishes the device data via MQTT messages to Eclipse Mosquitto.
Eclipse Mosquitto
Mosquitto is an MQTT message broker, meaning you can publish your own messages as well as subscribe to messages sent by others. In case of the bridge, Zigbee2MQTT publishes the sensor data to the Mosquitto server, while the Forwarder is subscribed to those.
Forwarder
The Forwarder is a self-provided Python3 program and listens to message on the Mosquitto server. It removes disabled attributes (a list that can be modified via the web interface), substitutes known attibute keys as well as reformats and compresses the data by using the MessagePack format. The resulting compressed data is pushed to a list in a Redis server.
Redis
Redis is an in-memory data store and is used as a cache / message queue in our case. It receives compressed sensor data from the Forwarder and holds the data until LoRaWAN TX retrieves it.
Bridge LoRaWAN Interface watcher
LoRaWAN Interface Watcher is a Python3 program that regularly checks the status of the LoRaWAN Interface and restarts the container, if it crashed. Furthermore, it sends a notification to the Gateway via a system event. It uses the Docker API via a socket.
Docker should restart containers automatically when they crash using the restart policy we configured. Unfortunately, Docker refuses to do so, when a device needed by the container is missing. As our desired behaviour would be to restart the container as soon as the device is readded, we chose to implement it using our LoRaWAN Interface Watcher.
Bridge LoRaWAN Interface
LoRaWAN TX is a python program, which acts as an interface between the physical LoRaWAN modem and the software components on the bridge.
It has three main tasks:
- Bridge system time synchronization via TimeSyncRequest feature of LoRaWAN 1.0.3.
- Listening to downlink data events and forwarding the data towards automation manager
- Fetching compressed device data and user/system events from Redis and pushing them towards the gateway.
NodeRED
NodeRED implements user defined automations. MQTT is used for interfacing Zigbee devices and Redis (via NodeRED Redis add-on) for interfacing LoRaBridge events.
Automation manager
Automation manager performs decompression of automation configuration commands, stores automations in an intermediate data structure and ultimately composes NodeRED flows and uploads them to NodeRED.
Web Interface
Our web interface is a self-provided SvelteKit web application that shows the ZigBee devices, which are retrieved via the SSE server. It enables you to disable unnecessary sensor attributes, in order to further reduce the transmitted data. Authentication is provided with basic auth via nginx.
Info
The default port via Basic Auth is 3000
Example
SSE Server
The SSE server is a self-provided TypeScript application. It retrieves a list of ZigBee devices from the Zigbee2MQTT server and provides the data per HTTP as well as any updates to the data (e.g. new devices, additional attributes) per server-sent events (SSE) for the web interface.
Basic Auth
This component consists of a nginx and provides authentication for the web interface via Basic HTTP Authentication.
Ofelia
Ofelia is a job scheduler design for docker environments. This container is only connected to the host network and we use Ofelia to periodically retrieve the current IP addresses and store them inside a file. This file is mounted into the LCD UI container and provides thie IP information. This setup is necessary, because docker does not allow a container to belong to multiple networks at once (per docker-compose.yml).
Gateway
Butix, based on works by Lucasbosch and Cmykey, Raspberry Pi 3 illustration, modified, CC BY-SA 4.0
Note
The grey arrows symbolize the logical communication flow. The black/white and brownie/orange colored arrows in the gateway diagram above show the direct communation paths.
As shown here, a gateway needs a LoRaWAN hat for establishing links to the bridges as well as optionally a network connection.
Gateway Flow UI
The web interface is based on the SvelteKit web application framework in combination with the Svelte UI component framework. The interface uses Flowbite Svelte for individual UI components (such as styled buttons) and Svelte Flow for flow visualization.
To enable undo functionality, individual versions of a flow are stored in a CouchDB database, which resides locally in the browser. Furthermore, the changes are synchronized with a CouchDB database on the gateway. This allows flows to be modified offline and uploaded later, ensuring no data is lost if the internet connection is lost.
Usage
We placed special emphasis on the usability of the user interface to make the creation of automations as intuitive as possible. For this purpose, the layout of the interface was built on well-known, commonly used structures.
At the top, there is a menu bar where settings can be configured, and actions can be executed. The rest of the area is divided into three columns.
The left column contains the available automation nodes (graphical elements for constructing the flow diagram) that can be dragged and dropped into the visualization area.
The middle column is used for visualizing the flow diagram. Here, the previously added nodes can be connected to define the automation process. Additionally, the properties of each node or automation element can be configured via various input fields. Device-related nodes, such as "Sensor" and "Output Device," also provide a list of available devices and the sensor attributes accessible per device.
The right column is for managing individual flows. Flows can be thought of as simplified files in this context. Accordingly, new flows can be created, and existing flows can be renamed or deleted. Clicking on a specific flow in the right column loads it and displays it in the middle column.
Changes to the flow diagram are saved automatically and can be undone or redone via the menu bar. Additionally, the flow can also be transferred to the bridge and activated or deactivated via the menu bar.
The "Status" button opens a dedicated status area that contains messages from the bridge, such as critical error messages.
Info
The default port is 3000
Gateway Flow Manager
To integrate the web interface into the system, the Flow Manager was created as a standalone Python3 component. The Flow Manager loads flows from the Redis database, which are stored there by the web interface when deploying the flows. It compares new versions with the previous version, if applicable, and generates the necessary compressed action commands. These commands are transmitted to the bridge, where they are converted into the actual Node-RED flow by the Automation Manager. After transmission, the correctness of the transferred flow is verified on both sides via a checksum comparison.
CouchDB
CouchDB is a document-oriented NoSQL database with an emphasis on synchronization. It is used by the Flow UI as an in browser instance for storing each change (version) of a flow created via the webinterface. Furthermore, the changes are synchronized with a CouchDB instance on the gateway. This allows flows to be modified offline and uploaded later, ensuring no data is lost if the internet connection is lost.
We created our own CouchDB Docker image, as there is no prebuilt image for armhf (32 bit) systems. This image is only used for the database on gateway.
Packet Forwarder
The Packet Forwarder is a self-provided C application based on this repository that interacts with the LoRaWAN hat. It receives the LoRaWAN packets and publishes the data to the ChirpStack Gateway Bridge on port 1700/udp.
Eclipse Mosquitto (Gateway)
Mosquitto is an MQTT message broker, meaning you can publish your own messages as well as subscribe to messages sent by others. In case of the gateway, the ChirpStack Gateway Bridge publishes the forwarded LoRaWAN data to the Mosquitto server. Various other services read/write data from to the Mosquitto server, while modifying the data. The next service in line in the logical processing chain is ChirpStack.
ChirpStack Gateway Bridge
The ChirpStack Gateway Bridge converts LoRa Packet Forwarder protocols in a common data format. It receives the data from the Packet Forwarder, converts it and publishes it to the Mosquitto for the ChirpStack.
ChirpStack
ChirpStack handles parts of the LoRaWAN communication like the authentication as well as provides a device inventory as well as a web interface for displaying LoRaWAN connections, data and for configuration. It persists data in a PostgreSQL database and uses Redis as a session store and for non-persistent data. It receives and publishes data via Mosquitto.
Info
The default port is 8080
Example
PostgreSQL
PostgreSQL is an object-relational database and by the ChirpStack for storing persistent data.
Redis (Gateway)
Redis is an in-memory data store and is used for storing session and non-persistent data by the ChirpStack.
Converter
The Converter is a self-provided Python3 application, which listens for device data and other types of messages published by the ChirpStack Gateway Bridge via MQTT message on the Mosquitto. For the device data type, it decompresses the data, undoes the key substitution and reformats the data. Afterwards, the data is published back to the Mosquitto server. Other types of message are processed accordingly and passed to other service either via Redis or Mosquitto.
Device Manager
The Device Manager is a self-provided Python3 application keeping track of the seen devices via the Redis server. It publishes MQTT messages on the Mosquitto server for device discovery events and status (data) updates. These messages are picked up by the HA Integration service.
HA Integration
The HA Integration is a self-provided Python3 application. It translates the MQTT messages sent by the Device Manager into messages understood by the MQTT integration of the Home Assistant.
Tip
This additional translation step enables easy integration of other services (e.g. a replacement for Home Assistant or an extra web interface).
Home Assistant
Our Home Assitant is a preconfigured version of the official Home Assistant, a home automation web interface, and is used to diplay the devices and sensor data. It subscribes to MQTT messages via Mosquitto.
Info
The default port is 8123
Example
LoRaWAN module (LilyGo LoRa32 Dongle)
Firmware
The firmware for the ESP32-based LoRaWAN module (LilyGo LoRa32 Dongle) performs two tasks. First, it retrieves sensor/device data from the bridge forwarder over the LoRaWAN Interface container via a serial USB interface and sends it to ChirpStack over a LoRaWAN connection. Second, compressed automation configuration packages are transmitted to the module via LoRaWAN downlink (as part of MAC commands and ACK packets). If no device data is scheduled for uplink transmission, the firmware sends a heartbeat packet to enable periodic downlink communication.