Running your own private MQTT broker on your local network? This is great for all your local Home Automation devices but would about things that are outside your network? Apps like OwnTracks can communicate your GPS to an MQTT broker but you’ll need to open up your network to the outside.
Today’s article details a hybrid-cloud solution using an external MQTT broker in conjunction with a private MQTT broker running on your local network. Home Assistant (or any Home Automation platform) can subscribe to topics on your local MQTT broker while still receiving messages from the external one.
Network Topology
Let’s take a look at how our network will look. There are two MQTT brokers, one running on a Digital Ocean droplet that is exposed to the internet. Any apps or remote MQTT devices can publish and subscribe to topics there. If you haven’t heard of Digital Ocean before it’s a cloud infrastructure company that allows you to rent “Droplets” or VMs. For $5 a month you can get a machine with 1GB of memory and a 25GB SSD, which is plenty for running an MQTT broker using Docker.
Some reasons you may want to do this:
- You don’t want to expose a port on your home network (or you can’t)
- You don’t want your local devices communicating off your network
- If your internet goes out, you want your local devices to still work
- You have a mix of local MQTT devices (like ESP8266s) and things that are external (your cell phone when away from home) and you want to keep them isolated
For my setup, I’m going to use an MQTT broker running on a Digital Ocean droplet. Home Assistant and a second MQTT broker will be running on my network. Home Assistant only knows about the MQTT broker on my network and any messages that are sent to the external broker are “bridged” to the local broker. Any messages sent to my local broker by local devices are not bridged, they just stay on my network (although this is easily configurable if you have a use case).
Installation and Configuration
Setting up bridged MQTT brokers is surprisingly simple, all it takes is a little configuration file editing to get it going. If you want to learn how to run a MQTT broker on a Raspberry Pi and hook it up to Home Assistant, check out my previous article.
External Broker
First, I’ll be making my external MQTT broker. For Digital Ocean, I went with one of the $5 Ubuntu 18.04 droplets.
After ssh-ing into the VM, I followed these steps to get Docker installed.
Next, we can pull the official Mosquitto docker image.
|
1 |
docker pull eclipse-mosquitto |
Now, make a few directories that will be volumes in the container that will be used by Mosquitto. Directories are needed for logs and data.
|
1 2 3 |
mkdir -p /var/mosquitto/logs mkdir -p /var/mosquitto/data chmod a+w /var/mosquitto |
Next, we need to create a minimal configuration file. Go ahead and create a new file named mosquitto.conf. Mine will be sitting at /root/mosquitto.conf on my droplet. You need the following basic settings:
|
1 2 3 4 |
persistence true persistence_location /mosquitto/data/ log_dest file /mosquitto/log/mosquitto.log |
We’re now ready to create and run the container. Issue the following command to start the container and forward ports 1883 and 9001 and mount the files and directories into the right place.
|
1 |
docker run -d -it -p 1883:1883 -p 9001:9001 -v /root/mosquitto.conf:/mosquitto/config/mosquitto.conf -v /var/mosquitto/data:/mosquitto/data -v /var/mosquitto/log:/mosquitto/log eclipse-mosquitto |
This starts up the container in daemon mode, so it will be running for us in the background. Run docker ps to see the container running. You can also view the log by running tail -f /var/mosquitto/log/mosquitto.log.
Now since this is an external service, you probably want to add a username and password to it, so that people don’t just use your MQTT broker (or try something worse). Log into the running container (first get the ID by running docker ps.
|
1 |
docker exec -it <container_id> /bin/sh |
Now, generate a password file using the following mosquitto_passwd command. Follow the prompts to enter a password for your broker.
|
1 |
mosquitto_passwd -c /mosquitto/data/passwd <username> |
This should create a password file at /mosquitto/data/passwd. Update your configuration file to tell Mosquitto about the new password requirements.
|
1 2 3 4 5 6 7 |
persistence true persistence_location /mosquitto/data/ log_dest file /mosquitto/log/mosquitto.log password_file /mosquitto/data/passwd allow_anonymous false |
Finally, restart the container so that it uses the new configuration file and password.
|
1 |
docker restart <container_id> |
Local Broker
The steps are pretty similar for getting your local MQTT server working. Go ahead and get Docker installed on your host and pull the image. However, the configuration file will need to “bridge” connetions from the remote broker to this one.
|
1 2 3 4 5 6 7 8 9 10 11 |
persistence true persistence_location /mosquitto/data/ log_dest file /mosquitto/log/mosquitto.log # External MQTT Broker connection external-bridge address <external_ip_address_broker> topic # in remote_username <username> remote_password <password> |
The important additions are:
connection:defines the start of a bridged connectionaddress:configures the IP address of the MQTT broker to bridge totopic:defines the topic patterns to be shared between the brokers. For this setup, we are subscribing to all topics on the external server, but we are not sharing our local messages with the external server. See the Mosquitto documentation for more information.remote_usernameandremote_password:if you set up a username and password on your external broker put them here to authenticate
Once you’ve got configuration file written, go ahead and start up your docker container. Make sure to update the paths to you configuration file and data/log directories to match your system.
|
1 |
docker run -d -it -p 1883:1883 -p 9001:9001 -v /var/mosquitto.conf:/mosquitto/config/mosquitto.conf -v /var/mosquitto/data:/mosquitto/data -v /var/mosquitto/log:/mosquitto/log |
Testing it Out
So we’ve got these Docker containers running, but how do we know it actually works? To test it out, we can make use of the Mosquitto command line utilities to subscribe and publish MQTT messages. If you have another tool you like for testing out MQTT that should work too.
To get the Mosquitto command line tools, check out their download page. We are going to be using the mosquitto_pub and mosquitto_sub commands. There are also some Docker images which have the client tools installed if you want to use them. There are a whole bunch of different tools to test MQTT though, so choose the one that fits your needs. I’ll be using the command line tools but you should be able to follow along.
You should open two different terminal windows. In the first one we are going to subscribe to our local broker by using the mosquitto_sub command.
|
1 |
mosquitto_sub -h localhost -v -t "testing/" |
This is listening for any messages coming in on the testing/ topic on the local broker. Next, let’s send a message to the external broker.
|
1 |
mosquitto_pub -h <external_ip> -t "testing/" -m "Testing bridged brokers!" -u <username> -P <password> |
This publishes a message on the external broker. If the terminal window that is subscribed to the topic shows the message, you know it got bridged correctly! The message was pushed to the external server but bridged to the local server where our client was subscribed to. Any external clients can now publish to the external server and Home Assistant or any subscribers locally will receive the message.
If you get stuck at any point, I recommend running tail -f on the Mosquitto log file. That should point out any typos or silly mistakes in the configuration files and at least help you get started debugging bigger issues.
Summary
Thanks for following along my walkthrough of using two MQTT brokers in bridged mode. There are a lot of interesting things you can do with this bridged connection like automatically add prefixs to topic names, relay messages externally and filter on certain topics. So I encourage you to read through the Mosquitto docs and come up with a configuration file that makes sense for you.
Let me know in the comments or through social media if you’re using something like this in production and what benefits you see! Feel free to reach out to me too if you get stuck at any point.