MQTT - Mosquitto MQTT Broker setup on the Ubuntu 20.04
I quoted a lot from "How to Install and Secure the Mosquito MQTT Messaging Broker on Ubuntu 18.04".
MQTT is a machine-to-machine messaging protocol, designed to provide lightweight publish/subscribe communication to “Internet of Things” devices. It is commonly used for geo-tracking fleets of vehicles, home automation, environmental sensor networks, and utility-scale data collection.
Mosquitto is a popular MQTT server (or broker, in MQTT parlance) that has great community support and is easy to install and configure.
MQTT is often thought of only as IoT platform and applied to devices such as Arduino and many sensors. However, Message Queue (MQ) is a protocol that has long been used to deliver asynchronous messages. It can be applied to various applications for asynchronous message delivery. The Public, Subscribe model for a particular topic can also be useful for implementing chat rooms, etc. There are rumors that Facebook's messenger actually used Mosquito.(This is something I haven't confirmed yet.)
MQTT communication architecture
MQTT does not consist of a client-server structure like HTTP, normal TCP, etc., but consists of a broker, publisher, and subscriber structure.
MQTT Brokers
The role of the broker that mediates messages in the middle is important, and the software that plays this role is called an MQTT broker (server).
The most popular of these open source products is Mosquitto from the Eclipse Foundation. There are many brokers that can implement MQTT, but I will explain using the most popular and easily accessible mosquitto.
MQTT Topic
Publishers and Subscribers work on a per-topic basis. Topics can be hierarchically organized using forward slashes (/), so that a large number of sensor devices can be efficiently managed. For example, if you have a sensor that measures various states of your computer, you could configure it as follows.
If you've ever implemented the HTTP Restful API, you'll know that forward slashes ("/") in URIs are used to separate resources.
In MQTT, topics can also be separated using forward slashes, so that subscribers and publishers can process messages efficiently.
If your subscriber wants to receive temperature, brightness, and CO2 density from all three topics belonging to /Home/Kitchen, you can do it at once with "/Home/Kitchen/#".
MQTT QoS level
QoS-related information brought from
- http://www.steves-internet-guide.com/understanding-mqtt-qos-1/
- http://www.steves-internet-guide.com/understanding-mqtt-qos-2/
MQTT has its origins in IBM's Message Queue. Message Queue uses TCP/IP but uses an asynchronous transmission method. Therefore, even when the communication of subscribe is cut off, the broker can keep the message and send it when communication is restored.
However, if all messages are processed in this way, the load is large, so MQTT can be used in three QoS modes.
MQTT provides 3 QOS levels-
- QOS 0 – Once (not guaranteed)
- QOS 1 – At Least Once (guaranteed)
- QOS 2 – Only Once (guaranteed)
QOS 0 – Once
This is the fastest method and requires only 1 message. It is also the most unreliable transfer mode. The message is not stored on the sender, and is not acknowledged. The message will be delivered only once, or not at all. Once the message has been sent by the client it is deleted from the outbound message queue. Therefore with this QOS level there is no possibility of duplicate messages.
QOS 1 – At Least Once
This level guarantees that the message will be delivered at least once, but may be delivered more than once. (See Flow Diagram on right.) Publishing with QOS of 1 requires 2 messages. The sender sends a message and waits for an acknowledgement (PUBACK).
If it receives an acknowledgement then it notifies the client app, and deletes the message from the outbound queue.. If it doesn’t receive an acknowledgement it will resend the message with the DUP flag set (Duplicate Flag). The message will continue to be resent at regular intervals, until the sender receives an acknowledgement.
QOS 2 – Only Once
This level guarantees that the message will be delivered only once. This is the slowest method as it requires 4 messages.
- The sender sends a message and waits for an acknowledgement (PUBREC)
- The receiver sends a PUBREC message
- If the sender doesn’t receive an acknowledgement ( PUBREC) it will resend the message with the DUP flag set.
- When the sender receives an acknowledgement message PUBREC it then sends a message release message (PUBREL). The message can be deleted from the queue.
- If the receiver doesn’t receive the PUBREL it will resend the PUBREC message
- When the receiver receives the PUBREL message it can now forward the message onto any subscribers.
- The receiver then send a publish complete (PUBCOMP) .
- If the sender doesn’t receive the PUBCOMP message it will resend the PUBREL message.
- When the sender receives the PUBCOMP the process is complete and it can delete the message from the outbound queue, and also the message state.
Installation
First of all, I will install Mosquito MQTT Broker in Ubuntu. Eclipse Mosquitto is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and is suitable for use on all devices from low power single board computers to full servers.
Installation is very simple. I will install Mosquito client for testing.
root@ubuntusrv:/# apt-get update root@ubuntusrv:/# apt-get install mosquitto mosquitto-clients
Simple Test
By default, Ubuntu will start the Mosquitto service after install.
root@ubuntusrv:/etc/mosquitto/certs# service --status-all [ + ] apparmor [ + ] apport [ + ] atd [ - ] console-setup.sh [ + ] cron [ - ] cryptdisks [ - ] cryptdisks-early [ + ] dbus [ + ] grub-common [ - ] hwclock.sh [ + ] irqbalance [ - ] iscsid [ - ] keyboard-setup.sh [ + ] kmod [ - ] lvm2 [ - ] lvm2-lvmpolld [ + ] mosquitto [ + ] multipath-tools [ - ] open-iscsi [ + ] open-vm-tools [ - ] plymouth [ - ] plymouth-log [ + ] procps [ - ] rsync [ + ] rsyslog [ - ] screen-cleanup [ + ] ssh [ + ] udev [ + ] ufw [ + ] unattended-upgrades [ - ] uuidd
Let’s test the default configuration. We’ll use one of the Mosquitto clients we just installed to subscribe to a topic on our broker.
Topics are labels that you publish messages to and subscribe to. They are arranged as a hierarchy, so you could have sensors/outside/temp and sensors/outside/humidity, for example. How you arrange topics is up to you and your needs. Throughout this tutorial we will use a simple test topic to test our configuration changes.
Log in to your server a second time, so you have two terminals side-by-side. In the new terminal, use mosquitto_sub to subscribe to the test topic:
root@ubuntusrv:/# mosquitto_sub -t test
-t is the topic name. You’ll see no output after hitting ENTER because mosquitto_sub is waiting for messages to arrive. Switch back to your other terminal and publish a message:
The options for mosquitto_pub are the same as mosquitto_sub, though this time we use the additional -m option to specify our message. Hit ENTER, and you should see hello world pop up in the other terminal. You’ve sent your first MQTT message!
root@ubuntusrv:/# mosquitto_pub -t "test" -m "hello world"
mosquitto_sub, mosquitto_pub are both mosquitto client programs.
The example above is tested locally. In most cases, we will probably communicate with different hosts, so we will test them on different hosts.
Test MQTT on Different Hosts
To test MQTT on different hosts, you can use the -h option to specify the host on which the MQTT broker is working.
On different hosts, you can see the same output coming from the mosquitto_sub program that subscribed the test topic.
Access Control with ID/Password
Let’s configure Mosquitto to use passwords. Mosquitto includes a utility to generate a special password file called mosquitto_passwd. This command will prompt you to enter a password for the specified username, and place the results in /etc/mosquitto/passwd.
root@ubuntusrv:/# mosquitto_passwd -c /etc/mosquitto/passwd your_id
This command will create or update passwd files that includes your_id and its encrypted password
Now we’ll open up a new configuration file for Mosquitto and tell it to use this password file to require logins for all connections:
root@ubuntusrv:/# vim /etc/mosquitto/conf.d/default.conf
root@ubuntusrv:/# cat /etc/mosquitto/conf.d/default.conf
allow_anonymous false
password_file /etc/mosquitto/passwd "allow_anonymous false" will disable all non-authenticated connections, and the password_file line tells Mosquitto where to look for user and password information. Save and exit the file.
CentOS users
If you are using CentOS, you might not find /etc/mosquitto/conf.d directory. In this case, modify the /etc/mosquitto/mosquito.conf file,
# vim /etc/mosquitto/mosquitto.conf
#### Add the following lines to the bottom of the file.
allow_anonymous false
password_file /etc/mosquitto/passwd Now restart the mosquitto broker again.
root@ubuntusrv:/# systemctl restart mosquitto
From now on, access to the Mosquito broker is only possible by using id/password.
Secure communication over SSL
Before you can use SSL, you must create a certificate. I will use the script file provided by https://github.com/owntracks/tools/blob/master/TLS/generate-CA.sh.
The ifconfig command must be installed to use this script. In recent Ubuntu or CentOS, the ipaddr command is mainly used, so it may not be installed. If the ifconfig command is not installed, install net-tools first.
root@ubuntusrv:/# apt-get install net-tools
Create certificates in the /etc/mosquitto/certs directory. First download the generate-CA.sh file. And modify the IP and host information at the beginning of the script file.
# IPLIST="172.13.14.15 192.168.1.1"
# HOSTLIST="a.example.com b.example.com"
IPLIST="211.53.210.33" # Enter Your IP list
HOSTLIST="myhost_pbx" # Enter Your Hostname list
Then run the script.
root@ubuntusrv:/etc/mosquitto/certs# wget https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh
root@ubuntusrv:/etc/mosquitto/certs# chmod 755 generate-CA.sh root@ubuntusrv:/etc/mosquitto/certs# ./generate-CA.sh
Be Careful : The "generate-CA.sh" file creates a certificate with a validity of 825 days. If you want to change the duration, find and modify "825" in the script file.
Now the cert directory looks like this. Files that start with ubuntusrv will be renamed on your computer. This value is the hostname.
root@ubuntusrv:/etc/mosquitto/certs# ls -al total 48 drwxr-xr-x 2 root root 4096 Oct 7 12:40 . drwxr-xr-x 5 root root 4096 Oct 7 11:59 .. -r--r--r-- 1 root root 2041 Oct 7 12:40 ca.crt -r-------- 1 root root 3272 Oct 7 12:40 ca.key -rw-r--r-- 1 root root 41 Oct 7 12:40 ca.srl -rwxr-xr-x 1 root root 8835 Oct 7 12:31 generate_CA.sh -rw-r--r-- 1 root root 130 Mar 3 2020 README -r--r--r-- 1 root root 2634 Oct 7 12:40 ubuntusrv.crt -rw-r--r-- 1 root root 1695 Oct 7 12:40 ubuntusrv.csr -r-------- 1 root root 3243 Oct 7 12:40 ubuntusrv.key
Paste in the following at the end of the file, leaving the two lines we already added:
root@ubuntusrv:/etc/mosquitto/conf.d# cat default.conf
allow_anonymous false
password_file /etc/mosquitto/passwd
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/ubuntusrv.crt
keyfile /etc/mosquitto/certs/ubuntusrv.keyAgain, be sure to leave a trailing newline at the end of the file.
"listener 8883" sets up an encrypted listener on port 8883. This is the standard port for MQTT + SSL, often referred to as MQTTS. The next three lines, certfile, cafile, and keyfile, all point Mosquitto to the appropriate Let’s Encrypt files to set up the encrypted connections. Save and exit the file, then restart Mosquitto to update the settings:
Now restart the mosquitto broker again.
root@ubuntusrv:/# systemctl restart mosquitto
Update the firewall to allow connections to port 8883 if you are using the ufw firewall.
root@ubuntusrv: # ufw allow 8883 Output Rule added Rule added (v6)
Copy the ca.crt file to a remote computer that communicates with the Mosquitto Broker. The ca.crt file can be created using copy/paste.
spypiggy@ubuntu:~$ cd /etc spypiggy@ubuntu:/etc$ sudo mkdir -p mosquitto/certs spypiggy@ubuntu:/etc$ cd mosquitto/certs/ spypiggy@ubuntu:/etc/mosquitto/certs$ vim ca.crt spypiggy@ubuntu:/etc/mosquitto/certs$ sudo vim ca.crt
You can see that SSL-enabled communications with id/password are working properly.
Procotols
MQTT has two communication protocols. One is mqtt and the other is websocket.
And these two protocols can use security protocols using SSL. Therefore, four protocols can be used. The following is the protocol and port settings section of the mosquitto.conf file.
#This is MQTT Protocol port 1883 protocol mqtt #This is Secure MQTT Protocol listener 8883 protocol mqtt cafile /etc/mosquitto/CA/ca.crt certfile /etc/mosquitto/CA/ubuntu.crt keyfile /etc/mosquitto/CA/ubuntu.key #This is Websocket Protocol listener 9001 protocol websockets #This is Secure Websocket Protocol listener 9003 protocol websockets cafile /etc/mosquitto/CA/ca.crt certfile /etc/mosquitto/CA/tubuntu.crt keyfile /etc/mosquitto/CA/ubuntu.key
If you set it as above, you can check the port information as follows.
[root@ubuntu]# netstat -nltp|grep mosquitto
tcp 0 0 0.0.0.0:8883 0.0.0.0:* LISTEN 277738/mosquitto
tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN 277738/mosquitto
tcp6 0 0 :::9001 :::* LISTEN 277738/mosquitto
tcp6 0 0 :::9003 :::* LISTEN 277738/mosquitto
tcp6 0 0 :::8883 :::* LISTEN 277738/mosquitto
tcp6 0 0 :::1883 :::* LISTEN 277738/mosquitto Test Secured MQTT on Different Hosts
If you want to test MQTTS on a remote client, you need to be careful about the hostname.
You should use hostnames instead of IP addresses. Files using the host name from the certificate I created earlier were created. Therefore, modify and use the hosts file of the client you want to test.
If it is Windows, edit the hosts file in the following directory. You must edit the file with administrator privileges to save it.
An error may occur if you use the host's IP address as shown below. The following is a test using the secured mqtt protocol.
C:\Users\spypi>mosquitto_pub -h 211.53.209.60 -p 8883 --cafile d:\temp\ca.crt -t test -u dongu -P dongdong -m "Hello"
Error: protocol error
C:\Users\spypi>mosqui
tto_pub -h ubuntusrv -p 8883 --cafile d:\temp\ca.crt -t test -u dongu -P dongdong -m "Hello"
If you are a Linux client, you can use it by editing the /etc/hosts file.
Wrapping Up
The process of installing and testing the Mosquito MQTT Broker is not difficult. We also tested id/pw application and SSL communication. Next, use MQTTcla instead of Mosquito_sub, Mosquito_pub. I'll try to test it by implementing it myself.
If you are interested in mqt client implemented by Python, see MQTT - Python Client.
If you are interested in mqt client implemented by c/c++, see MQTT - c/c++ Client.
Useful links
댓글
댓글 쓰기