How to Make a Cool Messenger That Works Fast with a Weak Internet

Why You Should Use MQTT and Node.js for Real-Time Messaging

By Eugene Golovan

Currently, more and more applications that we develop in S-PRO require real-time messaging and data transfer. In most cases, we use the Socket.IO library for Node.js. However, there are alternatives that have their advantages. MQTT (Message Queue Telemetry Transport) is one of them. And that’s what we’ll talk about here!

What is MQTT

mqtt.org

“It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimize network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.” — http://mqtt.org/faq

Thus, MQTT is a simplified network protocol that runs over TCP/IP. It helps to exchange messages between devices via publish-subscribe pattern. The first version of the protocol was developed by Dr. Andy Stanford-Clark (IBM) and Arlen Nipper (Arcom) in 1999 and published under a royalty-free license. The MQTT 3.1.1 specification was standardized by the OASIS consortium in 2014.

Why will you like it?

• Easy to use. The protocol is a program unit without additional functionality, which can be easily integrated into any complex system.
• The publish-subscribe pattern is convenient for most solutions. It allows devices to communicate and publish/receive messages that were not previously known or predefined.
• Easy to manage.
• The load on the communication channel is reduced.
• It works with a slow or unstable connection.
• There is no restriction on the format of the transmitted content.

What about benefits?

• binary
• low overhead (2+ bytes)
• multiple QoS level
• offline messaging
• topic wildcards +, #
• retained messages, last will and testament, heartbeats and other things…

Let’s look at it!

Binary & low overhead

There is a very low overhead in the structure of transmitted binary data. It means that in comparison with many other protocols (for example, with HTTP) it almost does not load the network with the transfer of information, which is necessary only for the functioning of the protocol. According to measurements made in 3G networks, the MQTT’s capacity is 93 times higher than the REST (Representational State Transfer) protocol has that runs over HTTP.

Multiple QoS level

MQTT can specify the Quality of Service (QoS) level. In general, there are three levels:

• QoS 0. The recipient does not confirm receiving the messages. The sender, accordingly, transmits the message just once, without attempting to retransmit it in the future. This is the “send and forget” method.
• QoS 1. It is guaranteed that the receiver will receive a message at least once. In this case, the subscriber can receive the same message several times. And the sender will make repeated attempts of sending until it receives a confirmation of the successful delivery of the message.
• QoS 2. The slowest message delivery procedure corresponds to this level of service quality, but it is the most reliable one. Its main feature is the implementation of the “one-time message delivery” strategy. It provides a four-step message delivery confirmation procedure.

You choose a specific level of service quality based on:

• the characteristics of the transmitted data
• the importance of it to be delivered.

Offline messaging/persistent session

This option can be enabled or disabled. When “0” is set, the broker saves the session and all the client’s subscriptions. The next time the connection is established it shows all the messages from QOS1 and QOS2 that were received by the broker during the disconnection. Accordingly, when “1” is set and the connection is renewed, the client will need to re-subscribe to the topics.

Topic wildcards

A topic is a UTF-8 string, which is used by the broker to filter messages for each connected client. A topic consists of one or more topic levels. Each topic level is separated by a forward slash (topic level separator).

When a client subscribes to a topic it can use the exact topic the message was published to or it can subscribe to more topics at once by using wildcards.

A single-level wildcard is a substitute for one topic level. The plus symbol represents a single-level wildcard in the topic.

While the single-level wildcard only covers one topic level, the multi-level wildcard covers an arbitrary number of topic levels. In order to determine the matching topics, it is required that the multi-level wildcard is always the last character in the topic and it is preceded by a forward slash.

Other features

Also, MQTT has some other cool features like retained messages, last will and testament, and heartbeats. We won’t cover them in this article but the info is widely available on the Internet. So, if you are interested, go ahead.

Brokers and Clients for Node.js

Let’s do some code! In this section, we’ll show you how easily you can get a working solution with Node.js. Also, you’ll get to know some modules. Let’s try to implement such scheme in practice.

MQTT requires a broker, which is the heart of any publish/subscribe protocol. The broker is primarily responsible for receiving all messages, filtering them, deciding who is interested in them and then sending the message to all subscribed clients.

As a broker, we will use Mosca.

Mosca is an MQTT broker as an NPM module. You can use it both from the command line and from the npm module. In this case, we will use it as a module.

To support offline messages we will use Ascoltatori with MongoDB. In general, Ascoltatori is a simple publish/subscribe library supporting the following brokers/protocols:

• Redis, a key/value store created by @antirez.
• MongoDB, a scalable, high-performance, document-oriented database.
• Mosquitto and all implementations of the MQTT protocol.
• RabbitMQ and all implementations of the AMQP protocol.
• ZeroMQ to use Ascoltatori in a P2P fashion.
• QlobberFSQ, a shared file system queue.
• Apache Kafka, a high-throughput distributed messaging system.
• Memory-only routing, using Qlobber.

You can see the backend code below. This is the code of a real working server that includes an MQTT broker, a https server to support receiving MQTT packets via WebSockets, an Ascoltatori adapter to save messages to the MongoDB database.

// Require MQTT broker
var mosca = require('mosca');
// Define connection to MongoDB
var mongo_con = 'mongodb://localhost:27017/mqtt';
// This settings is required to enable persistent session feature.
// All messages will be stored in MongoDB
var ascoltatore = {
type: 'mongo',
url: mongo_con,
pubsubCollection: 'ascoltatori',
mongo: {}
};
// Final settings for Mosca MQTT broker
var settings = {
port: 1883,
backend: ascoltatore,
persistence: {
factory: mosca.persistence.Mongo,
url: mongo_con
}
};
// Define HTTP and MQTT servers
var http     = require('http'),
httpServ = http.createServer(),
mqttServ = new mosca.Server(settings);
// Attach HTTP to MQTT server
mqttServ.attachHttpServer(httpServ);
httpServ.listen(3000);
// Triggers when mqtt server is ready to accept requests
mqttServ.on('ready', ready);
// Triggers when new message is published
mqttServ.on('published', function(packet, client) {
console.log(packet.topic + ': ' + packet.payload);
});
function ready() {
console.log('Mosca server is up and running');
}

To implement the MQTT client, we will use the npm MQTT.js module.

//Require MQTT library
var mqtt = require('mqtt');
// Define client connecting to our MQTT server
// clean: false means do not start new session on reconnect
// This allows us to use persistent sessions feature of MQTT protocol
// In addition clientId must be some unique for each client string
var client  = mqtt.connect('mqtt://localhost', {
clean: false,
clientId: 'console_client'
});
// Triggers on connect
// Cleant subscribes on topic with qos: 1 which means QOS level. Client will be getting all messages on reconnect.
// Then it publishes new message in to topic. This message will be stored in DB and sent to subscribers. Offline subscribers with QOS 1 will get it on reconnect.
//
client.on('connect', function () {
client.subscribe('/hello/s-pro', {qos: 1});
client.publish('/hello/s-pro', 'Hello, S-PRO!', {qos: 1});
});
// Do something when new message arrives.
client.on('message', function (topic, message) {
console.log(topic + ': ' + message.toString());
});

Let’s consider one more example of the client’s implementation. In this case, it will be MQTT over WebSockets. This example will be especially useful for clients who work with browsers or hybrid mobile applications.

All the code will be similar to the previous one. The only difference is the client’s definition.

client = mqtt.connect('ws://mqtt.tk:3000/mqtt', {
clean: false,
clientId: username
});

As you can see, here we connect to our server via sockets. You can find the working code on our Github. All examples of the code work perfectly, and they were used by the author during the presentation as a living example.

Scalability

This section is beyond the scope of this article because of its vastness. But it’s important to mention the extent to which this solution is expandable. You can create a scalable MQTT infrastructure using Node.js, Redis, HAProxy quite easily. Another option is using Apache Kafka. At the end of the article, you’ll see several references to interesting solutions in this area.

Conclusion

So, in this article, we have examined and tested in practice how easy it is to use the MQTT protocol for messaging using JavaScript and Node.js. Mosca broker can be used on the server side. Furthermore, it supports WebSockets. That is, if necessary you can receive messages from IoT devices and transfer them to mobile or desktop applications over TCP via WebSockets. Moreover, MQTT is very scalable. But even without that, it is very fast and can process 10k+ messages per second from 10k+ concurrent connections.

Useful links:

https://github.com/S-PRO/Moscato-RTM — The perfectly working code that we managed to create during our experiment. Simply test it!

https://www.youtube.com/watch?v=WE7GVIFRV7Q – This video from the expert of IoT Matteo Collina was the real inspiration for this article!

http://mqtt.org/ - Official website of the protocol.

https://github.com/mcollina/mosca — Mosca npm module.

https://github.com/mqttjs/MQTT.js — MQTT.js npm module.

http://www.hivemq.com/blog/mqtt-essentials/ — HiveMQ has a huge amount of useful info as well.

https://medium.com/@lelylan/how-to-build-an-high-availability-mqtt-cluster-for-the-internet-of-things-8011a06bd000

https://www.linkedin.com/pulse/iot-kafka-mqtt-bridge-using-mosca-robert-fuller

If you liked this, show your support by clapping us to share with other people on Medium.

Follow us on Facebook, Instagram, LinkedIn, Behance, Medium and visit our corporate blog for more news and articles on smart solutions.

Any questions? Feel free to contact us!