Code, Design, General, Technology

How we ditched HTTP and transitioned to MQTT!

In the field of location tracking there needs to be lot of back-and-forth communication between devices and the backend. Device transmits location stream and health information (battery level, network strength, etc.). Backend processes this information, applies business logic on top and sends configuration commands back to devices in order to orchestrate tracking. These configuration commands determine when to start/stop tracking, frequency at which to collect GPS data (time and distance), frequency at which to transmit GPS data and so on.

In a world with patchy mobile networks making all this communication robust is quite a task. It is important to choose the right network protocol and design the communication semantics to get maximum benefit of the protocol’s capabilities. We recently switched a large part of our device-backend communication from HTTP to MQTT. This blog is about how we achieved it and our learning from it so far.

First of all, here’s why we decided to move to MQTT:

  • MQTT is much more battery efficient (than HTTP) in maintaining long running connections and transmitting data periodically
  • With MQTT we now have a real 2-way connection where server can push commands to the clients whenever required

With these goals in mind we setup a Mosquitto broker on an EC2 instance. We combined header and body of our regular HTTP request into a single MQTT message that devices send to the Mosquitto broker. We made a simple MQTT topic scheme that maps 1:1 with our API endpoints. So far so good.

Now the challenge was processing these large number of incoming messages and POST’ing them from the Mosquitto broker to our API server. We thought of writing a subscriber to Mosquitto which would parse these incoming messages and make API calls for each message. But when this subscriber is deployed onto multiple hosts, each of those hosts would get a copy of EVERY message. We would then have to make sure that only one subscriber is making the POST call in response to a given message. To achieve this we would have needed to maintain state across subscribers and so on.

This is when we stumbled across AWS IoT and it’s powerful rule engine. At the core of it, IoT has its own MQTT broker and you can define custom rules to process each incoming message. E.g. You can store the incoming message into S3/DynamoDB, push a notification on SNS/SQS, or write a lambda function that will be invoked in response to each incoming message. We created a bridge from our Mosquitto broker to AWS IoT so that messages on Mosquitto are forwarded to the IoT. We defined rules so that our own lambda function is invoked in response to each incoming message on a topic. The lambda function simply parses the message, takes out header and body, and makes a usual HTTP POST call to the API server. This bridging approach has given us simplicity of Mosquitto and flexibility of IoT’s rule engine. Here is a diagram of our current architecture.

pasted-image-at-2016_11_10-12_08-pm

Here are some of our learnings so far:

  • Quality of Service (QoS) selection: MQTT provides three QoS levels (0, 1, 2). This is super useful because the protocol handles re-transmission and guarantees the delivery of the message even when the underlying network is unreliable. It is important to choose appropriate QoS levels for different messages in your application. E.g.  we chose QoS 0 (fire and forget) for transmitting health because losing the occasional health message does not affect us. But we use QoS 1 (delivery  guaranteed at least once) while sending configuration commands from server since we cannot afford to lose the commands and our clients can handle duplication.
  • Retain messages: Another neat feature of MQTT is that the broker can retain last message on each topic. In a rightly designed topic scheme retained messages can act as a status information for the client. E.g. Our topic scheme for sending configuration commands is in the form of ‘Push/SDKControls/<driver_id>’, so whenever a driver comes online he receives the last command sent by our API server as the last message for him is retained at the broker. This makes the protocol asynchronous by relaxing the requirement for client to be connected to network when message is sent from server. We can simply fire the message from our API server and just rely on the MQTT broker to forward the message when clients connect back to network.
  • Processing MQTT messages: While it has been super easy to setup the Mosquitto broker and get the messages flowing into it, we did not find any libraries which allow rule engine type processing of these messages. Maybe there is a tooling gap in the MQTT ecosystem. Hence we chose to forward the messages onto AWS IoT and use the rule engine they provide for processing these messages. The initial bridge setup is little tricky because IoT broker does not implement full MQTT protocol specification but once we got it working it has been reliable.
  • Logging: It is useful to run the Mosquitto broker in verbose mode and capture all logging in a searchable store. We are using Papertrail and have setup remote syslog on the broker host to transmist logs to Papetrail.

We are using these MQTT client libraries for Android, iOS and Python. We really liked the MQTT Essentials series by HiveMQ and highly recommend reading through it if you are planning to use the protocol in your production services.

We are yet to benchmark the battery gains from switching to MQTT, those might come in another blog. We have achieved the goal of having a real 2-way communication where we can reliably push commands down to our SDK clients. Overall we have been happy with the setup so far and thanks to Mosquitto-IoT bridge the effort required to switch have been minimal.

Have questions? Suggestions? Join the discussion on slack.

Like what we are doing? Sign up to use HyperTrack and build location tracking features!

You might also enjoy:

Standard

10 thoughts on “How we ditched HTTP and transitioned to MQTT!

  1. Pingback: How we ditched HTTP and transitioned to MQTT! – Scapbi's Weblog

  2. Hans says:

    Distributing clients over workers is not that hard is you are only looking for workers to proxy requests you your API

    Instead of using the API endpoint as topic one could use a key as topic, eg queue/ where key is a value determined by the client ( e.g random number between 1 and 255, make sure it is not too small).
    Each worker can now subscribe to one or more of these queues, if one of the workers gets way more load than its peers then you only need to distribute a few of its subscriptions to other workers, which could be done automatically with a few simple rules and a control queue watched by a controller worker. Tombstone messages could be used by a controller to detect failing workers and by a watchdog to detect failing controllers.

    • Amit Rathi says:

      Yes, it’s possible to do a self load balancing scheme like that. But it requires me to write more code to manage subscription, monitor load, write rules for re-distribution of traffic and so on. And I am a lazy developer (for a good reason), I want to write less code and maintain even less. That’s why we mentioned in the blog that MQTT ecosystem could benefit from a general purpose rule engine library like that. In absence of which it made sense to bridge this to IoT and use their rule engine for processing.

    • Amit Rathi says:

      There isn’t much out there suggesting that Websocket is battery efficient. Whereas MQTT is designed for resource constrained devices and is know to work well in patchy networks. We also get to use features like QoS, retained messages, last will & testament etc. not sure what alternatives for these exist in the Websocket world.

  3. Pingback: Battery Efficient Real-Time GPS Tracking – HyperTrack

    • Amit Rathi says:

      We want to keep size of our SDK as low as possible, so developers don’t hesitate to include it in the app. AWS Android SDK required to connect to IoT broker is quite large (~500kb) and so we decided to skip it and just include a simple MQTT paho client.

  4. Pingback: Centralized logging at HyperTrack | HyperTrack

  5. Pingback: The pitfalls of using location streams as an interface for building live location features | HyperTrack

Leave a Reply

Your email address will not be published. Required fields are marked *