Tag Archives: mqtt

Public key encryption of MQTT messages, with C and python

Posted by KarlP on 17 August, 2011 2 comments

WARNING Don’t do this. It does work, but you are limited to sending messages that are smaller than the public key size less padding bytes. The “correct” way is to use the higher level EVP_SealXXXX and EVP_OpenXXXX methods. (which are substantially uglier, but are “the right thing”™ I’ll try and package up some sample code in the neart future

As a follow on from yesterday, where we simply signed the message body being sent to the MQTT server, today we will use public key encryption to secure the message bodies themselves. For an environment where multiple people are connected to the MQTT server, but only one of them is authorized to be reading all the data, this seems a nice way of doing it. We don’t care if any given client is compromised, they only get our public key.

First off, I was less than impressed with the state of the public key encryption/decryption code in python. There’s nothing in the standard library, and PyCrypto doesn’t support reading in standard openssl key files, so I didn’t even bother looking much further. KeyCzar promises to be useful, doing “the right thing” instead of forcing you to know everything about crypto yourself. (I resent having to read the docs closely enough to make sure I chose PKCS_OEAP padding, rather than just PKCS padding for the RSA encryption. I do NOT want to be a crypto guy, and I know that I will never know enough) However, KeyCzar also insists on doing it’s own key management, so I moved on. M2Crypto seems to be the most fully implemented, but, unfortunately, it keeps the terrible/wonderful OpenSSL API. I’m using OpenSSL in the C code, so at least it’s consistent, even if it is a terribly verbose and awkward API.

Other things that I learnt were important, in the C code.

  • Remember to call XXX_free() on OpenSSL objects that are created. (such as RSA keys from PEM_read_bio_RSA_PUBKEY)
  • Remember that openssl calls to encrypt can only encrypt clear text less than a certain size. (256 bytes in my case)
  • Remember that the limit on clear text size does NOT account for padding sizes! RSA_size(key) returns the total size, but with the recommended RSA_PKCS1_OAEP_PADDING the overhead is 41 bytes!

The heart of the C encryption side:

int encrypt_message(unsigned char **encrypted, unsigned char *clear, uint32_t clear_len) {
    RSA *pubkey = rsa_get_public_key();
    int rsa_size = RSA_size(pubkey);
    assert(clear_len < rsa_size - 41);  // 41 is the padding size for RSA_PKCS1_OAEP_PADDING
    ILOG("rsa size = %d\n", rsa_size);
    *encrypted = malloc(RSA_size(pubkey));
    uint32_t encrypted_len = RSA_public_encrypt(clear_len, clear, *encrypted, pubkey, RSA_PKCS1_OAEP_PADDING);
    RSA_free(pubkey);
    if (encrypted_len == -1) {
        fatal("Failed to encrypt data. %s", ERR_error_string(ERR_get_error(), NULL));
    }
    return encrypted_len;
}

rsa_get_public_key() is some openssl gloop code that you should replace, it provides a hard coded public key I generated yesterday for some tests.

The python decryption code is even simpler:

    # This keyfile has had the passphrase removed....
    private_key = M2Crypto.RSA.load_key("/home/karlp/karl.test.private.nopf.key")
    clear_text = private_key.private_decrypt(payload_bytes, M2Crypto.RSA.pkcs1_oaep_padding)
    log.info("Decrypted message: <%s>", clear_text)

You can get the code:

Update: The first edition of this post referred to problems with easily getting the binary payload from the MQTT message. This has since been solved, and will be integrated into a future release of mosquitto.

HMAC signing MQTT messages, with C and python

Posted by KarlP on 15 August, 2011 1 comments

MQTT as a protocol is pretty light on security. There’s a username/password fields, but they are clear text. Mosquitto, the implementation that we’re using right now supports using these fields to limit read/write access to given topics, but it feels a bit hard to manage. We would have to have a file with each topic in use, and multiple lines for each and every user in our system. (We have multiple clients over the internet, and we don’t want to let them see each others messages)

Instead, we’re going to have clients use our public key to encrypt their messages, and then use a shared key per client to sign the messages. This means that we can be sure that the message was not tampered with, that only someone with knowledge of a certain client’s key can pretend to be that client, and that no-one without our private key can actually read the messages. Our listener will first verify the signatures on each message. If a messages was received on topic “clientAAA” but fails to verify with the shared secret key of clientAAA, then we drop the message before even attempting to decrypt. If they match, we can decrypt, and process the contents.

I’ve put together a basic demo of the HMAC signing part, with a message publisher in C, and a message verifier/parser in python. Hopefully it will be useful to someone.
The code is available:

Binary payloads with Mosquitto v0.12 in python

Posted by KarlP on 9 August, 2011 None comments

Update 2011-09-28 This post only applies to Mosquitto v0.12. From v0.13, the ctypes extraction is done by mosquitto itself, and the msg.payload is a proper python string. msg.payload_len and msg.payload_str disappear.

Prior to 0.12, mosquitto‘s python bindings converted the message payload to a string. This was very convenient, you defined a simple on_message handler like so:

def on_message(msg):
    print "Received message on topic:%s, body: %s" % (msg.topic, msg.payload)

This was all well and good, but if you were receiving binary data on a topic, you were out of luck. This has now been rectified, but it is a breaking change for people expecting string data. To keep interpreting the payload as a string, just use msg.payload_str instead of msg.payload

But, how about that binary data? If you don’t get this right first time, you might get quite confused about all the ctypes magic types that keep showing up, but it’s not really that difficult when you get it worked out.

def on_message(msg):
    print "Received message on topic %s, length: %d bytes" % (msg.topic, msg.payloadlen)
    print "As binary data (hex):   ",
    for i in range(msg.payloadlen):
        print "%s " % (hex(msg.payload[i])),
    print ""
    print "As binary data (chars): ",
    for i in range(msg.payloadlen):
        print "   %c " % (chr(msg.payload[i])),
    print ""
    print "As string data (msg.payload_str): %s" % (msg.payload_str)

And, the output after sending a message witha \0 character in the middle:

  Received message on topic test, length: 10 bytes
  As binary data (hex):    0x68  0x65  0x6c  0x6c  0x6f  0x0  0x6b  0x61  0x72  0x6c  
  As binary data (chars):     h     e     l     l     o          k     a     r     l  
  As string data (msg.payload_str): hello

There, not so hard now, and your code didn’t need to know anything about c types :)

MQTT dissector / decoder for Wireshark

Posted by KarlP on 27 June, 2011 12 comments

While debugging some problems we were having with TCP performance, I wanted a way to visualize the MQTT traffic stream a little better in wireshark. I found Wireshark Generic Dissector and thought it should be useful, seeing as I had no desire to start writing C code for packet decoding. WSGD looks to be pretty interesting for writing decoders for private protocols and the like, but there aren’t many other examples of how it’s used. I got helpful prompt help from the lead developer though, so even though it’s a little arcane, it’s still something I can recommend :)

MQTT decoding in Wireshark through WSGD

MQTT decoding in Wireshark through WSGD

My decoding isn’t complete, by any means, but given the complete lack of any other examples out there, I thought this would probably be helpful, even in it’s current state. You’ll need to install WSGD as per the instructions at that site, and then you’ll need this zip….

Just follow the instructions here and by all means, let me know how you go :)

Update: 2014-02-01: There’s an alternative lua plugin, available on github. I haven’t tried it, but lua plugins are actually easier to use and extend. If I’d know about the lua plugin style (and known lua) at the time, I would have done it that way. Note, I haven’t actually _tried_ the lua plugin yet :)