ESP-IDF Tutorial: A proposal for verifying data from embedded devices

I'll soon return to the basic tutorials on the ESP-IDF. This tutorial is a one-off on a topic I find interesting.

"How can I validate that data from an embedded device came from that device?"

This sounds like a job for public key cryptography. I'm no crypto engineer by any means, but I can certainly use libraries. The ESP IDF ships with mbedtls, which is a surprisingly comprehensive suite of crypto. The library is owned by some people called ARM mbed, so I am guessing you can run it on any ARM.

Here's the scheme:

  1. Create a public/private key pair (RSA in this example) on the embedded device and store it in memory. In our case in the SPIFFS file system.
  2. Send the public key to the server
  3. Hash and sign every data packet you will send the server with the private key.
  4. Embed the signature as a header in the HTTP request
  5. Validate the signature on the server when receiving data
The remainder of this post will show you how to do those things. The code is kind of intense if you are used to Arduino code, and will be entirely presented in C.

Notes:

  • If you are using IDF 4.0 you will need to add "mbedtls" to you CMakeLists.txt.
  • You may well need a slight increase in the default stack size of 3k or so

Includes

First let's include and define some things that will be common for everything we are doing:
#include "mbedtls/base64.h"
#include "mbedtls/config.h"
#include "mbedtls/platform.h"
#include "mbedtls/entropy.h"
#include "mbedtls/ctr_drbg.h"
#include "mbedtls/bignum.h"
#include "mbedtls/x509.h"
#include "mbedtls/rsa.h"
#include "mbedtls/md.h"

#define VIRC_CRYPTO_OK                  (0)
#define VIRC_CRYPTO_KEY_SIZE            (512)
#define VIRC_CRYPTO_KEY_EXPONENT        (65537)
#define VIRC_CRYPTO_PRIVKEY_FILENAME    ("/storage/rsa_priv.txt")
#define VIRC_CRYPTO_PUBKEY_FILENAME     ("/storage/rsa_pub.txt")
#define VIRC_CRYPTO_SIG_LENGTH          (64)
#define VIRC_CRYPTO_HASH_LENGTH         (128)

Generating an RSA public, private key pair and storing it in SPIFFS

This function generates a key pair and stores them in SPIFFS. This code is extremely long and spends a lot of time carefully checking return values and logging the results.


static const char* TAG = "virc-crypto";

// static variables we use for generating and loading
static mbedtls_rsa_context rsa;
static mbedtls_md_context_t md;
static mbedtls_entropy_context entropy;
static mbedtls_ctr_drbg_context ctr_drbg;
static mbedtls_mpi N, P, Q, D, E, DP, DQ, QP;

static bool
keygen
(void)
{
  ESP_LOGI(TAG, "RSA keygen");
  uint8_t err;

// init ALL the things
  mbedtls_ctr_drbg_init( &ctr_drbg );
  mbedtls_rsa_init( &rsa, MBEDTLS_RSA_PKCS_V15, 0 );
  mbedtls_mpi_init( &N );
  mbedtls_mpi_init( &P );
  mbedtls_mpi_init( &Q );
  mbedtls_mpi_init( &D );
  mbedtls_mpi_init( &E );
  mbedtls_mpi_init( &DP );
  mbedtls_mpi_init( &DQ );
  mbedtls_mpi_init( &QP );
  mbedtls_entropy_init( &entropy );

// We use an ID here that is the mac address of the device.
// It is the "personalization" data. You can use what you like
  err = mbedtls_ctr_drbg_seed(&ctr_drbg, mbedtls_entropy_func, &entropy,
      virc_core_id(), VIRC_CORE_ID_LEN);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_ctr_drbg_seed (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_ctr_drbg_seed");
  
  err = mbedtls_rsa_gen_key(&rsa, mbedtls_ctr_drbg_random, &ctr_drbg,
      VIRC_CRYPTO_KEY_SIZE, VIRC_CRYPTO_KEY_EXPONENT);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_gen_key (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_gen_key");

  err = mbedtls_rsa_export(&rsa, &N, &P, &Q, &D, &E);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_export (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_export");

  err = mbedtls_rsa_export_crt( &rsa, &DP, &DQ, &QP );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_export_crt (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_export_crt");

// Key is now generated, we need to write the various bits to files
// First the public key
  FILE *fpub = fopen(VIRC_CRYPTO_PUBKEY_FILENAME, "wb+");
  if (fpub == NULL) {
    ESP_LOGE(TAG, "failed fopen pub (wb+) (%s)", VIRC_CRYPTO_PUBKEY_FILENAME);
    return false;
  }
  ESP_LOGD(TAG, "ok fopen pub (%s)", VIRC_CRYPTO_PUBKEY_FILENAME);
  err = mbedtls_mpi_write_file( "N = ", &N, 16, fpub );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file pub N= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file pub N=");

  err = mbedtls_mpi_write_file( "E = ", &E, 16, fpub );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file pub E= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file pub E=");
  err = fclose( fpub );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed fclose pub (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok fclose pub");

// Now the private key
  FILE *fpriv = fopen(VIRC_CRYPTO_PRIVKEY_FILENAME, "wb+");
  if (fpub == NULL) {
    ESP_LOGE(TAG, "failed fopen priv (wb+) (%s)", VIRC_CRYPTO_PRIVKEY_FILENAME);
    return false;
  }
  ESP_LOGD(TAG, "ok fopen priv (%s)", VIRC_CRYPTO_PRIVKEY_FILENAME);

  err = mbedtls_mpi_write_file( "N = " , &N , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv N= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv N=");

  err = mbedtls_mpi_write_file( "E = " , &E , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv E= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv E=");

  err = mbedtls_mpi_write_file("D = " , &D , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv D= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv D=");

  err = mbedtls_mpi_write_file("P = " , &P , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv P= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv P=");

  err = mbedtls_mpi_write_file("Q = " , &Q , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv Q= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv Q=");

  err = mbedtls_mpi_write_file("DP = ", &DP, 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv DP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv DP=");

  err = mbedtls_mpi_write_file("DQ = ", &DQ, 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv DQ= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv DQ=");

  err = mbedtls_mpi_write_file("QP = ", &QP, 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv QP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv QP=");

  err = fclose( fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed fclose priv (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok fclose priv");

// Free the heap objects
  mbedtls_mpi_free(&N);
  mbedtls_mpi_free(&P);
  mbedtls_mpi_free(&Q);
  mbedtls_mpi_free(&D);
  mbedtls_mpi_free(&E);
  mbedtls_mpi_free(&DP);
  mbedtls_mpi_free(&DQ);
  mbedtls_mpi_free(&QP);
  mbedtls_rsa_free(&rsa);
  mbedtls_ctr_drbg_free(&ctr_drbg);
  mbedtls_entropy_free(&entropy);

  ESP_LOGI(TAG, "ok RSA keygen");
  return true;
}

Well, that wasn't exactly fun, was it. It did take me a while to work it out from examples in the mbedtls code base and I am sure you can successfully modify it.

Loading the private key for signing

Now that we have stored the key, we need to load it into memory from disk. It is assumed you only need to generate it one time, say on first load, or after reformatting the file system, but you will have to load it every time. We use the same variables above and note that we only load the private key. The public key is not of any use to the embedded device any more, except to send it to the server to use to validate our signatures.

The following function shows how to load the key we just generated into memory.


bool keyload() {
  mbedtls_rsa_init(&rsa, MBEDTLS_RSA_PKCS_V15, 0);
  mbedtls_mpi_init(&N);
  mbedtls_mpi_init(&P);
  mbedtls_mpi_init(&Q);
  mbedtls_mpi_init(&D);
  mbedtls_mpi_init(&E);
  mbedtls_mpi_init(&DP);
  mbedtls_mpi_init(&DQ);
  mbedtls_mpi_init(&QP);

  uint8_t err;

  ESP_LOGI(TAG, "RSA read");
  FILE* fpriv = fopen(VIRC_CRYPTO_PRIVKEY_FILENAME, "rb");
  if (fpriv == NULL) {
    ESP_LOGE(TAG, "failed fopen priv (%s)", VIRC_CRYPTO_PRIVKEY_FILENAME);
    return false;
  }
  ESP_LOGD(TAG, "ok fopen (rb) (%s) priv", VIRC_CRYPTO_PRIVKEY_FILENAME);

  err = mbedtls_mpi_read_file( &N , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv N= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv N=");

  err = mbedtls_mpi_read_file( &E , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv E= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv E=");

  err = mbedtls_mpi_read_file( &D , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv D= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv D=");

  err = mbedtls_mpi_read_file( &P , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv P= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv P=");

  err = mbedtls_mpi_read_file( &Q , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv Q= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv Q=");

  err = mbedtls_mpi_read_file( &DP , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv DP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv DP=");

  err = mbedtls_mpi_read_file( &DQ , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv DQ= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv DQ=");

  err = mbedtls_mpi_read_file(&QP , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv QP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv QP=");

  err = mbedtls_rsa_import( &rsa, &N, &P, &Q, &D, &E );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_import (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_import");

  err = mbedtls_rsa_complete(&rsa);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_complete (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_complete");

  err = mbedtls_rsa_check_privkey(&rsa);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_check_privkey (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_check_privkey");

  mbedtls_mpi_free(&N);
  mbedtls_mpi_free(&P);
  mbedtls_mpi_free(&Q);
  mbedtls_mpi_free(&D);
  mbedtls_mpi_free(&E);
  mbedtls_mpi_free(&DP);
  mbedtls_mpi_free(&DQ);
  mbedtls_mpi_free(&QP);

  ESP_LOGI(TAG, "ok RSA read");
  return true;
}

You can see that it is mostly the exact reverse of what we just did to store the key. We load the individual sections of the key and use them to import into the RSA context and now we can use the key for signing.

Signing data with our key

Now we don't actually sign the data. We take a hash, get a signature of the hash, then base 64 encode it so that we can use it in an HTTP header -- binary blobs make terrible HTTP headers.

// For a given buffer buf, of size len, put the signature into the array sig.
// sig will be base64 encoded and suitable for use as a HTTP header

bool virc_crypto_sign
(uint8_t *buf, size_t len, uint8_t sig[128])
{
  ESP_LOGI(TAG, "md5 sha256 rsa sign");

  uint8_t err = -1;
  
  mbedtls_md_init(&md);
  
  err = mbedtls_md_setup(&md, mbedtls_md_info_from_type(MBEDTLS_MD_SHA256), 0);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_setup (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_setup");

  err = mbedtls_md_starts(&md);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_starts (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_starts");

  err = mbedtls_md_update(&md, buf, len);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_update (len=%d)", len);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_update (len=%d)", len);

  uint8_t hash[32];
  err = mbedtls_md_finish(&md, hash);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_finish (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_finish");

  uint8_t raw[VIRC_CRYPTO_SIG_LENGTH];
  err = mbedtls_rsa_pkcs1_sign(&rsa, NULL, NULL, MBEDTLS_RSA_PRIVATE, MBEDTLS_MD_SHA256, 20, hash, raw);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_pkcs1_sign (%d) ", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_pkcs1_sign");

  mbedtls_md_free(&md);

  size_t olen;
  err = mbedtls_base64_encode(sig, VIRC_CRYPTO_HASH_LENGTH, &olen, raw, VIRC_CRYPTO_SIG_LENGTH);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_base64_encode (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_base64_encode (len=%d)", olen);
  ESP_LOGI(TAG, "ok md5 sha256 rsa sign");
  return true;
}

And we are done. Now what?

You can add the header to an HTTP client and send it along with the data in whatever scheme you have planned.

As the server, we can be relatively (within the bounds of crypto surety) sure of two things:

  1. The same entity that generated and sent us the public key also sent us this data
  2. The data has not been modified en route
These two assurances go a long way in the Internet of Things.

More complete code: https://gitlab.com/afshar-oss/virc_crypto




Popular posts from this blog

PyGTK, Py2exe, and Inno setup for single-file Windows installers

I am lucky, as part of my job, I get to code PyGTK. The downside is of course that I have to deploy these applications on Windows. Now these Windows users (especially at the price they are paying for this stuff) don't install dependencies, don't understand what a scripting language or interpreter is, and frankly they should not have to. Unfortunately, the list of dependencies for a PyGTK application (at the very minimum) is Python, GTK, PyGTK, PyGObject, PyCairo. 5 installers! So very early on in this project I found the way to give them exactly what they want and deserve, a single-file executable installer. There are a few guides online about how to achieve this, but none seem to work, and none are particularly new. How do we achieve this? The toolchain involved consists of two elements: 1. Py2exe website 2. Inno Setup website NOTE: The documentation for all the tools used is extensive, I will not repeat everything in there. Py2exe Py2exe is a distutils extension that searches...
Read more

Kiwi proxy widgets, a common widget API

Image
While we are on the notion of Kiwi , the PyGTK helper library, we should be discussing the proxy widgets. They offer a certain set of advantages over their plain PyGTK counterparts: They have a common signal emission API for change in content They offer a common API for setting and reading their value. They offer validation, and coercion They contain a few general improvements over the plain PyGTK widgets What does this mean? Simply it means you can make any data widget (eg Entry, CheckButton, Combo etc), you can connect its signal "content-changed", you can read the data with widget.read(), and set the value with widget.update(). If you have read any of the previous blogs about adapting Zope.schema fields to widgets, and you are alert, you will notice that this is amazingly useful. It means that we can define an interface for a general InputWidget type, something like: class IInputWidget(Interface): def read(): """Read the value in the widget"""...
Read more

Using threads in PyGTK

Well, all this thought of asynchronicity is getting to me. But since we mentioned how to use subprocesses and Twisted, I think it is only fair to talk a bit about threads. So, what will we be doing? Create an abstraction for running threads in PyGTK Use it Now the thing which we will be wanting to run is a generator, call it "take_really_long". And it will surely block, and might be a bit intensive, and well we obviously need something asynchronous. A generator is a bit nicer to use than a single plain long-running function, because we can update the user interface every time it returns, making it much much easier to do things like Progress Bars. You must remember two things when using threads with PyGTK: 1. GTK Threads must be initialised with gtk.gdk.threads_init: import gtk gtk.gdk.threads_init() 2. Any code that modifies the UI (basically any code) that is called from outside the main thread must be pushed into the main thread and called asynchronously in the main loop, w...
Read more