ESP-IDF Tutorial: A proposal for verifying data from embedded devices

I'll soon return to the basic tutorials on the ESP-IDF. This tutorial is a one-off on a topic I find interesting.

"How can I validate that data from an embedded device came from that device?"

This sounds like a job for public key cryptography. I'm no crypto engineer by any means, but I can certainly use libraries. The ESP IDF ships with mbedtls, which is a surprisingly comprehensive suite of crypto. The library is owned by some people called ARM mbed, so I am guessing you can run it on any ARM.

Here's the scheme:

  1. Create a public/private key pair (RSA in this example) on the embedded device and store it in memory. In our case in the SPIFFS file system.
  2. Send the public key to the server
  3. Hash and sign every data packet you will send the server with the private key.
  4. Embed the signature as a header in the HTTP request
  5. Validate the signature on the server when receiving data
The remainder of this post will show you how to do those things. The code is kind of intense if you are used to Arduino code, and will be entirely presented in C.

Notes:

  • If you are using IDF 4.0 you will need to add "mbedtls" to you CMakeLists.txt.
  • You may well need a slight increase in the default stack size of 3k or so

Includes

First let's include and define some things that will be common for everything we are doing:
#include "mbedtls/base64.h"
#include "mbedtls/config.h"
#include "mbedtls/platform.h"
#include "mbedtls/entropy.h"
#include "mbedtls/ctr_drbg.h"
#include "mbedtls/bignum.h"
#include "mbedtls/x509.h"
#include "mbedtls/rsa.h"
#include "mbedtls/md.h"

#define VIRC_CRYPTO_OK                  (0)
#define VIRC_CRYPTO_KEY_SIZE            (512)
#define VIRC_CRYPTO_KEY_EXPONENT        (65537)
#define VIRC_CRYPTO_PRIVKEY_FILENAME    ("/storage/rsa_priv.txt")
#define VIRC_CRYPTO_PUBKEY_FILENAME     ("/storage/rsa_pub.txt")
#define VIRC_CRYPTO_SIG_LENGTH          (64)
#define VIRC_CRYPTO_HASH_LENGTH         (128)

Generating an RSA public, private key pair and storing it in SPIFFS

This function generates a key pair and stores them in SPIFFS. This code is extremely long and spends a lot of time carefully checking return values and logging the results.


static const char* TAG = "virc-crypto";

// static variables we use for generating and loading
static mbedtls_rsa_context rsa;
static mbedtls_md_context_t md;
static mbedtls_entropy_context entropy;
static mbedtls_ctr_drbg_context ctr_drbg;
static mbedtls_mpi N, P, Q, D, E, DP, DQ, QP;

static bool
keygen
(void)
{
  ESP_LOGI(TAG, "RSA keygen");
  uint8_t err;

// init ALL the things
  mbedtls_ctr_drbg_init( &ctr_drbg );
  mbedtls_rsa_init( &rsa, MBEDTLS_RSA_PKCS_V15, 0 );
  mbedtls_mpi_init( &N );
  mbedtls_mpi_init( &P );
  mbedtls_mpi_init( &Q );
  mbedtls_mpi_init( &D );
  mbedtls_mpi_init( &E );
  mbedtls_mpi_init( &DP );
  mbedtls_mpi_init( &DQ );
  mbedtls_mpi_init( &QP );
  mbedtls_entropy_init( &entropy );

// We use an ID here that is the mac address of the device.
// It is the "personalization" data. You can use what you like
  err = mbedtls_ctr_drbg_seed(&ctr_drbg, mbedtls_entropy_func, &entropy,
      virc_core_id(), VIRC_CORE_ID_LEN);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_ctr_drbg_seed (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_ctr_drbg_seed");
  
  err = mbedtls_rsa_gen_key(&rsa, mbedtls_ctr_drbg_random, &ctr_drbg,
      VIRC_CRYPTO_KEY_SIZE, VIRC_CRYPTO_KEY_EXPONENT);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_gen_key (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_gen_key");

  err = mbedtls_rsa_export(&rsa, &N, &P, &Q, &D, &E);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_export (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_export");

  err = mbedtls_rsa_export_crt( &rsa, &DP, &DQ, &QP );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_export_crt (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_export_crt");

// Key is now generated, we need to write the various bits to files
// First the public key
  FILE *fpub = fopen(VIRC_CRYPTO_PUBKEY_FILENAME, "wb+");
  if (fpub == NULL) {
    ESP_LOGE(TAG, "failed fopen pub (wb+) (%s)", VIRC_CRYPTO_PUBKEY_FILENAME);
    return false;
  }
  ESP_LOGD(TAG, "ok fopen pub (%s)", VIRC_CRYPTO_PUBKEY_FILENAME);
  err = mbedtls_mpi_write_file( "N = ", &N, 16, fpub );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file pub N= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file pub N=");

  err = mbedtls_mpi_write_file( "E = ", &E, 16, fpub );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file pub E= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file pub E=");
  err = fclose( fpub );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed fclose pub (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok fclose pub");

// Now the private key
  FILE *fpriv = fopen(VIRC_CRYPTO_PRIVKEY_FILENAME, "wb+");
  if (fpub == NULL) {
    ESP_LOGE(TAG, "failed fopen priv (wb+) (%s)", VIRC_CRYPTO_PRIVKEY_FILENAME);
    return false;
  }
  ESP_LOGD(TAG, "ok fopen priv (%s)", VIRC_CRYPTO_PRIVKEY_FILENAME);

  err = mbedtls_mpi_write_file( "N = " , &N , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv N= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv N=");

  err = mbedtls_mpi_write_file( "E = " , &E , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv E= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv E=");

  err = mbedtls_mpi_write_file("D = " , &D , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv D= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv D=");

  err = mbedtls_mpi_write_file("P = " , &P , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv P= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv P=");

  err = mbedtls_mpi_write_file("Q = " , &Q , 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv Q= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv Q=");

  err = mbedtls_mpi_write_file("DP = ", &DP, 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv DP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv DP=");

  err = mbedtls_mpi_write_file("DQ = ", &DQ, 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv DQ= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv DQ=");

  err = mbedtls_mpi_write_file("QP = ", &QP, 16, fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_write_file priv QP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_write_file priv QP=");

  err = fclose( fpriv );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed fclose priv (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok fclose priv");

// Free the heap objects
  mbedtls_mpi_free(&N);
  mbedtls_mpi_free(&P);
  mbedtls_mpi_free(&Q);
  mbedtls_mpi_free(&D);
  mbedtls_mpi_free(&E);
  mbedtls_mpi_free(&DP);
  mbedtls_mpi_free(&DQ);
  mbedtls_mpi_free(&QP);
  mbedtls_rsa_free(&rsa);
  mbedtls_ctr_drbg_free(&ctr_drbg);
  mbedtls_entropy_free(&entropy);

  ESP_LOGI(TAG, "ok RSA keygen");
  return true;
}

Well, that wasn't exactly fun, was it. It did take me a while to work it out from examples in the mbedtls code base and I am sure you can successfully modify it.

Loading the private key for signing

Now that we have stored the key, we need to load it into memory from disk. It is assumed you only need to generate it one time, say on first load, or after reformatting the file system, but you will have to load it every time. We use the same variables above and note that we only load the private key. The public key is not of any use to the embedded device any more, except to send it to the server to use to validate our signatures.

The following function shows how to load the key we just generated into memory.


bool keyload() {
  mbedtls_rsa_init(&rsa, MBEDTLS_RSA_PKCS_V15, 0);
  mbedtls_mpi_init(&N);
  mbedtls_mpi_init(&P);
  mbedtls_mpi_init(&Q);
  mbedtls_mpi_init(&D);
  mbedtls_mpi_init(&E);
  mbedtls_mpi_init(&DP);
  mbedtls_mpi_init(&DQ);
  mbedtls_mpi_init(&QP);

  uint8_t err;

  ESP_LOGI(TAG, "RSA read");
  FILE* fpriv = fopen(VIRC_CRYPTO_PRIVKEY_FILENAME, "rb");
  if (fpriv == NULL) {
    ESP_LOGE(TAG, "failed fopen priv (%s)", VIRC_CRYPTO_PRIVKEY_FILENAME);
    return false;
  }
  ESP_LOGD(TAG, "ok fopen (rb) (%s) priv", VIRC_CRYPTO_PRIVKEY_FILENAME);

  err = mbedtls_mpi_read_file( &N , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv N= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv N=");

  err = mbedtls_mpi_read_file( &E , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv E= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv E=");

  err = mbedtls_mpi_read_file( &D , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv D= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv D=");

  err = mbedtls_mpi_read_file( &P , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv P= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv P=");

  err = mbedtls_mpi_read_file( &Q , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv Q= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv Q=");

  err = mbedtls_mpi_read_file( &DP , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv DP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv DP=");

  err = mbedtls_mpi_read_file( &DQ , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv DQ= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv DQ=");

  err = mbedtls_mpi_read_file(&QP , 16, fpriv);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_mpi_read_file priv QP= (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_mpi_read_file priv QP=");

  err = mbedtls_rsa_import( &rsa, &N, &P, &Q, &D, &E );
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_import (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_import");

  err = mbedtls_rsa_complete(&rsa);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_complete (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_complete");

  err = mbedtls_rsa_check_privkey(&rsa);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_check_privkey (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_check_privkey");

  mbedtls_mpi_free(&N);
  mbedtls_mpi_free(&P);
  mbedtls_mpi_free(&Q);
  mbedtls_mpi_free(&D);
  mbedtls_mpi_free(&E);
  mbedtls_mpi_free(&DP);
  mbedtls_mpi_free(&DQ);
  mbedtls_mpi_free(&QP);

  ESP_LOGI(TAG, "ok RSA read");
  return true;
}

You can see that it is mostly the exact reverse of what we just did to store the key. We load the individual sections of the key and use them to import into the RSA context and now we can use the key for signing.

Signing data with our key

Now we don't actually sign the data. We take a hash, get a signature of the hash, then base 64 encode it so that we can use it in an HTTP header -- binary blobs make terrible HTTP headers.

// For a given buffer buf, of size len, put the signature into the array sig.
// sig will be base64 encoded and suitable for use as a HTTP header

bool virc_crypto_sign
(uint8_t *buf, size_t len, uint8_t sig[128])
{
  ESP_LOGI(TAG, "md5 sha256 rsa sign");

  uint8_t err = -1;
  
  mbedtls_md_init(&md);
  
  err = mbedtls_md_setup(&md, mbedtls_md_info_from_type(MBEDTLS_MD_SHA256), 0);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_setup (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_setup");

  err = mbedtls_md_starts(&md);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_starts (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_starts");

  err = mbedtls_md_update(&md, buf, len);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_update (len=%d)", len);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_update (len=%d)", len);

  uint8_t hash[32];
  err = mbedtls_md_finish(&md, hash);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_md_finish (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_md_finish");

  uint8_t raw[VIRC_CRYPTO_SIG_LENGTH];
  err = mbedtls_rsa_pkcs1_sign(&rsa, NULL, NULL, MBEDTLS_RSA_PRIVATE, MBEDTLS_MD_SHA256, 20, hash, raw);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_rsa_pkcs1_sign (%d) ", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_rsa_pkcs1_sign");

  mbedtls_md_free(&md);

  size_t olen;
  err = mbedtls_base64_encode(sig, VIRC_CRYPTO_HASH_LENGTH, &olen, raw, VIRC_CRYPTO_SIG_LENGTH);
  if (err != VIRC_CRYPTO_OK) {
    ESP_LOGE(TAG, "failed mbedtls_base64_encode (%d)", err);
    return false;
  }
  ESP_LOGD(TAG, "ok mbedtls_base64_encode (len=%d)", olen);
  ESP_LOGI(TAG, "ok md5 sha256 rsa sign");
  return true;
}

And we are done. Now what?

You can add the header to an HTTP client and send it along with the data in whatever scheme you have planned.

As the server, we can be relatively (within the bounds of crypto surety) sure of two things:

  1. The same entity that generated and sent us the public key also sent us this data
  2. The data has not been modified en route
These two assurances go a long way in the Internet of Things.

More complete code: https://gitlab.com/afshar-oss/virc_crypto




Popular posts from this blog

PyGTK, Py2exe, and Inno setup for single-file Windows installers

I am lucky, as part of my job, I get to code PyGTK. The downside is of course that I have to deploy these applications on Windows. Now these Windows users (especially at the price they are paying for this stuff) don't install dependencies, don't understand what a scripting language or interpreter is, and frankly they should not have to. Unfortunately, the list of dependencies for a PyGTK application (at the very minimum) is Python, GTK, PyGTK, PyGObject, PyCairo. 5 installers! So very early on in this project I found the way to give them exactly what they want and deserve, a single-file executable installer. There are a few guides online about how to achieve this, but none seem to work, and none are particularly new. How do we achieve this? The toolchain involved consists of two elements: 1. Py2exe website 2. Inno Setup website NOTE: The documentation for all the tools used is extensive, I will not repeat everything in there. Py2exe Py2exe is a distutils extension that searches
Read more

ESP-IDF for Arduino Users, Tutorials Part 2: Delay

This is the second part of a series of ESP-IDF tutorials that I will complete as I learn stuff. Read part 1. One of the best things about Arduino is the ability to just block for a period with: delay(1000); // hang on a second, buddy It’s also one of the worst things. As soon as you need to do a few things at the same time, you will end up using better systems, like calling millis() or using the elapsedMillis library. Anyway, the IDF has you covered. You can delay, but amazingly it appears that once you multi-task (which you will with Free RTOS), the delay doesn’t block the other tasks from running. My Notes: Read the docs, now or later https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/freertos.html Delay doesn’t have to block like Arduino. Delay some milliseconds like this: vTaskDelay(pdMS_TO_TICKS(10)); Since vTaskDelay accepts ticks, the pdMS_TO_TICKS macro will convert for you. Help yourself out by adding your own macros if you are lazy. I am lazy. #
Read more

How to install IronPython2 with Mono on Ubuntu

This is just an instructional post with the hope that it might help others in the future. I spent a long time trying to get IronPython2 running on Mono. Now IronPython1 ships with Ubuntu, so apt-get install ironpython , if that's all that you want. Ironically, I don't really know the difference between IronPython1 and IronPython2, but "2 has got to be better than 1", right? The good news is that its very possible, but you will have to step out of your comfort zones. You will need: Mono 2.4 (sources) and IronPython 2.0.1 (binary) First build Mono (note that you need Ubuntu's Mono to bootstrap, so apt-get install mono ) but make sure you put it somewhere special so it doesn't mess with the system-installed Mono. This advice is taken from an article on the Mono wiki . $ tar xvjf mono-2.4.tar.bz2 $ cd mono $ mkdir -p ~/opt/mono $ ./configure --prefix=~/opt/mono $ make && make install So this is putting Mono in opt/mono in my home directory (which is wher
Read more