Support secure boot and flash encryption

[dzungpv]

Now only ESP-IDF support secure boot and flash encryption: https://esp32.com/viewtopic.php?t=10029
Please add support for it in platformio

[Nexus1212]

I am also interested in this function!
Could you add this?

[jeremiahlandi]

I am casting my vote for this feature as well. (+1)

[robertpoll]

+1 here too - would be great to have it added.

In the meantime though it's possible (although a bit fiddly) to get it to work by creating a bootloader in a ESP-IDF project and flashing it to your ESP32. Then you can compile your PIO project, sign the firmware.bin using espsecure.py and flash it using esptool.py.

I've only tried it with WITHOUT hardware secure boot but can't see a reason why that couldn't be made to work too. Obviously you need to make sure you don't use the PIO upload as this will overwrite to bootloader.

If there is interest, and proper implementation in PIO is a way off I could probably write up some instructions.

[Nexus1212]

@robertpoll
Which partitions table did you use in PIO to generate the firmware.bin and which one in esp-idf? That should be the same?

Did you also try to flash a ota sketch? In that case it could be possible to flash it again via arduino ide?

[robertpoll]

I did use the same partition table in PIO and esp-idf although I suspect that they may not have to be the same as the partition table gets flashed to the ESP32 when you download the firmware. I'll run a test though as it's an interesting question. This is the partition table I used:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x100000,
app1,     app,  ota_1,   0x110000,0x100000,
spiffs,   data, spiffs,  0x210000,0x1f0000,

The partitions are set up for OTA as you can see although I've only tested serial flashing at the moment. My interest is OTA from an HTTP server for production updates rather than for development - I don't use Arduino IDE but will test that too if that's what you're interested in..?

[robertpoll]

Just run a quick test with the ArduinoOTA library and it seems to work (I used the OTAWebUpdater example but with the Arduino IDE 1.8.12) and a bootloader built with the partition table above (which is different form the Arduino one). @Nexus1212 is this what you're aiming to do?

[Nexus1212]

Hello robertpoll,
that is great :) That is the workaround I have looking for. I want to try it myself:

1. The ota work, but the esp can not use the new flash data. I get the following error message via serial monitor:

Start updating sketch
E (164482) esp_image: image at 0x150000 has invalid magic byte
E (164482) boot_comm: image has invalid chip ID, expected at least 0, found 14168
E (164485) boot_comm: image has invalid chip revision, expected at least 1, found 142
Error[4]: End Failed

The reaseon is the new data, created by ota update in the partition "app1", are not encrypted. That means, ota via arduino ide or plattfomIO wirte the data in planetext in the flash...

Did you uptade the esp via OTA with a preencypted app over your OTAWEbUpdater?
Do you get also an error?

Btw.: I found allready out it is possible to flash ota with PIO. Arduino ide is no needed....

2. OK, I have another idea: What do you think about to change the default bootloader of PlatformIO, with a bootloader of esp-idf with security feature on?
   The Bootloader plattformIO should be:
   C:\Users...\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\tools\sdk\bin

We can try to flash with plattformIO a sketch with supported OTA update and the changed bootloader. After flashing the esp32 chip should encrypt and sign the bootloader and applikation. Updates are possible over ota. What do you think?

3. In the manual of the esp32 flash encryption is written, if the falsh encryption is "on" the bootloader becomes bigger. That mean, I have to change the bootloader offset to 9000. Did you the same? In that case should your partitions table not work... Maybe you have activated the option "require flash encryption to be already enabled"?

[robertpoll]

1. You need to sign the new firmware.bin before you upload it using the .pem file you used to create the bootloader:

espsecure.py sign_data -v1 --keyfile secure_boot_signing_key.pem --output ./firmware-signed.bin ./firmware.bin

Just a minor point, it's signed and not encrypted. I haven't looked at the flash encryption option at this point but presumably that's possible too.

2. Are you suggesting just overwriting one of the pre-compiled boot loaders that come with PIO? I think that would probably work but as the key is complied into the bootloader you'd need to switch the file over when changing projects (assuming you don't want projects to share a key!). I think a simpler option is to set up a simple esp-idf project outside of PIO with the boot verification turned on and just flash that. Then you can build your real project in PIO with ArduinoOTA in and just flash the firmware.bin once you've signed it as above. From then on you can use OTAWebUpdater as this doesn't try to update the bootloader. Does that make sense?

3. No, I reduced the log level on the bootloader to Error, and then it fits in the existing bootloader partition. You should able to enlarge it but the app partitions have to be aligned on 64kB boundaries I think so app_0 would need to be 0x20000 and app_1 to 0x120000 (for example). You'd also need to change the partition table location in menuconfig.

I think this should work well where you want to push firmware updates infrequently. If you're trying to use as part of a development process it might be a bit cumbersome....?

[Nexus1212]

1. Secure boot is not active. It is only flash encryption active. That means to sign the bootloader should not solf my problem!?
   My problem is if I want to flash via ota a firmware, I receive a error. I am using the "OTA Web Update Sketch".

If I choose the firmware in the "serverIndex"-Page and push the "update" Button I get the errormeassage on serial monitor:

Update: firmware.bin
E (140313) esp_image: image at 0x110000 has invalid magic byte
E (140314) boot_comm: image has invalid chip ID, expected at least 0, found 7465
E (140316) boot_comm: image has invalid chip revision, expected at least 1, found 247
Could Not Activate The Firmware
ets Jun  8 2016 00:22:57

The firmware is in plaintext, created by PIO (E:\Eigene Dateien\Documents\PlatformIO\Projects\OTA Web Update.pio\build\esp32dev). Is there a hint?

I created with "esptool.py --port COM7 -b 921600 read_flash" a dump of the flash and checked with a Hex-Editor it. The data on app0 are encrypted, the data on app1 are in planetext.

I think it is required to pre-encrypt the firmware for ota flasching. But if I try it, it does not work :(
Error:

Update: otaWebUpdateApp1-pre-encrypted.bin
Wrong Magic Byte
Wrong Magic Byte
Wrong Magic Byte

I used this command for the pre-encryption:
espsecure.py encrypt_flash_data OTAServer1.bin -k my_flash_encryption_key.bin -o OTAServer1-enc.bin -a 0x110000

Could please anyone help me?

2. Hmm, I think also... the risk is to high to forget to change back the bootloader in PIO ^^

3. That is also the better way. I get a lot of more trouble if I change the partion table (WiFi do not work...) Now, I reduced the loglevel of the bootloader and everything is working fine :)

[Nexus1212]

Hello robertpoll,
why the ota feature is working in your test? But not in mine?

Please have a look on my steps and tell me if there is anything wrong:
(Board: ESP32, Flash encryption allready "on" in development mode, secure boot "off")

1. Create a project "Hello-World" from the exampels with esp-Idf V4.0
2. With menuconfig activate flash encryption, 4mb flash, log on error and flashmode qio
3. flash with esp idf to the esp: idf.py encrypted-flash (the chip is allready encrypted in development mode)
4. Create a new Firmware from the exampels OTAWebUpdater with PIO
5. Two ways, both are working:
   -Flash the new Firmware without bootloader in planetext with the esp-idf: idf.py encrypted-app-flash
   -Pre-encrypt the Firmware with espsecure and flash it with esptool
6. Open the "webinterface", choose the nearly same firmware again in planetext and push update:

Update: firmware.bin
E (140313) esp_image: image at 0x110000 has invalid magic byte
E (140314) boot_comm: image has invalid chip ID, expected at least 0, found 7465
E (140316) boot_comm: image has invalid chip revision, expected at least 1, found 247
Could Not Activate The Firmware
ets Jun  8 2016 00:22:57

The only different what I see is, you use ardnino ide. But I think that is not the reason?

[robertpoll]

Hi @Nexus1212

My example that was working was for signed firmware/OTA without flash encryption - apologies for the confusion.

I've just tried ArduinoOTA on a flash encrypted ESP and it also doesn't work. The docs seem to indicate that the default is that you don't encrypt the firmware before sending via OTA - remember that the encryption key is created on the ESP the first time you flash it (in eFuse BLK1), so you won't know it. The firmware gets encrypted by the ESP once it gets there. The docs say:

OTA updates to encrypted partitions will automatically write encrypted, as long as the esp_partition_write function is used.>
so I'm guessing ArudinoOTA doesn't use that function - a very brief look at Updater.cpp seems to confirm it's using ESP.flashWrite().

Seems like there are the following options:

1. As you suggest, using pre-encrypted firmware. When you tried it did you burn your encryption key to the eFuse? If you did had the ESP you used ever run encrypted flash before? It's not completely clear but it seems likely that a key can only be wrote once so if the ESP auto generated one you might be stuck with it.

2. I haven't tested it but you could use the esp-idf OTA code instead of ArduinoOTA. There's an example that fetches firmware from a web server that looks like it wouldn't be much work to incorporate into an Arduino project.

3. Modify ArduinoOTA to use the encrypted flash function - this might be a simple thing to test as it looks like the functions have the same signatures.

Let me know what you think - I should be able to test option 3 today.

[Nexus1212]

Thank you for your replay :)

1. Yes, I created the key and burn it manual with esptools. And yes, the esp did allready his first selfs encryption. And also yes, it is only possible to burn the key one times.

2. Do you mean this one? To test that a webserver is required... I'm not realy experienced in that! It is possible to test it with a simple network storage?

3. That sounds difficult! If I understand the manual right, the only way to write pre-encrypted firmware is "esp_spi_flash_write" . The next problem could be the ota updater checks the first byte (magic byte). It should be allways "E9", but if the app is encrypted the first byte is also encrypted (!=E9)... The next thing is, you have to know which partition table is active, because you have to set in espsecure the memory adresse for pre-encryption... There ara a lot of problems.... But if it works, it is possible to share encrypted updates, that can only used on your own devices :) I think that is a gread function!!!

[robertpoll]

1. Do you have a new ESP to try with?
2. Yes, that's the one. It's not too tricky to set up a simple web server, but see below...
3. I've managed to get this to work. You're able to decrypt the flash on read using spi_flash_read_encrypted() so checking the magic byte isn't a problem in itself. However, the code as is writes 0xFF to the magic byte initially and then overwrites with 0xe9 when it's successfully written the entire flash. This works because you can change the bits 1->0 but of course when 0xFF is encrypted some of the bits are 0. At the moment I'm reading back the entire sector, erasing it and re-writing with the correct magic byte which works but isn't efficient. You have to write in 16 byte blocks to encrypted storage so am thinking that not writing the first 16 byte block (saving them in a variable) and writing them later in _enablePartition().

Just one other thing though, depending on your security requirements, the OTAUpdater as is uses http not https so the unencrypted firmware goes across the wire so it's only protected once it gets to your device.

[Nexus1212]

1. Yes, I allready did it:
   
   Or do you mean another test? I have one esp32 more :)

2. That sounds not practicable ^^ But thank you very much for your test!

Just one other thing though, depending on your security requirements, the OTAUpdater as is uses http not https so the unencrypted firmware goes across the wire so it's only protected once it gets to your device.

If it is possible to send and flash the encrypted firmware via the http, the firmware should be save (Because encrypted, only device can de-encrypt)! That means, the main goal should be to flash via ota encrypt. But the libarys does not support that... I think we are needing a big change in the arduino OTA lib for that....

[robertpoll]

Hi,

1. Yes, if you have a fresh one it would be a good test.
2. So I've now made the modifications to ArduinoOTA and it's working. It's backward compatible with non encrypted flash also. The modifications are in Updater.cpp actually and you can download them from here. All you need to do is drop the .cpp and .h into your src directory and the compiler should find them.

I was about to submit a pull request to see if they are willing to incorporate the changes - there's an argument that it's an incomplete solution so that may not happen but worth having the discussion. I notice though that there's another PR in to do the same thing, although it looks to me as if it has a problem with the re-writing of the magic byte.

Anyway, in the meantime you're welcome to use the version below - Caveat Emptor but if you do then please let me know how it goes, and if you have any problems.

[Nexus1212]

Hello robertpoll,

I have a lot of new ESP32. Please tell me what exact should I test? I encypted allready a ESP32, burned a key and flashed the ota example.

I tested with your new update-lib the ota example again and it works now :) Now it is poosible to flash a new firmware in planetext on a encrypted esp via ota! Thats great! I hope they accapt your pull request. Could you send me the link to the pull request? I want to track the status :)

I tested also to flash a encrypted firmware. That does not work, because the wrong magic byte is detected. Do you mean it makes sense, to deactivate the check for the first byte bevor flashing?

Best regards :)

[robertpoll]

Hold off on the testing - you're right about the issue with the magic byte checking for flashing pre-encrypted firmware. Turning it off won't work with my changes as it will get re-encrypted. It's not possible for Updater.cpp to automatically tell whether it's dealing with encrypted firmware so maybe an option to turn off the checking and save directly even if the partition is marked as encrypted. It means changing ArduinoOTA to add the option.

Great that the change is working for you - I'll look at the above before submitting the PR, but will let you know when I do.

[Nexus1212]

I found another problem:

Spiff does not work right.
I use spiff to provide html-pages. If the firmware is encrypted, the page-sending does not work.

The problem is in the "wifiClient.cpp"

size_t WiFiClient::write(Stream &stream)
{
    uint8_t * buf = (uint8_t *)malloc(1360);
    if(!buf){
        return 0;
    }
    size_t toRead = 0, toWrite = 0, written = 0;
    size_t available = stream.available();
    while(available){
		
        toRead = (available > 1360)?1360:available;
        toWrite = stream.readBytes(buf, toRead);
        written += write(buf, toWrite);
        available = stream.available();
    }
    free(buf);
    return written;

If encryption is active, "available = stream.available();" becomes never "0" and ends in an endless loop. I think the reason is a problem in the encryption?

I am not sure how the "Stream-available" Function work:
https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/StreamString.cpp

[robertpoll]

I've just tested basic SPIFFS here with the new Updater.cpp and it seems OK. Are you running an encrypted SPIFFS partition? Also are you updating your SPIFFS OTA or have you just flashed it over serial?

[Nexus1212]

No, the Spiff partition is not entcrypted.
I have just flashed it over serial!

It is possible to use spiff, but only the stream function does not work...

[robertpoll]

I think all SPIFFS file access is via streams - could you put together a little example for me that I can test. Also let me know what versions of Arduino and ESP-IDF you're using. Presumably this isn't related to the Updater changes - so it doesn't work with the unmodified libraries either?

[Nexus1212]

Short exampel:

#include <Arduino.h>

#include <WiFi.h>
#include <WiFiClient.h>
#include <SPIFFS.h>
#include <DNSServer.h>
#include <WebServer.h>

DNSServer dnsServer;
WebServer server(80);

void setup() {
  Serial.begin(115200);
  while (!Serial) Serial.println("Setup Start");

  WiFi.mode(WIFI_AP);
  WiFi.softAP("testAP");

  delay(100);

  IPAddress apIP(8, 8, 8, 8); 
  IPAddress netMsk(255, 255, 255, 0);
  WiFi.softAPConfig(apIP, apIP, netMsk);

  dnsServer.setErrorReplyCode(DNSReplyCode::NoError);
  dnsServer.start(53, "*", apIP);
  if (!SPIFFS.begin(true)) {
    Serial.println("Filesystem Erros");
  }
  else{
    File root = SPIFFS.open("/");
    File file = root.openNextFile();
    while(file){
      Serial.print("FILE: ");
      Serial.println(file.name());
      file = root.openNextFile();
    }
  }

    server.on("/index", []() {
    Serial.println ("call-index");
    File file;
    String path = "/index.html";
    String content = "text/html";
    if (SPIFFS.exists(path)){
        file = SPIFFS.open(path, "r"); // Open it
        server.streamFile(file, content); // And send it to the client
        file.close();
        }
        else Serial.println ("file not found");
      });

  server.onNotFound([]() {
    server.send(404, "text/html", "<h1>404</h1><p>Page not found...</p>")
  });
  server.begin();
}

void loop() {
dnsServer.processNextRequest();
server.handleClient();
}

index.html (put the file in "data"):

<h1>Testpage</h1>

<p>Spiff is working</p>

Case 1:
-flash this sketch and the file "index.html" via PIO
-log in smartphone in wiFi "testAP"
-open the page "testtest.com/index"
-->My smartphone shows the testpage

Case 2:
-flash a exampel project with encrypted flash bootloader
-flash this sketch via idf.py (encrypted-app-flash) and the file "index.html" via PIO
-log in smartphone in wiFi "testAP"
-open the page "testtest.com/index"
-->My smartphone shows nothing, the esp32 is in a endless loop...

I am using the newest PIO version:

• framework-arduinoespressif32 2.10004.191002 (1.0.4)
• tool-esptoolpy 1.20600.0 (2.6.0)
• toolchain-xtensa32 2.50200.80 (5.2.0)

ESP-IDF:
-V4.0

[robertpoll]

Just tested this and it's working for me on the ESP with encrypted boot. Is your PIO project using the same partition table that you used for the esp-idf project that you used to create/flash the encrypted bootloader? I'm using the same PIO version as you, and the latest esp-idf.

[Nexus1212]

The partition table was not the same! I used in esp-idf your table and in PIO the default.csv... I forget to change it... Sorry!

OK, now Spiff works! Thank you!

I tested both ota sketches with firmware in plainetext. Both are working (spiff, encrypted preferences, wifi, webserver, websocket and many other libs... ). All works great AND not more slowlier...

The last nice feature is a possibility to flash pre-encrypted firmware via OTA ^^ Do you see a solution for it? :)

[robertpoll]

1. Great that you got it working. I'll submit a PR for the change to the Arduino project.

2. I've looked at the pre-encrypted option and the problem is that the encryption scheme depends on the location of the firmware in flash - so for example firmware encrypted using espsecure.py encrypt_flash_data --address 0x10000 ... won't run if loaded to a different address in flash. Obviously with OTA you don't know which partition you'll be loaded to so can't reliably encrypt the file.

Taking a step back - if the objective is to protect the firmware on its way to the ESP then probably using HTTPS is a better option. There's a HTTTPS web server here that is probably worth a look.

[Nexus1212]

Yes, it depends on the location in flash. A solution could to provide always two ota firmware-updates (one for each ota partition) and the esp pick the right one in dependence which patition on the esp is currently active :)

[Aietes]

It would be great to see flash encryption as a feature in PlatformIO! Other things like encrypting the NVS also rely on this feature. The fact that encryption can be circumvented (with considerable effort by a pro) should not discourage developers from using it, since it's still better to have some level of security as opposed to none.

[devrim-oguz]

Any updates on this?

[roysG]

any update?

[MMI]

Another 👍 vote

[hyzgit]

any update?

[kfine100]

I also needed to do OTA updates with flash encryption and updater.cpp would corrupt the update. I downloaded robertpoll's updater.cpp above and it solved the problem. It would be nice to see flash encryption as a PIO update........any news on this?

[Gabriel-Gardin]

Another 👍 vote

[Koxx3]

one more ! we really need a tutorial for this 👍

[kfine100]

As I stated above, I am able to use Robert Poll's updater.cpp to load firmware over OTA. One thing that caught me is I forgot to enable flash encryption on boot, and in this case the updater can't even get started. So make sure to enable it in menuconfig.

This would not be secure if the download is over an unencrypted http server. So, I connect with https using WiFiClientSecure, defining it as
WiFiClientSecure http_client_Secure;
Then attaching the appropriate certificate for my server. Then calling Robert Poll's updater.cpp like this:
written = Update.writeStream(http_client_Secure);
Looking at the packets with wireshark I don't see any plaintext. So, what the discussion of using HttpsOTAUpdate above when it seems less convenient? Maybe someone can enlighten me as to why it would be more secure?

[bwjohns4]

What's the latest on secure boot from PIO's espidf framework?

[viniciusdesa]

Looking forward for updates in this topic!

[dzungpv]

Years later, I still looking this feature in PI for some project base on Arduino.

[chrisdimoff]

just hopefully nudging this

[Nathan-ma]

Happy new year folks! Its been half a decade so far! It was a pleasure to participate on this topic with you all!

See you guys on 2030!

[viniciusdesa]

Yea, we will send humans to Mars before this feature get released, you bet!

[Jason2866]

There will probably no new feature to espressif stuff added. See #1225
and look in other issues and PRs. Development is reduced to minimum