Library for face detection Missing on Arduino core for the ESP32 V3.1

[Sig2018]

Board

ESP32s3 cam Dev Board

Device Description

ESP32s3 cam Dev Board

Hardware Configuration

Camera OV2640

Version

v3.1.0

IDE Name

Arduino IDE

Operating System

Windows 10

Flash frequency

80Mhz

PSRAM enabled

yes

Upload speed

921600

Description

On arduino IDE when switch to board ESP32 version 3.1.x the files for face recognition are missing.
Running compilation will report error:
#include "face_recognition_112_v1_s8.hpp" Not Found.
It works right on ESP32 board version 3.0.7

Sketch

/*
This code is based on:
1. CameraWebServer example
2. Offline version of it with face detection only (without recognition) - https://github.com/espressif/arduino-esp32/issues/9671#issuecomment-2126825897
3. Adafruit's example - https://github.com/adafruit/Adafruit_Learning_System_Guides/tree/main/MEMENTO/Memento_Face_Detect_Recognize

This code demonstrate face detection and recognition on Xiao ESP32S3 Sense. Short blink for face detection, long blink for face reconition.
For enrolling new face you need to short D10 to GND and release. Then, you need to direct the camera to the face till long blink or Serial print indicate the face enrolled.

Notes:
1. You can use CameraWebServer example to enroll faces with web monitor
2. You can use "Erase all flash before sketch upload -> enabled" to remove enrolled faces.
3. It's recommended to enroll few different ID's from different angles for every face to make it easier to recognized.

Tools configurations for uploading: 
  a. Partition scheme: Maximum App
  b. PSRAM: OPI PSRAM

Written by Omri David
*/
#include "esp_camera.h"
#include "face_recognition_112_v1_s8.hpp"
#include "face_recognition_tool.hpp"
#include "fb_gfx.h"
#include "human_face_detect_mnp01.hpp"
#include "human_face_detect_msr01.hpp"
#include "ra_filter.h"
#include <vector>
#include <esp_system.h>
#include <FastLED.h>

String sketch_name = "offline_face_detection_recognition9_organized";

//Esp32s3 CAMERA_MODEL_ESP32S3_EYE  definitions:
#define PWDN_GPIO_NUM -1
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 15
#define SIOD_GPIO_NUM 4
#define SIOC_GPIO_NUM 5

#define Y2_GPIO_NUM 11
#define Y3_GPIO_NUM 9
#define Y4_GPIO_NUM 8
#define Y5_GPIO_NUM 10
#define Y6_GPIO_NUM 12
#define Y7_GPIO_NUM 18
#define Y8_GPIO_NUM 17
#define Y9_GPIO_NUM 16

#define VSYNC_GPIO_NUM 6
#define HREF_GPIO_NUM 7
#define PCLK_GPIO_NUM 13

//for enrolling new faces
uint8_t ENROLL_BUTTON = 21;

// LED WS2812
#define LED_PIN     48
#define NUM_LEDS    1
CRGB leds[NUM_LEDS];

// Colores predefinidos
#define COLOR_WAITING   CRGB::Blue
#define COLOR_NEW       CRGB::White
#define COLOR_UNKNOWN   CRGB::Red
#define COLOR_KNOWN     CRGB::Green
#define COLOR_ENROLLED  CRGB::BlueViolet

// The number of faces to save
// NOTE - these faces are saved to the ESP32's flash memory and survive between
// reboots
#define FACE_ID_SAVE_NUMBER 7

// Threshold (0.0 - 1.0) to determine whether the face detected is a positive
// match NOTE - This value is adjustable, you may "tune" it for either a more
// confident match
#define FR_CONFIDENCE_THRESHOLD 0.6

// True if you want to save faces to flash memory and load them on boot, 
// otherwise False
#define SAVE_FACES_TO_FLASH true



/**** FR and FD ****/

// pointer to the camera's framebuffer
camera_fb_t *fb = NULL;

// Recognizer model
// S8 model - faster but less accurate
FaceRecognition112V1S8 recognizer;
// Use two-stage fd and weights
HumanFaceDetectMSR01 s1(0.1F, 0.5F, 10, 0.2F);
HumanFaceDetectMNP01 s2(0.5F, 0.3F, 5);
bool is_enrolling = false;  // 0: not enrolling, 1: enrolling

// Constantes globales
const int MAX_RETRIES = 3;
const int DELAY_BETWEEN_RETRIES = 500; // ms

int Caranum = 0; 


/**
 * @brief Run face recognition on the framebuffer.
 *
 * @param fb Pointer to the framebuffer data.
 * @param results Pointer to the list of detected faces.
 * @return int The ID of the recognized face.
 */
static int run_face_recognition(fb_data_t *fb, std::list<dl::detect::result_t> *results) {
    // Imprimir estado inicial
    Serial.println("\n=== Inicio de reconocimiento facial ===");
    int initial_count = recognizer.get_enrolled_id_num();
    Serial.printf("IDs en memoria al inicio: %d\n", initial_count);
    
    std::vector<int> landmarks = results->front().keypoint;
    int id = -1;
    
    Tensor<uint8_t> tensor;
    tensor.set_element((uint8_t *)fb->data)
        .set_shape({ fb->height, fb->width, 3 })
        .set_auto_free(false);

    // Si estamos en modo enrolamiento
    if (initial_count < FACE_ID_SAVE_NUMBER && is_enrolling) {
        Serial.println("\n-> Iniciando proceso de enrolamiento");
        // Cuando quieras enrolar un rostro
        Serial.println("Introduce el nombre para este rostro:");

        // Esperamos hasta que haya datos disponibles en el puerto serie
        while (!Serial.available()) {
            delay(100);  // pequeña pausa para no saturar el procesador
        }

        // Leemos el nombre introducido
        String nombreArduino = Serial.readStringUntil('\n');  // Lee hasta encontrar un salto de línea
        nombreArduino.trim();  // Elimina espacios y saltos de línea extras

        // Convertimos el String de Arduino a std::string
        std::string Caratxt = std::string(nombreArduino.c_str());
        id = recognizer.enroll_id(tensor, landmarks, Caratxt, true);
        
        if (id >= 0) {
            Serial.printf("Nuevo rostro enrolado con ID: %d\n", id);
            
            // Verificar estado de la partición 
            Serial.println("Verificando partición...");
            int partStatus = recognizer.check_partition();
            Serial.printf("Estado de la partición: %d\n", partStatus);

            
            // Verificar estado final
            int final_count = recognizer.get_enrolled_id_num();
            Serial.printf("IDs en memoria después del proceso: %d\n", final_count);
            
            updateLED(COLOR_ENROLLED);
            delay(3000);
            updateLED(COLOR_WAITING);
            delay(10);
        } else {
            Serial.println("Error: Fallo en el enrolamiento");
        }
        is_enrolling = false;
    } else if (initial_count >= FACE_ID_SAVE_NUMBER && is_enrolling) {
        Serial.printf("Error: Ya se alcanzó el máximo de rostros (%d)\n", FACE_ID_SAVE_NUMBER);
        is_enrolling = false;
    }

    // Parte de reconocimiento
    face_info_t recognize = recognizer.recognize(tensor, landmarks);
    if (recognize.id >= 0 && recognize.similarity >= FR_CONFIDENCE_THRESHOLD) {
        Serial.printf("\n-> Rostro reconocido:\n");
        Serial.printf("   ID: %d\n", recognize.id);
        Serial.printf("   Similaridad: %.2f\n", recognize.similarity);
        std::string Nombre = recognize.name;
        Serial.printf("   Nombre: %s\n", Nombre.c_str());
        updateLED(COLOR_KNOWN);
        delay(3000);
        updateLED(COLOR_WAITING);
        delay(10);
    } else if (recognizer.get_enrolled_id_num() > 0) {
        Serial.printf("\n-> Rostro desconocido:\n");
        Serial.printf("   Similaridad: %.2f\n", recognize.similarity);
        updateLED(COLOR_UNKNOWN);
        delay(3000);
        updateLED(COLOR_WAITING);
        delay(10);
    } else {
        Serial.println("\n-> No hay rostros enrolados para comparar");
    }

    Serial.println("=== Fin de reconocimiento facial ===\n");
    return recognize.id;
}

void list_partitions() {
  esp_partition_iterator_t it = esp_partition_find(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, NULL);

  while (it != NULL) {
    const esp_partition_t* partition = esp_partition_get(it); 
    printf("Partition: %s\n", partition->label);
    printf("  Type: 0x%x\n", partition->type);
    printf("  Subtype: 0x%x\n", partition->subtype);
    printf("  Address: 0x%x\n", partition->address);
    printf("  Size: %d\n", partition->size);
    it = esp_partition_next(it); 
  }
}

bool initCamera() {
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sccb_sda = SIOD_GPIO_NUM;
  config.pin_sccb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.grab_mode = CAMERA_GRAB_WHEN_EMPTY;
  config.fb_location = CAMERA_FB_IN_PSRAM;
  config.frame_size = FRAMESIZE_240X240;  //FRAMESIZE_240X240;
  config.pixel_format = PIXFORMAT_RGB565;
  //config.fb_count = 2;
  config.jpeg_quality = 12;                   // Solo aplica para JPEG
  config.fb_count = 1;                        // Probar con 1 buffer primero

  // Initialize the camera
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("ERROR: Camera init failed with code 0x%x", err);
    return false;
  }

  // Configure the camera's sensor
  sensor_t *s = esp_camera_sensor_get();
  if (s) {

    s->set_vflip(s, 0);
    s->set_hmirror(s, 0);

}
  // s->set_brightness(s, -1); //causing more: cam_hal: EV-VSYNC-OVF
  // s->set_contrast(s, 1); //causing more: cam_hal: EV-VSYNC-OVF

  return true;
}

void updateLED(CRGB color) {
    leds[0] = color;
    FastLED.show();
}


void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  Serial.println("sketch name: " + sketch_name);


  pinMode(ENROLL_BUTTON, INPUT_PULLUP);

  if (!initCamera()) {
    Serial.println("Camera init failed!");
  }
  delay(3000); // Pequeña pausa para estabilización
  // Inicializar LED
  FastLED.addLeds<WS2812, LED_PIN, GRB>(leds, NUM_LEDS);
  FastLED.setBrightness(50);
  updateLED(COLOR_WAITING);

 
    // Initialize face recognition filter and partition
  ra_filter_init(&ra_filter, 20);

  list_partitions();

  recognizer.set_partition(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY,
                           "fr");
    // Verificar estado inicial de la flash
    Serial.println("\nVerificando partición flash:");
    int partStatus = recognizer.check_partition();
    Serial.printf("Estado de la partición: %d\n", partStatus);

    if (partStatus < 0) {
    Serial.println("Error con la partición, intentando reinicializar...");
    recognizer.set_partition(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, "fr");
    partStatus = recognizer.check_partition();
    Serial.printf("Nuevo estado de la partición: %d\n", partStatus);
    }
    
    Serial.println("\nCargando IDs guardados:");
    int loaded = recognizer.set_ids_from_flash();
    Serial.printf("IDs cargados desde flash: %d\n", loaded);
    
    Serial.println("\nEstado inicial del sistema:");
    int enrolled = recognizer.get_enrolled_id_num();
    Serial.printf("IDs en memoria: %d\n", enrolled);
    
    Serial.println("\n=== Sistema listo ===\n");

  Serial.println("start detecting and recognizing faces");
}

void loop() {
  // If the enroll button is pressed, enroll a new face
  if (digitalRead(ENROLL_BUTTON) == LOW) {
    is_enrolling = true;
    Serial.println("Enrolling face..");
    updateLED(COLOR_NEW);   // 
    delay(1000);                   // wait for a second
    updateLED(COLOR_WAITING);;  // turn the LED to wait color
    delay(1000);
  }
  // Antes de la captura, verificar estado de la cámara
  sensor_t * s = esp_camera_sensor_get();
  if (!s) {
      Serial.println("ERROR: Camera sensor not found!");
      return;
  }
  // Imprimir configuración actual de la cámara
  Serial.printf("Camera Settings:\n");
  Serial.printf("- Resolution: %dx%d\n", s->status.framesize, s->status.framesize);
  // Serial.printf("- Quality: %d\n", s->status.quality);
  // Serial.printf("- Brightness: %d\n", s->status.brightness);
  // Serial.printf("- Contrast: %d\n", s->status.contrast);
  // Verificar PSRAM
  if(!psramFound()) {
      Serial.println("PSRAM not found - Face detection requires PSRAM");
      return;
  }
  // Intento de captura
  Serial.println("Attempting to capture image...");
  // capture from the camera into the frame buffer
  fb = esp_camera_fb_get();
  if (!fb) {
    Serial.printf("ERROR: Camera capture failed\n");
    Serial.println("ERROR: Camera capture failed");
    Serial.println("Debug info:");
    Serial.printf("- ESP Free Heap: %d\n", ESP.getFreeHeap());
    Serial.printf("- ESP Free PSRAM: %d\n", ESP.getFreePsram());
  } else {
    // Face detection:
    Serial.printf("Camera capture ok\n");
    Serial.printf("- Image Size: %d bytes\n", fb->len);
    Serial.printf("- Format: %d\n", fb->format);
    if (fb->buf == NULL) {
        Serial.println("ERROR: Image buffer is NULL!");
    } else {
        // Intentar detección de rostro
        Serial.println("Attempting face detection...");
    
        std::list<dl::detect::result_t> &candidates =
          s1.infer((uint16_t *)fb->buf, { (int)fb->height, (int)fb->width, 3 });
        std::list<dl::detect::result_t> &results = s2.infer(
          (uint16_t *)fb->buf, { (int)fb->height, (int)fb->width, 3 }, candidates);
        if (results.size() > 0) {
          Serial.println("Detected face!");
          updateLED(COLOR_NEW);
          delay(2000);          
          
          int face_id = 0;

          size_t out_len, out_width, out_height;
          uint8_t *out_buf;
          bool s;

          out_len = fb->width * fb->height * 3;
          out_width = fb->width;
          out_height = fb->height;
          out_buf = (uint8_t *)malloc(out_len);
          if (!out_buf) {
            log_e("out_buf malloc failed");
          }

          //convert to rgb888 for better perfomances
          s = fmt2rgb888(fb->buf, fb->len, fb->format, out_buf);
          free(out_buf);
          if (!s) {
            free(out_buf);
            log_e("conversion to rgb888 failed");
          }
          esp_camera_fb_return(fb);

          fb_data_t rfb;
          rfb.width = out_width;
          rfb.height = out_height;
          rfb.data = out_buf;
          rfb.bytes_per_pixel = 3;
          rfb.format = FB_BGR888;

          // Face recognition is SLOW! So, only attempt it if we are enrolling a
          // new face, or have previously enrolled a face
          if (recognizer.get_enrolled_id_num() > 0 || is_enrolling) {
            Serial.println("Reconociendo");
            face_id = run_face_recognition(&rfb, &results);
          } else {
            face_id = 0;
            Serial.println("no enrolled faces -> not running face recognition");
            updateLED(COLOR_WAITING);  // turn the LED to wait color
            delay(500);
          }
        }
      }
  }
  // Release the framebuffer
  esp_camera_fb_return(fb);
}

Debug Message

Not a debug but compilation error #include "face_recognition_112_v1_s8.hpp" Not Found.

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[Jason2866]

Face recognition has been removed in Arduino Core 3.1.x
Last version where it is available is Arduino core 3.0.7

[Sig2018]

Ohh really?
Any replacement or workaround?
I didn't found any reference of the removal on documentation.
@Jason2866 thank you for your reply.

[me-no-dev]

The face detections and recognition library was changed and no longer supports the chips before ESP32-P4 and the old versions that do support them can not be compiled with the new versions of ESP-IDF, so we had to remove support.

[Gfisk25]

So is there any way of still using it??

[Jason2866]

Only with Arduino Core 3.0.7

[CongducPham]

I've read that ESP-DL v3.0 has been released and that normally it supports ESP32S3. Any plan to have back ESP-DL v3.0 in the Arduino ESP core in the future?