Ultra Low Power Person Detection with Multimodal Wakeup Hierarchy for Battery-Powered Security Cameras

This project implements person detection by combining a PIR motion sensor and a monochrome image sensor, aimed at reducing the uptime of the power-hungry primary camera used for recording. All inferences are done at the edge using a neural network built with TensorFlow Lite, applying TinyML techniques such as quantization and model pruning in order to optimize the NN model for microcontroller use. The system is built on an off-the-shelf microcontroller: XIAO ESP32S3 Sense from SEEED Studio, powered by the dual-core Xtensa® 32-bit LX7 MCU.

Team

1. Christopher Yang
2. Ananay Garg

Project Files

• Website
• Proposal
• Midterm Checkpoint Presentation Slides
• Final Presentation Slides
• Final Report
• Demo Videos:
  • Master Demo Video
  • 5V Demo Video
  • Wifi Streaming Demo Video

Software Submodule/Cloning the repo

As the software code of our repo was heavily dervied from exisiting person detection work, We cloned the 'espressif/esp-tflite-micro' repo and added our commits on top of it. All of our commit history related to software code lies in that github repository. This main repo (under software/) contains a reference to another repo that contains all the software code.

You need to do a couple of extra steps to ensure proper initilisation. Once you clone the repo. Please do a 'git submodule init' followed by 'git submodule update' to initialise all the submodules.