Merge pull request #60 from feranick/master

 Update libedgetpu repo for compatibility of recent versions of Tensorflow.

Makefile

@@ -47,8 +47,10 @@ STRIPPED_SUFFIX = .stripped
 endif
 
 EDGETPU_RUNTIME_DIR := /tmp/edgetpu_runtime
-LIBUSB_URL := https://github.com/libusb/libusb/releases/download/v1.0.24/libusb-1.0.24.7z
-LIBUSB_SHA256 := 620cec4dbe4868202949294157da5adb75c9fbb4f04266146fc833eef85f90fb
+#LIBUSB_URL := https://github.com/libusb/libusb/releases/download/v1.0.24/libusb-1.0.24.7z
+#LIBUSB_SHA256 := 620cec4dbe4868202949294157da5adb75c9fbb4f04266146fc833eef85f90fb
+LIBUSB_URL := https://github.com/libusb/libusb/releases/download/v1.0.27/libusb-1.0.27.7z
+LIBUSB_SHA256 := 19835e290f46fab6bd8ce4be6ab7dc5209f1c04bad177065df485e51dc4118c8
 USBDK_URL := https://github.com/daynix/UsbDk/releases/download/v1.00-22/UsbDk_1.0.22_x64.msi
 USBDK_SHA256 := 91f6f695e1e13c656024e6d3b55620bf08d8835ef05ee0496935ba6bb62466a5
 LIBEDGETPU_BIN ?= $(MAKEFILE_DIR)/out/

README.md

@@ -8,29 +8,48 @@ This software is distributed in the binary form at [coral.ai/software](https://c
 
 There are three ways to build libedgetpu:
 
-* Docker + Bazel [Recommended]: Compatible with Linux, MacOS and Windows (via Dockerfile.windows and build.bat), this method ensures a known-good build enviroment and pulls all external depedencies needed.
+* Docker + Bazel: Compatible with Linux, MacOS and Windows (via Dockerfile.windows and build.bat), this method ensures a known-good build enviroment and pulls all external depedencies needed.
 * Bazel: Supports Linux, macOS, and Windows (via build.bat). A proper enviroment setup is required before using this technique.
 * Makefile: Supporting only Linux and Native builds, this strategy is pure Makefile and doesn't require Bazel or external dependencies to be pulled at runtime.
 
-### Bazel + Docker
+### Bazel + Docker [Recommended]
+
+For Debian/Ubuntu, install the following libraries:
+```
+$ sudo apt install docker.io devscripts
+```
 
 Build Linux binaries inside Docker container (works on Linux and macOS):
 ```
-$ DOCKER_CPUS="k8" DOCKER_IMAGE="ubuntu:18.04" DOCKER_TARGETS=libedgetpu make docker-build
-$ DOCKER_CPUS="armv7a aarch64" DOCKER_IMAGE="debian:stretch" DOCKER_TARGETS=libedgetpu make docker-build
+DOCKER_CPUS="k8" DOCKER_IMAGE="ubuntu:22.04" DOCKER_TARGETS=libedgetpu make docker-build
+DOCKER_CPUS="armv7a aarch64" DOCKER_IMAGE="debian:bookworm" DOCKER_TARGETS=libedgetpu make docker-build
 ```
 
 All built binaries go to the `out` directory. Note that the bazel-* are not copied to the host from the Docker container.
 
+To package a Debian deb for `arm64`,`armhf`,`amd64` respectively:
+```
+debuild -us -uc -tc -b -a arm64 -d
+debuild -us -uc -tc -b -a armhf -d
+debuild -us -uc -tc -b -a amd64 -d
+```
+
 ### Bazel
+The version of `bazel` needs to be the same as that recommended for the corresponding version of tensorflow. For example, it requires `Bazel 6.1.0` to compile TF 2.15.0.
 
-For proper environment setup check `docker` directory.
+Current version of tensorflow supported is `2.15.0`.
 
 Build native binaries on Linux and macOS:
 ```
 $ make
 ```
 
+Required libraries for Linux:
+
+```
+$ sudo apt install python3-dev
+```
+
 Build native binaries on Windows:
 ```
 $ build.bat
@@ -42,19 +61,31 @@ $ CPU=armv7a make
 $ CPU=aarch64 make
 ```
 
+To package a Debian deb:
+```
+debuild -us -uc -tc -b
+```
+NOTE for MacOS: Compilation with MacOS fails. Two requirements:
+- install `flatbuffers` (via macports)
+- after failure in compilation, add the following line to the temporary file that is created by bazel in `/var/tmp/_bazl_xxxxx/xxxxxxxxxxxxx/external/local_config_cc/BUILD` line 48:
+```
+"darwin_x86_64": ":cc-compiler-darwin",
+```
+Repeat compilation.
+
 ### Makefile
 
 If only building for native systems, it is possible to significantly reduce the complexity of the build by removing Bazel (and Docker). This simple approach builds only what is needed, removes build-time depenency fetching, increases the speed, and uses upstream Debian packages.
 
-To prepare your system, you'll need the following packages (both available on Debian Bullseye or Buster-Backports):
+To prepare your system, you'll need the following packages (both available on Debian Bookworm, Bullseye or Buster-Backports):
 ```
 sudo apt install libabsl-dev libflatbuffers-dev
 ```
 
-Next, you'll need to clone the [Tensorflow Repo](https://github.com/tensorflow/tensorflow) at the desired checkout (using TF head isn't advised). If you are planning to use libcoral or pycoral libraries, this should match the ones in those repos' WORKSPACE files. For example, if you are using TF2.5, we can check that [tag in the TF Repo](https://github.com/tensorflow/tensorflow/commit/a4dfb8d1a71385bd6d122e4f27f86dcebb96712d) and then checkout that address:
+Next, you'll need to clone the [Tensorflow Repo](https://github.com/tensorflow/tensorflow) at the desired checkout (using TF head isn't advised). If you are planning to use libcoral or pycoral libraries, this should match the ones in those repos' WORKSPACE files. For example, if you are using TF2.15, we can check that [tag in the TF Repo](https://github.com/tensorflow/tensorflow/tree/r2.15) get the latest commit for that stable release and then checkout that address:
 ```
 git clone https://github.com/tensorflow/tensorflow
-git checkout a4dfb8d1a71385bd6d122e4f27f86dcebb96712d -b tf2.5
+git checkout r2.15
 ```
 
 To build the library:

WORKSPACE

@@ -16,6 +16,79 @@ workspace(name = "libedgetpu")
 load(":workspace.bzl", "libedgetpu_dependencies")
 libedgetpu_dependencies()
 
+# ==================================================================
+
+# We must initialize hermetic python first.
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+
+http_archive(
+    name = "bazel_skylib",
+    sha256 = "74d544d96f4a5bb630d465ca8bbcfe231e3594e5aae57e1edbf17a6eb3ca2506",
+    urls = [
+        "https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
+        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
+    ],
+)
+
+http_archive(
+    name = "rules_python",
+    sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
+    strip_prefix = "rules_python-0.26.0",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.26.0/rules_python-0.26.0.tar.gz",
+)
+
+load("@rules_python//python:repositories.bzl", "py_repositories")
+
+py_repositories()
+
+load("@rules_python//python:repositories.bzl", "python_register_toolchains")
+load(
+    "@org_tensorflow//tensorflow/tools/toolchains/python:python_repo.bzl",
+    "python_repository",
+)
+
+python_repository(name = "python_version_repo")
+
+load("@python_version_repo//:py_version.bzl", "HERMETIC_PYTHON_VERSION")
+
+python_register_toolchains(
+    name = "python",
+    ignore_root_user_error = True,
+    python_version = HERMETIC_PYTHON_VERSION,
+)
+
+load("@python//:defs.bzl", "interpreter")
+load("@rules_python//python:pip.bzl", "package_annotation", "pip_parse")
+
+NUMPY_ANNOTATIONS = {
+    "numpy": package_annotation(
+        additive_build_content = """\
+filegroup(
+    name = "includes",
+    srcs = glob(["site-packages/numpy/core/include/**/*.h"]),
+)
+cc_library(
+    name = "numpy_headers",
+    hdrs = [":includes"],
+    strip_include_prefix="site-packages/numpy/core/include/",
+)
+""",
+    ),
+}
+
+pip_parse(
+    name = "pypi",
+    annotations = NUMPY_ANNOTATIONS,
+    python_interpreter_target = interpreter,
+    requirements = "@org_tensorflow//:requirements_lock_" + HERMETIC_PYTHON_VERSION.replace(".", "_") + ".txt",
+)
+
+load("@pypi//:requirements.bzl", "install_deps")
+
+install_deps()
+
+# ==================================================================
+
 load("@org_tensorflow//tensorflow:workspace3.bzl", "tf_workspace3")
 tf_workspace3()
 

api/allocated_buffer.cc

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 #include "api/allocated_buffer.h"
+#include "stddef.h"
 
 #include "port/logging.h"
 

api/allocated_buffer.h

@@ -16,6 +16,7 @@
 #define DARWINN_API_ALLOCATED_BUFFER_H_
 
 #include <functional>
+#include "stddef.h"
 
 namespace platforms {
 namespace darwinn {

debian/changelog

@@ -1,3 +1,30 @@
+libedgetpu (16.0tf2.15.0-1) stable; urgency=medium
+  * Build against TF v2.15.0
+ -- Nicola Ferralis <[email protected]>  Tue, 31 Jan 2024 09:10:34 -0500
+libedgetpu (16.0tf2.13.1-2) stable; urgency=medium
+  * Updated libusb to v1.0.26
+ -- Nicola Ferralis <[email protected]>  Tue, 30 Jan 2024 10:10:34 -0500
+libedgetpu (16.0tf2.13.1) stable; urgency=medium
+  * Build against TF v2.13.1
+ -- Nicola Ferralis <[email protected]>  Sat, 27 Jan 2024 23:40:12 -0500
+libedgetpu (16.0tf2.12.1) stable; urgency=medium
+  * Build against TF v2.12.1
+ -- Nicola Ferralis <[email protected]>  Sat, 27 Jan 2024 23:32:52 -0500
+libedgetpu (16.0tf2.11.1) stable; urgency=medium
+  * Build against TF v2.11.1
+ -- Nicola Ferralis <[email protected]>  Sat, 27 Jan 2024 22:50:51 -0500
+libedgetpu (16.0pp3) stable; urgency=medium
+  * Build against TF v2.10.1
+ -- Nicola Ferralis <[email protected]>  Sat, 27 Jan 2024 22:50:51 -0500
+libedgetpu (16.0pp2) stable; urgency=medium
+  * Build against TF v2.9.3
+ -- Nicola Ferralis <[email protected]>  Sat, 27 Jan 2024 18:55:51 -0500
+libedgetpu (16.0-0ppa1) stable; urgency=medium
+  * Build agaist TF v2.8.4 
+ -- Nicola Ferralis <[email protected]>  Sat, 27 Jan 2024 18:42:08 -0500
+libedgetpu (16.0) stable; urgency=medium
+  * New release
+ -- Coral <[email protected]>  Fri, 09 Jul 2021 12:56:53 -0700
 libedgetpu (16.0) stable; urgency=medium
   * New release
  -- Coral <[email protected]>  Fri, 09 Jul 2021 12:56:53 -0700

debian/rules

@@ -35,6 +35,10 @@ override_dh_auto_install:
 	cp -f $(LIBEDGETPU_BIN)/direct/$(CPU)/$(FILENAME) $(LIB_MAX)/$(FILENAME)
 	ln -fs $(FILENAME) $(LIB_MAX)/$(SONAME)
 
+# Use xz compression for compatibility with Debian
+override_dh_builddeb:
+	dh_builddeb -- -Zxz
+
 # Skip auto build and auto clean.
 override_dh_auto_clean:
 override_dh_auto_build:

docker/Dockerfile

@@ -54,7 +54,7 @@ RUN if grep 'Bionic Beaver' /etc/os-release > /dev/null; then \
         && DEBIAN_FRONTEND=noninteractive apt-get install -y gcc-9 g++-9; \
     fi
 
-ARG BAZEL_VERSION=4.2.2
+ARG BAZEL_VERSION=6.1.0
 RUN wget -O /bazel https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh && \
     bash /bazel && \
     rm -f /bazel

docker/Dockerfile.windows

@@ -28,7 +28,7 @@ RUN choco install -y 7zip
 RUN setx /M PATH $($Env:PATH + ';C:\Program Files\7-Zip')
 
 # Install msys2
-ARG MSYS_VERSION=20200602
+ARG MSYS_VERSION=20240113
 ADD http://repo.msys2.org/distrib/x86_64/msys2-base-x86_64-${MSYS_VERSION}.tar.xz c:/windows/temp
 RUN 7z.exe x c:\windows\temp\msys2-base-x86_64-$env:MSYS_VERSION.tar.xz
 RUN 7z.exe x c:\msys2-base-x86_64-$env:MSYS_VERSION.tar -o"c:\\"
@@ -41,16 +41,16 @@ RUN 7z.exe x -y c:\windows\temp\patch-$env:PATCH_VERSION-x86_64.pkg.tar.xz
 RUN 7z.exe x -y c:\patch-$env:PATCH_VERSION-x86_64.pkg.tar -o"c:\\msys64"
 
 # Install vim (for xxd)
-ARG VIM_VERSION=8.2.0592-1
-ADD http://repo.msys2.org/msys/x86_64/vim-${VIM_VERSION}-x86_64.pkg.tar.xz c:/windows/temp
-RUN 7z.exe x -y c:\windows\temp\vim-$env:VIM_VERSION-x86_64.pkg.tar.xz
+ARG VIM_VERSION=9.1.0000-1
+ADD http://repo.msys2.org/msys/x86_64/vim-${VIM_VERSION}-x86_64.pkg.tar.zst c:/windows/temp
+RUN 7z.exe x -y c:\windows\temp\vim-$env:VIM_VERSION-x86_64.pkg.tar.zst
 RUN 7z.exe x -y c:\vim-$env:VIM_VERSION-x86_64.pkg.tar -o"c:\\msys64"
 
 # Install libusb release package
-ARG LIBUSB_VERSION=1.0.24
+ARG LIBUSB_VERSION=1.0.27
 ADD https://github.com/libusb/libusb/releases/download/v${LIBUSB_VERSION}/libusb-${LIBUSB_VERSION}.7z c:/windows/temp
 RUN 7z x -oc:\libusb c:\windows\temp\libusb-$env:LIBUSB_VERSION.7z
 
 # Install Bazel
-ARG BAZEL_VERSION=4.2.2
+ARG BAZEL_VERSION=6.1.0
 ADD https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-windows-x86_64.exe c:/windows/system32/bazel.exe

workspace.bzl

@@ -6,11 +6,51 @@ load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
 
 # TF release 2.5.0 as of 05/17/2021.
-TENSORFLOW_COMMIT = "a4dfb8d1a71385bd6d122e4f27f86dcebb96712d"
-TENSORFLOW_SHA256 = "cb99f136dc5c89143669888a44bfdd134c086e1e2d9e36278c1eb0f03fe62d76"
+#TENSORFLOW_COMMIT = "a4dfb8d1a71385bd6d122e4f27f86dcebb96712d"
+#TENSORFLOW_SHA256 = "cb99f136dc5c89143669888a44bfdd134c086e1e2d9e36278c1eb0f03fe62d76"
 
-CORAL_CROSSTOOL_COMMIT = "6bcc2261d9fc60dff386b557428d98917f0af491"
-CORAL_CROSSTOOL_SHA256 = "38cb4da13009d07ebc2fed4a9d055b0f914191b344dd2d1ca5803096343958b4"
+# TF release 2.5.3 as of 05/17/2021.
+#TENSORFLOW_COMMIT = "a4dfb8d1a71385bd6d122e4f27f86dcebb96712d"
+#TENSORFLOW_SHA256 = "cb99f136dc5c89143669888a44bfdd134c086e1e2d9e36278c1eb0f03fe62d76"
+
+# TF release 2.8.4 as of 11/14/2022.
+#TENSORFLOW_COMMIT = "1b8f5c396f0c016ebe81fe1af029e6f205c926a4"
+#TENSORFLOW_SHA256 = "2f44f6b57e065a545d16a64a428d91ec2006def421a6ea44940e159048f15f0c"
+
+# TF release 2.9.3 as of 11/14/2022.
+#TENSORFLOW_COMMIT = "a5ed5f39b675a1c6f315e0caf3ad4b38478fa571"
+#TENSORFLOW_SHA256 = "cbed52f7f29fb4eaec3cb146f4aba4f622fdf3d3471c88cdf787428005a854c6"
+
+# TF release 2.10.1 as of 11/14/2022.
+#TENSORFLOW_COMMIT = "fdfc646704c37bdf450525f6ced9d80df86e4993"
+#TENSORFLOW_SHA256 = "094320c39b502bcaa3412744517e21e3cd80f55db1371e6692dfe23eeb3cef04"
+
+# TF release 2.11.1 as of 03/16/2023.
+#TENSORFLOW_COMMIT = "a3e2c692c18649329c4210cf8df2487d2028e267"
+#TENSORFLOW_SHA256 = "9568da04825d949ce51a67d80cc90e239717b39e8de8c14ee9df0af90b04faa0"
+
+# TF release 2.12.1 as of 06/27/2023.
+#TENSORFLOW_COMMIT = "8e2b6655c0c488290179ab90a0daed0f6d3006f7"
+#TENSORFLOW_SHA256 = "31a4ba0abab134e632c8e71ac6d1f92443858f04e8485f51ba7eeaad8d0cf9c4"
+
+# TF release 2.13.1 as of 09/12/2023.
+#TENSORFLOW_COMMIT = "f841394b1b714c5cc5366536411cf146c8c570df"
+#TENSORFLOW_SHA256 = "fa01678847283115e0b359ebb4db427ab88e289ab0b20376e1a2b3cb775eb720"
+
+# TF release 2.14.1 as of 11/10/2023.
+#TENSORFLOW_COMMIT = "99d80a9e254c9df7940b2902b14d15914dbbbcd9"
+#TENSORFLOW_SHA256 = "bede963ce97c4badcbb3149acd7c35a6a4954fa3361b777272a58a300e7e8f1d"
+
+# TF release 2.15.0 as of 11/10/2023. Current
+TENSORFLOW_COMMIT = "6887368d6d46223f460358323c4b76d61d1558a8"
+TENSORFLOW_SHA256 = "bb25fa4574e42ea4d452979e1d2ba3b86b39569d6b8106a846a238b880d73652"
+
+#CORAL_CROSSTOOL_COMMIT = "6bcc2261d9fc60dff386b557428d98917f0af491"
+#CORAL_CROSSTOOL_SHA256 = "38cb4da13009d07ebc2fed4a9d055b0f914191b344dd2d1ca5803096343958b4"
+
+# Crosstool release as of 02/28/2023
+CORAL_CROSSTOOL_COMMIT = "8e885509123395299bed6a5f9529fdc1b9751599"
+CORAL_CROSSTOOL_SHA256 = "f86d488ca353c5ee99187579fe408adb73e9f2bb1d69c6e3a42ffb904ce3ba01"
 
 def libedgetpu_dependencies(
         tensorflow_commit = TENSORFLOW_COMMIT,