Adding Image Enhancement for Android

Signed-off-by: Ravi Kumar Neti <[email protected]>

ai-solutions/android/02-ImageEnhancement/README.md

@@ -0,0 +1,131 @@
+# Table of Contents
+
+- [Table of Contents](#table-of-contents)
+- [Introduction](#introduction)
+    + [About "Image Super Resolution"](#about--image-super-resolution-)
+    + [Pre-Requisites](#pre-requisites)
+- [Model Selection and DLC conversion](#model-selection-and-dlc-conversion)
+    + [Model Overview](#model-overview)
+    + [Steps to convert model to DLC](#steps-to-convert-model-to-dlc)
+- [Source Overview](#source-overview)
+    + [Source Organization](#source-organization)
+    + [Code Implementation](#code-implementation)
+- [Build APK file with Android Studio](#build-apk-file-with-android-studio)
+- [Results](#results)
+
+# Introduction
+
+### About "Image Enhancement" 
+
+- Current project is an sample Android application for AI-based Low-light Image Enhancement using [Qualcomm® Neural Processing SDK for AI](https://developer.qualcomm.com/sites/default/files/docs/snpe/index.html) framework. 
+- We have used 4 Models in this Solution
+- This sample enhances a low-light image to make it brighter.
+- DLC models take only fixed input size.
+- If users intend to use a different model in this demo framework, **image pre/post processing will be needed**. 
+- Current pre/post processing is specific to the models used. 
+
+### Pre-Requisites 
+
+- Qualcomm® Neural Processing SDK for AI setup should be completed by following the guide here : https://developer.qualcomm.com/sites/default/files/docs/snpe/setup.html
+- Android Studio to import sample project
+- Android NDK to build native code
+- Install opencv using ```pip install opencv-python```
+
+# Model Selection and DLC conversion
+
+### Model Overview
+
+Please refer to Models repository for model overview
+<TODO> Add public link
+
+### Steps to convert model to DLC
+Please refer to Models repository for model overview
+<TODO> Add public link
+
+# Source Overview
+
+### Source Organization
+
+- <DIR> demo: Contains demo video, GIF 
+- <DIR> enhancement: Contains source files in standard Android app format.
+- app\src\main\assets : Contains Model binary DLC
+- enhancement\src\main\java\com\qcom\enhancement : Application java source code
+- enhancement\src\main\cpp : Application C++(native) source code
+- sdk : Contains openCV sdk (Will be generated using _ResolveDependencies.sh_ )
+
+### Code Implementation
+
+- Model Initialization
+
+   `public boolean loadingMODELS(char runtime_var, String dlc_name)`
+  - runtime_var: Possible options are D, G, C. 
+  - dlc_name: Name of the DLC.
+
+- Running Model
+
+  - Following is the Java Function, that handles model execution. This function iternally calls sub functions to handle pre-processing and post-processing
+
+      `inferSNPE(inputMat.getNativeObjAddr(), outputMat.getNativeObjAddr())`
+       - inputMat is opencv Matrix that contains input image.
+       - outputMat is the destination for the output image
+
+   - C++ function that handles preprocessing for the input image.
+
+       `preprocess(std::vector<float32_t> &dest_buffer, cv::Mat &img, std::vector<int> dims) `
+
+   - C++ function that handles postprocessing after we receive input from model
+
+       `postprocess(cv::Mat &outputimg)`
+
+   - SNPE API function that runs the network and give result 
+
+       `snpe->execute(inputMap, outputMap);`
+
+
+# Build APK file with Android Studio  
+
+1. Clone this repo.
+2. Generate DLC using the steps mentioned.
+3. Run below script, from the directory where it is present, to resolve dependencies of this project.
+
+	`bash resolveDependencies.sh`
+
+   * This script will download opencv and paste to sdk directory, to enable OpenCv for android Java.
+   * This script will copy snpe-release.aar file from $SNPE_ROOT to "snpe-release" directory in Android project.
+
+	**NOTE - If you are using SNPE version 2.11 or greater, please change following line in resolveDependencies.sh.**
+   ```
+	From: cp $SNPE_ROOT/android/snpe-release.aar snpe-release
+	To : cp $SNPE_ROOT/lib/android/snpe-release.aar snpe-release
+	```
+
+4. Import folder VisionSolution3-ImageEnhancement as a project in Android Studio 
+5. Do gradle sync
+6. Compile the project. 
+7. Output APK file should get generated : enhancement-debug.apk
+8. Prepare the Qualcomm Innovators development kit(QIDK) to install the application (Do not run APK on emulator)
+9. Install and test application : enhancement-debug.apk
+
+```java
+adb install -r -t enhancement-debug.apk
+```
+
+10. launch the application
+
+Following is the basic "Image Enhancement" Android App 
+
+1. Select one of the models
+2. Select one of the given images from the drop-down list
+3. Select the run-time to run the model (CPU, GPU or DSP)
+4. Observe the result of model on screen
+5. Also note the performance indicator for the particular run-time in mSec
+
+Same results for the application are shown below 
+
+# Results
+
+- Demo video, and performance details as seen below:
+
+![Demo video.](demo/EnhancementDemo.gif)
+
+###### *Qualcomm Neural Processing SDK and Snapdragon are products of Qualcomm Technologies, Inc. and/or its subsidiaries. AIMET Model Zoo is a product of Qualcomm Innovation Center, Inc.*

ai-solutions/android/02-ImageEnhancement/build.gradle

@@ -0,0 +1,26 @@
+// Top-level build file where you can add configuration options common to all sub-projects/modules.
+buildscript {
+    ext.kotlin_version = '1.6.10'
+    repositories {
+        google()
+        jcenter()
+    }
+    dependencies {
+        classpath 'com.android.tools.build:gradle:7.2.1'
+        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
+        // NOTE: Do not place your application dependencies here; they belong
+        // in the individual module build.gradle files
+    }
+}
+
+allprojects {
+    repositories {
+        google()
+        jcenter()
+    }
+
+}
+
+task clean(type: Delete) {
+    delete rootProject.buildDir
+}

ai-solutions/android/02-ImageEnhancement/enhancement/.gitignore

@@ -0,0 +1 @@
+/build

ai-solutions/android/02-ImageEnhancement/enhancement/build.gradle

@@ -0,0 +1,66 @@
+apply plugin: 'com.android.application'
+
+android {
+    compileSdkVersion 30
+    buildToolsVersion "30.0.3"
+
+    defaultConfig {
+        applicationId "com.qcom.aistack_lowlightenhance"
+        minSdkVersion 26
+        targetSdkVersion 30
+        versionCode 1
+        versionName "1.0"
+
+        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
+        externalNativeBuild {
+            cmake {
+                cppFlags "-std=c++11 -frtti -fexceptions"
+                arguments "-DOpenCV_DIR=" + project(':sdk').projectDir + "/native/jni",
+                        "-DANDROID_TOOLCHAIN=clang"
+                targets "ImageEnhancement"
+            }
+            ndk {
+                abiFilters 'arm64-v8a'
+            }
+        }
+    }
+
+    packagingOptions {
+        pickFirst 'lib/x86/libc++_shared.so'
+        pickFirst 'lib/x86_64/libc++_shared.so'
+        pickFirst 'lib/arm64-v8a/libc++_shared.so'
+        pickFirst 'lib/armeabi-v7a/libc++_shared.so'
+    }
+
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
+        }
+    }
+
+    compileOptions {
+        sourceCompatibility JavaVersion.VERSION_1_8
+        targetCompatibility JavaVersion.VERSION_1_8
+    }
+    ndkVersion '21.4.7075529'
+    externalNativeBuild {
+        cmake {
+            path file('src/main/cpp/CMakeLists.txt')
+        }
+    }
+}
+
+dependencies {
+    implementation fileTree(dir: 'libs', include: ['*.jar'])
+    implementation project(path: ':sdk')
+    implementation 'androidx.appcompat:appcompat:1.2.0'
+    testImplementation 'junit:junit:4.12'
+    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
+    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
+    implementation 'com.google.android.material:material:1.2.1'
+    implementation 'androidx.constraintlayout:constraintlayout:2.0.4'
+    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
+    androidTestImplementation 'com.android.support.test:rules:1.0.2'
+    androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'
+}

ai-solutions/android/02-ImageEnhancement/enhancement/proguard-rules.pro

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile

ai-solutions/android/02-ImageEnhancement/enhancement/src/main/AndroidManifest.xml

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+package="com.qcom.aistack_lowlightenhance">
+
+<application
+    android:allowBackup="true"
+    android:extractNativeLibs="true"
+    android:icon="@mipmap/ic_launcher"
+    android:label="@string/app_name"
+    android:roundIcon="@mipmap/ic_launcher_round"
+    android:supportsRtl="true"
+    android:theme="@style/Theme.Enhancement">
+    <uses-native-library
+        android:name="libcdsprpc.so"
+        android:required="true"/>
+    <activity android:name="com.qcom.aistack_lowlightenhance.SNPEActivity" android:exported="true">
+        <intent-filter>
+            <action android:name="android.intent.action.MAIN" />
+            <category android:name="android.intent.category.LAUNCHER" />
+        </intent-filter>
+    </activity>
+</application>
+
+</manifest>

ai-solutions/android/02-ImageEnhancement/enhancement/src/main/cpp/CMakeLists.txt

@@ -0,0 +1,68 @@
+
+# For more information about using CMake with Android Studio, read the
+# documentation: https://d.android.com/studio/projects/add-native-code.html
+
+# Sets the minimum version of CMake required to build the native library.
+
+cmake_minimum_required(VERSION 3.18.1)
+
+# Declares and names the project.
+
+project("ImageEnhancement")
+
+# Creates and names a library, sets it as either STATIC
+# or SHARED, and provides the relative paths to its source code.
+# You can define multiple libraries, and CMake builds them for you.
+# Gradle automatically packages shared libraries with your APK.
+
+###OPENCV
+#find_package(OpenCV REQUIRED)  ##FAILED, cannot find libcpufeatures.so
+#set(OpenCV_STATIC on)
+#set(OpenCV_DIR C:/Users/shubgoya/Desktop/SNPEworkspace/github_workspace/HRNET_posenet/opencv45/native/jni)
+find_package(OpenCV REQUIRED)
+#INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
+
+
+###INCLUDE_DIRECTORIES
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/inc/)
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/inc/zdl)
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/inc/hpp)
+include_directories(${CMAKE_CURRENT_SOURCE_DIR})
+
+
+add_library( # Sets the name of the library.
+        ImageEnhancement
+
+             # Sets the library as a shared library.
+             SHARED
+
+             # Provides a relative path to your source file(s).
+              inference.cpp inference_helper.cpp Model.h ImageEnhancement.cpp
+              MBLLEN.h MBLLEN.cpp RUAS.h RUAS.cpp SCI.cpp SCI.h StableLLVE.h StableLLVE.cpp ZeroDCE.h ZeroDCE.cpp
+        )
+
+# Searches for a specified prebuilt library and stores the path as a
+# variable. Because CMake includes system libraries in the search path by
+# default, you only need to specify the name of the public NDK library
+# you want to add. CMake verifies that the library exists before
+# completing its build.
+
+find_library( # Sets the name of the path variable.
+              log-lib
+
+              # Specifies the name of the NDK library that
+              # you want CMake to locate.
+              log )
+
+# Specifies libraries CMake should link to your target library. You
+# can link multiple libraries, such as libraries you define in this
+# build script, prebuilt third-party libraries, or system libraries.
+
+target_link_libraries( # Specifies the target library.
+        ImageEnhancement
+
+                       # Links the target library to the log library
+                       # included in the NDK.
+                ${CMAKE_CURRENT_SOURCE_DIR}/../jniLibs/arm64-v8a/libSNPE.so
+
+                       ${log-lib} ${OpenCV_LIBS})

ai-solutions/android/02-ImageEnhancement/enhancement/src/main/cpp/MBLLEN.cpp

@@ -0,0 +1,42 @@
+// -*- mode: cpp -*-
+// =============================================================================
+// @@-COPYRIGHT-START-@@
+//
+// Copyright (c) 2023 of Qualcomm Innovation Center, Inc. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+//
+// @@-COPYRIGHT-END-@@
+// =============================================================================
+//
+// Created by shivmahe on 9/5/2023.
+//
+
+#include "MBLLEN.h"
+void MBLLEN::preprocess(std::vector<float32_t> &dest_buffer, cv::Mat &img, std::vector<int> dims)
+{
+    LOGI("MBLLEN PREPROCESS is called");
+    float * accumulator = reinterpret_cast<float *> (&dest_buffer[0]);
+    cv::Mat resized_img;
+    cv::resize(img,resized_img,cv::Size(dims[2],dims[1]),cv::INTER_CUBIC);
+    LOGI("input image SIZE width%d::%d height%d::%d",dims[1],resized_img.cols, dims[2],resized_img.rows);
+
+    //opencv read in BGRA by default, converting to BGR
+    cvtColor(resized_img, resized_img, CV_BGRA2RGB);
+    LOGI("num of channels: %d",resized_img.channels());
+    int lim = resized_img.rows*resized_img.cols*3;
+    for(int idx = 0; idx<lim; idx++){
+        float inputScale = 0.00392156862745f;
+        accumulator[idx] = resized_img.data[idx] * inputScale;
+    }
+    LOGI("input lim %d", lim);
+}
+
+void MBLLEN::postprocess(cv::Mat &outputimg){
+    LOGI("MBLLEN Class post-process");
+    outputimg.convertTo(outputimg,CV_8UC3, 255);
+}
+
+void MBLLEN::msg()
+{
+    LOGI("MBLLEN Class msg");
+}