STR ISS Image Detector (#98)

* Added images

* Added CRAFT and parseq libraries

* Made parse text file

* Merged CRAFT and parseq

* Added decoding for each boundary box found

* Added tests for other ocr libraries

* infrastructure work on docker processes

* Testing on various images

* making ci pr use the build script

* analyst notebook now depends on entire isaac for python packages

* fixing astrobee path

* sintax + analyst only in ubuntu 20

* Edited image cropping

* Added database and label finder

* Label detection

* Trade Study results

* Fixed imports

* Phrase detection wn working

* Fixed git submodules

* Fixed imports

* Jupyter Notebook outline

* Organization

* Jupyter Notebook OCR implementation

* Working on imports

* Made image str into Python package

* Image_str Python Package

* Optimizing parser

* Formatting

* organizing stuff

* adding some str dependencies

* Imports for Analyst Notebook

* Edited image slicing and similarity parameters

* Formatting

* Removed craft and parseq

* Removed folders

* removed folders

* Added folders to gitignore

* Jupyter Notebook display

* Working Analyst demo

* Fixed overlapping boundary boxes

* Removed images folder

* Organization

* Starting 3D location

* Getting 3D position

* Parsing 3D output

* Removed data.json

* Added graphing for 3D positions of labels

* Graphing of 3D positions of labels

* Graph display

* Stored data in .dat file

* Label .dat file and duplication removal

* Refactored to account for 3D locations

* save results to csv file

* Finding 3D locations

* Working Jupyter Notebook demo

* Removed changes in docker

* Created panorama link

* Started documentation and added bumble data

* Documentation

* Changed image to RGB and fixed find

* Changed how to detect duplicates

* Refactored code into a class

* Changed find result display

* Adding interactive graph

* New visualization and craft model download

* Removed models

* Removed test folders

* Fixed display

* Cleaned up code and documentation

* Finished Documentation

* Added display for running locally

* Speed up find label

* Added Documentation

* Edited Readme

* Documentation edits

* Edited imports on Analyst Notebook

* making packages independent from img_str; standardized python package

* removing submodule from gitmodules

* fix dependencies

* scikit soen't seem to exist, replacing it with scikit-image since it's the likely target

* deleting result files

* simplify cmake

* no nodelet plugins in this package

* fixing CI hierarchy

* docker compose version obsolete; fixes on name changes

* adding model compatible with focal

* generilizing the code + removing specific paths

* env vars check fix

* making folder consistent; still showing results if no bagfile

---------

Co-authored-by: Rachel Lu <[email protected]>
Co-authored-by: Marina Moreira <[email protected]>
Co-authored-by: Marina Moreira <[email protected]>

.gitignore

@@ -58,9 +58,11 @@ analyst/workspace/.ipython
 analyst/workspace/.jupyter
 analyst/workspace/.local
 analyst/workspace/.ipynb_checkpoints
+analyst/workspace/results
 analyst/workspace/scripts/.ipynb_checkpoints
 analyst/workspace/.cache
+analyst/workspace/craft_mlt_25k.pth
 analyst/workspace/data
 
 # vscode
-isaac.code-workspace
+isaac.code-workspace

analyst/readme.md

@@ -5,11 +5,12 @@
 
 **The jupyter notebooks will be able to access data that is in the `$HOME/data` and `$HOME/data/bags`, therefore, make sure all the relevant bag files are there**
 
+**If you want to run the OCR, make sure there is a `$HOME/data/str` folder with all the data**
 
 For the Analyst notebook to be functional, it needs to start side-by-side with the database and the IUI (ISAAC user interface).
 To do so, the recommended method is to use the remote docker images, as:
 
-	$ISAAC_SRC/scripts/docker/run.sh --analyst --no-sim --remote
+	$ISAAC_SRC/scripts/docker/run.sh --analyst --mount --no-sim --remote
 
 The ISAAC UI is hosted in: http://localhost:8080
 The ArangoDB database is hosted in: http://localhost:8529

anomaly/image_str/.gitignore

@@ -0,0 +1,7 @@
+**/image_str/result/
+**/image_str/images/
+*.json
+craft_mlt_25k.pth
+**/image_str/models/
+**/image_str/test_images/
+**/image_str/tests/

anomaly/image_str/CMakeLists.txt

@@ -0,0 +1,33 @@
+# Copyright (c) 2021, United States Government, as represented by the
+# Administrator of the National Aeronautics and Space Administration.
+#
+# All rights reserved.
+#
+# The "ISAAC - Integrated System for Autonomous and Adaptive Caretaking
+# platform" software is licensed under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with the
+# License. You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
+project(image_str)
+
+## Compile as C++14, supported in ROS Kinetic and newer
+add_compile_options(-std=c++14)
+
+
+## Find catkin macros and libraries
+find_package(catkin REQUIRED COMPONENTS
+)
+
+catkin_python_setup()
+
+catkin_package(
+)

anomaly/image_str/package.xml

@@ -0,0 +1,18 @@
+<?xml version="1.0"?>
+<package format="2">
+  <name>image_str</name>
+  <version>0.0.0</version>
+  <description>The OCR package</description>
+  <license>
+    Apache License, Version 2.0
+  </license>
+  <author email="[email protected]">
+    ISAAC Flight Software
+  </author>
+  <maintainer email="[email protected]">
+    ISAAC Flight Software
+  </maintainer>
+
+  <buildtool_depend>catkin</buildtool_depend>
+
+</package>

anomaly/image_str/readme.md

@@ -0,0 +1,52 @@
+\page str_anomaly Image STR
+
+Image STR Anomaly Detection Documentation
+====================
+
+Overview
+---------
+
+The Image anomaly detector uses Scene Text Recognition (STR) to parse through sci-cam images of the ISS. It creates a database of all the 
+labels found on the ISS and displays the results of a search label using the panorama interface https://ivr.ndc.nasa.gov/isaac_panos/.
+
+
+Prerequisites 
+---------
+
+This package relies on the CRAFT-Pytorch and PARSeq libraries. To ensure the Image STR package works smoothly, clone the repos associated with the 
+libraries and build them as Python packages.
+
+CRAFT-Pytorch
+
+    pip install git+https://github.com/marinagmoreira/CRAFT-pytorch.git
+
+PARSeq
+
+	pip install git+https://github.com/marinagmoreira/parseq.git@focal#egg=parseq
+
+The Image STR package also contains a setup file with all the other necessary Python packages. 
+
+	pip install .
+
+Running the Code
+---------
+
+The python code containing the label detection and search is in parse_img.py.
+
+Parameters
+```
+--bag_path BAG_PATH   Path to bag folder where the images came from.
+--image_file IMAGE_FILE
+                    Path to image to parse.
+--image_folder IMAGE_FOLDER
+                    Path to image folder to parse images.
+--result_folder RESULT_FOLDER
+                    Path to result folder to save results.
+--increment INCREMENT
+                    If True, will save the results of each individual
+                    image.
+--df_file DF_FILE   If provided, will create an ocr using data from csv
+                    file.
+```
+
+To see the demo, run the Analyst Notebook. Directions are specified in the readme.md in the Analyst Folder.

anomaly/image_str/scripts/image_str/graph.py

@@ -0,0 +1,74 @@
+import glob
+import os
+import re
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from mpl_toolkits.mplot3d import Axes3D
+
+
+def get_all_files(folder):
+    files = glob.glob(folder + "*_locations.csv")
+    return files
+
+
+def graph(files):
+    fig = plt.figure()
+    ax = fig.add_subplot(projection="3d")
+
+    x = []
+    y = []
+    z = []
+    df = pd.DataFrame(
+        columns=["label", "PCL Intersection", "Mesh Intersection", "image", "location"]
+    )
+    for file in files:
+        # with open(file, "r") as f:
+        #     for line in f:
+        #         if "PCL Intersection" in line:
+        #             nums = re.findall(
+        #                 r"[-+]?\d*\.\d+|\d+", line
+        #             )  # [x, y, z, roll, pitch, yaw]
+        #             x.append(float(nums[0]))
+        #             y.append(float(nums[1]))
+        #             z.append(float(nums[2]))
+        # data = np.genfromtxt(file, delimiter=';', dtype=str, missing_values=None)
+        # print("Line 900:", data[899])
+        df2 = pd.read_csv(file, delimiter=";", skiprows=[1])
+        df = pd.concat([df, df2])
+        # with open(file, 'r') as f:
+        #     lines = f.readlines()'
+
+        # # Print line 900
+        # print("Line 900:", lines[899])
+    print(df)
+    for row in df["PCL Intersection"].tolist():
+        nums = re.findall(r"[-+]?\d*\.\d+|\d+", row)
+        x.append(float(nums[0]))
+        y.append(float(nums[1]))
+        z.append(float(nums[2]))
+    #     nums = data[:, 1]
+    #     nums = [re.findall(r"[-+]?\d*\.\d+|\d+", i) for i in nums]
+    #     for i in nums:
+    #         x.append(float(i[0]))
+    #         y.append(float(i[1]))
+    #         z.append(float(i[2]))
+
+    ax.scatter(x, y, z)
+
+    ax.set_xlabel("X")
+    ax.set_ylabel("Y")
+    ax.set_zlabel("Z")
+    plt.title("Label Locations on ISS")
+
+    plt.show()
+    # return plt
+
+
+if __name__ == "__main__":
+    # files = get_all_files(os.getcwd())
+    # plt = graph(files)
+    # plt.show()
+    files = [os.path.join(os.getcwd(), "all_locations.csv")]
+    graph(files)

anomaly/image_str/scripts/image_str/net_utils.py

@@ -0,0 +1,90 @@
+import time
+from collections import OrderedDict
+
+import craft.craft_utils as craft_utils
+import craft.file_utils as file_utils
+import craft.imgproc as imgproc
+import cv2
+import image_str.utils as utils
+import numpy as np
+import torch
+import torch.backends.cudnn as cudnn
+import torch.nn as nn
+from craft.craft import CRAFT
+from torch.autograd import Variable
+
+
+# Credit to CRAFT-pytorch
+# https://github.com/clovaai/CRAFT-pytorch
+def copyStateDict(state_dict):
+    if list(state_dict.keys())[0].startswith("module"):
+        start_idx = 1
+    else:
+        start_idx = 0
+    new_state_dict = OrderedDict()
+    for k, v in state_dict.items():
+        name = ".".join(k.split(".")[start_idx:])
+        new_state_dict[name] = v
+    return new_state_dict
+
+
+def test_net(
+    net, image, text_threshold, link_threshold, low_text, cuda, poly, refine_net=None
+):
+    """ """
+
+    t0 = time.time()
+
+    canvas_size = 1280
+    mag_ratio = 1.5
+
+    # resize
+    img_resized, target_ratio, size_heatmap = imgproc.resize_aspect_ratio(
+        image, canvas_size, interpolation=cv2.INTER_LINEAR, mag_ratio=mag_ratio
+    )
+    ratio_h = ratio_w = 1 / target_ratio
+
+    # preprocessing
+    x = imgproc.normalizeMeanVariance(img_resized)
+    x = torch.from_numpy(x).permute(2, 0, 1)  # [h, w, c] to [c, h, w]
+    x = Variable(x.unsqueeze(0))  # [c, h, w] to [b, c, h, w]
+    if cuda:
+        x = x.cuda()
+
+    # forward pass
+    with torch.no_grad():
+        y, feature = net(x)
+
+    # make score and link map
+    score_text = y[0, :, :, 0].cpu().data.numpy()
+    score_link = y[0, :, :, 1].cpu().data.numpy()
+
+    # refine link
+    if refine_net is not None:
+        with torch.no_grad():
+            y_refiner = refine_net(y, feature)
+        score_link = y_refiner[0, :, :, 0].cpu().data.numpy()
+
+    t0 = time.time() - t0
+    t1 = time.time()
+
+    # Post-processing
+    boxes, polys = craft_utils.getDetBoxes(
+        score_text, score_link, text_threshold, link_threshold, low_text, poly
+    )
+
+    # coordinate adjustment
+    boxes = craft_utils.adjustResultCoordinates(boxes, ratio_w, ratio_h)
+    polys = craft_utils.adjustResultCoordinates(polys, ratio_w, ratio_h)
+    for k in range(len(polys)):
+        if polys[k] is None:
+            polys[k] = boxes[k]
+
+    t1 = time.time() - t1
+
+    # render results (optional)
+    render_img = score_text.copy()
+    render_img = np.hstack((render_img, score_link))
+    ret_score_text = imgproc.cvt2HeatmapImg(render_img)
+
+    return boxes, polys, ret_score_text