README.md

SA-DNet

SA-DNet is an end-to-end Semantic-Aware on-Demand registration-fusion network. SA-DNet can specify registration and fusion regions based on semantic information, achieving on-demand conceptualization. Furthermore, the sam, reg, and fuse modules are separated, allowing integration of new functionalities. The current architecture implemented in this project is as follows:

Note:

In this project, the HOL operators in reg have not been implemented. When SAM perceives regions that are too large, it leads to excessive computation in HOL, causing the system to freeze.

TODO

• Fixed bug.
• Update the new sam, reg, and fuse modules.
• To optimize and implement the HOL operator

Install

Download this project:

git clone [email protected]:Meng-Sang/SA-DNet.git
cd SA-DNet

Runtime environment

This project requires a runtime environment with Python=3.8 or higher. Additionally, it depends on the libraries specified in the requirements.txt file and 2 GitHub repositories (Grounding DINO and Segment Anything). The environment configuration is as follows:

Step 1: install python and requirements.txt

conda create -n sa-dnet  python=3.8
pip install -r requirement.txt

Step 2: Configure Grounding DINO

Install Grounding DINO

For Grounding DINO installation, please refer to Grounding DINO.

Download weight

# if model/sam/grounding_dino/weight not exist.
# please run: mkdir -p model/sam/grounding_dino/weight
# then:
cd model/sam/grounding_dino/weight
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth

Note:

If the network is unavailable, the following error may occur:

OSError: Can't load tokenizer

Step 3: Configure Segment Anything

Install Segment Anything

git clone [email protected]:facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

or

pip install git+git://github.com/facebookresearch/segment-anything.git

Download weight

# if model/sam/grounding_dino/weight not exist.
# please run: mkdir -p model/sam/segment_anything/weight
# then:
cd model/sam/segment_anything/weight
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Demo

Run with Python

import cv2
from utils.predict import sa_dnet

ir_image = cv2.imread("assets/ir/006393.jpg")
vi_image = cv2.imread("assets/vi/006393.jpg")
_, fuse_image = sa_dnet(ir_image, vi_image, "car", is_mask=True)
cv2.imwrite("assets/demo.jpg", fuse_image)

Show result

IR Image	VI Image
Matching Image (Car)
Unmask Fuse Image (Car)	Mask Fuse Image (Car)

Acknowledgments:

This project depends on the following repositories:

• Grounding DINO
• Segment Anything