Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed cv2/numpy bug + Better Setup & Usage Instructions + Fixed pointnav_policy_path for Reality #50

Closed
wants to merge 9 commits into from
Closed

Fixed cv2/numpy bug + Better Setup & Usage Instructions + Fixed pointnav_policy_path for Reality #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
65f32e6
Update detections.py
ayushzenith Sep 25, 2024
9b40291
Update README.md with better setup instructions
ayushzenith Sep 25, 2024
e3893e5
Update README.md
ayushzenith Sep 25, 2024
b480ae9
Update reality.yaml
ayushzenith Sep 25, 2024
71bb7cf
Update reality.yaml
ayushzenith Sep 25, 2024
ee22e44
Update README.md
ayushzenith Sep 25, 2024
98d01e7
Update server_wrapper.py
ayushzenith Sep 25, 2024
8e238c9
Update server_wrapper.py
ayushzenith Sep 25, 2024
1f0487f
Update README.md
ayushzenith Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 60 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,32 +35,52 @@ Understanding how humans leverage semantic knowledge to navigate unfamiliar envi

## :hammer_and_wrench: Installation

### Getting Started
### Getting Started: Environment

Create the conda environment:
```bash
conda_env_name=vlfm
conda create -n $conda_env_name python=3.9 -y &&
conda activate $conda_env_name
```

Check compiled cuda version/available cuda version and install the closest available corresponding torch. *This will be important for successful GroundingDINO installation*

If you are using habitat and are doing simulation experiments, install this repo into your env with the following:
```bash
pip install -e .[habitat]
```

If you are using the Spot robot, install this repo into your env with the following:
```bash
pip install -e .[reality]
```
Install all the dependencies:

Clone the below dependencies into the root directory of this repo:
```bash
git clone [email protected]:IDEA-Research/GroundingDINO.git
git clone [email protected]:WongKinYiu/yolov7.git # if using YOLOv7
git clone https://github.com/IDEA-Research/GroundingDINO.git
git clone https://github.com/WongKinYiu/yolov7.git # if using YOLOv7
git clone https://github.com/facebookresearch/habitat-lab
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already under pyproject.toml, installable under pip install -e .[habitat]

Copy link
Author

@ayushzenith ayushzenith Oct 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately for some reason in my Windows and Linux environment even though pip install -e .[habitat] ran without any errors it didn't git clone the repo's into the directory and files from the yolov7 and groundingdino were required to run the models from what I remember

git clone https://github.com/naokiyokoyama/bd_spot_wrapper.git
```
Follow the original install directions for GroundingDINO, which can be found here: https://github.com/IDEA-Research/GroundingDINO.

Nothing needs to be done for YOLOv7, but it needs to be cloned into the repo.

### Installing GroundingDINO (Only if using conda-installed CUDA)
Follow the original install directions for Habitat-lab, which can be found here: https://github.com/facebookresearch/habitat-lab.

Follow the original install directions for bd spot wrapper, which can be found here: https://github.com/naokiyokoyama/bd_spot_wrapper.git.

The following pip install's will also be required:

```bash
pip install hydra-core --upgrade
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to move these to pyproject.toml or the Dockerfile as much as possible.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True! If I get the chance will move it there nonetheless thought it was useful information incase the requirements in pyproject.toml wasn't enough to get your environment ready

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both habitat-lab and bd_spot_wrapper have been in the toml file already

pip install numba # Try not to downgrade numpy during this instalation if possible. Go to https://numba.readthedocs.io/en/stable/user/installing.html for more installation instructions.
pip install gym
pip install timm==0.6.12 # Ignore any issues about incompatibility. Need version >=0.6.12 for depth model.
```

#### Troubleshooting: Installing GroundingDINO (Only if using conda-installed CUDA)
Only attempt if the installation instructions in the GroundingDINO repo do not work.

To install GroundingDINO, you will need `CUDA_HOME` set as an environment variable. If you would like to install a certain version of CUDA that is compatible with the one used to compile your version of pytorch, and you are using conda, you can run the following commands to install CUDA and set `CUDA_HOME`:
Expand All @@ -81,7 +101,23 @@ ln -s ${CONDA_PREFIX}/lib/python3.9/site-packages/nvidia/cusolver/include/* ${C
export CUDA_HOME=${CONDA_PREFIX}
```

## :dart: Downloading the HM3D dataset

### :weight_lifting: Downloading weights for various models
The weights for MobileSAM, GroundingDINO, and PointNav must be saved to the `data/` directory. The weights can be downloaded from the following links:
- `mobile_sam.pt`: https://github.com/ChaoningZhang/MobileSAM
- `groundingdino_swint_ogc.pth`: https://github.com/IDEA-Research/GroundingDINO
- `yolov7-e6e.pt`: https://github.com/WongKinYiu/yolov7
- `pointnav_weights.pth`: included inside the [data](data) subdirectory

```bash
cd data/
wget -q https://github.com/ChaoningZhang/MobileSAM/blob/master/weights/mobile_sam.pt
wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
wget -q https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-e6e.pt
```


### :dart: Dataset: Downloading the HM3D dataset

### Matterport
First, set the following variables during installation (don't need to put in .bashrc):
Expand All @@ -97,7 +133,7 @@ DATA_DIR=</path/to/vlfm/data>
HM3D_OBJECTNAV=https://dl.fbaipublicfiles.com/habitat/data/datasets/objectnav/hm3d/v1/objectnav_hm3d_v1.zip
```

### Clone and install habitat-lab, then download datasets
### Make sure that you have cloned and installed *habitat-lab*, then download datasets
*Ensure that the correct conda environment is activated!!*
```bash
# Download HM3D 3D scans (scenes_dataset)
Expand All @@ -118,13 +154,6 @@ mv objectnav_hm3d_v1 $DATA_DIR/datasets/objectnav/hm3d/v1 &&
rm objectnav_hm3d_v1.zip
```

## :weight_lifting: Downloading weights for various models
The weights for MobileSAM, GroundingDINO, and PointNav must be saved to the `data/` directory. The weights can be downloaded from the following links:
- `mobile_sam.pt`: https://github.com/ChaoningZhang/MobileSAM
- `groundingdino_swint_ogc.pth`: https://github.com/IDEA-Research/GroundingDINO
- `yolov7-e6e.pt`: https://github.com/WongKinYiu/yolov7
- `pointnav_weights.pth`: included inside the [data](data) subdirectory

## :arrow_forward: Evaluation within Habitat
To run evaluation, various models must be loaded in the background first. This only needs to be done once by running the following command:
```bash
Expand All @@ -137,11 +166,28 @@ Run the following to evaluate on the HM3D dataset:
```bash
python -m vlfm.run
```
To run the evaluation and save the video of the simulator:
```
python -m vlfm.run habitat_baselines.eval.video_option='["disk"]'
```
To evaluate on MP3D, run the following:
```bash
python -m vlfm.run habitat.dataset.data_path=data/datasets/objectnav/mp3d/val/val.json.gz
```

## :arrow_forward: Run on Spot
To run program on spot:

To edit goal object edit env.goal in `config\experiments\reality.yaml`

```bash
export SPOT_ADMIN_PW=<YOUR_SPOT_ADMIN_PW>
export SPOT_IP=<SPOT_IP>

python -m vlfm.reality.run_bdsw_objnav_env
```


## :newspaper: License

VLFM is released under the [MIT License](LICENSE). This code was produced as part of Naoki Yokoyama's internship at the Boston Dynamics AI Institute in Summer 2023 and is provided "as is" without active maintenance. For questions, please contact [Naoki Yokoyama](http://naoki.io) or [Jiuguang Wang](https://www.robo.guru).
Expand Down
3 changes: 2 additions & 1 deletion config/experiments/reality.yaml
View file
Open in desktop
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The incorrect pointnav weights were provided in the reality config file assuming the reality config file correlated with the config to be used on the spot

Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ defaults:

policy:
name: "RealityITMPolicyV2"
pointnav_policy_path: "data/pointnav_weights.pth"
#pointnav_policy_path: "data/pointnav_weights.pth"
pointnav_policy_path: "data/spot_pointnav_weights.pth"
depth_image_shape: [212, 240] # height, width
pointnav_stop_radius: 0.9
use_max_confidence: False
Expand Down
2 changes: 1 addition & 1 deletion vlfm/vlm/detections.py
View file
Open in desktop
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cv2 error which was fixed by specifying the input int type (uint8)

Error reproducible in habitat and reality when using original environment setup instructions as of October 1, 2024 since no specific opencv version is specified and any numpy version after "1.22.4" is accepted

Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ def draw_bounding_box(

if color is None:
# Randomly choose a color from the rainbow colormap (so boxes aren't black)
single_pixel = np.array([[np.random.randint(0, 256)]])
single_pixel = np.array([[np.random.randint(0, 256, dtype=np.uint8)]])
single_pixel = cv2.applyColorMap(single_pixel, cv2.COLORMAP_RAINBOW)

# reshape to a single dimensional array
Expand Down
3 changes: 2 additions & 1 deletion vlfm/vlm/server_wrapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ def process_request() -> Dict[str, Any]:
payload = request.json
return jsonify(model.process_payload(payload))

app.run(host="localhost", port=port)
app.run(host="0.0.0.0", port=port) # app.run(host="localhost", port=port)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is for external access?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up the flask server to the local network. Still allows access to localhost so doesn't need any changes to any other code if models are still hosted on the device connected to spot.

But also allows for the option to host models elsewhere on the network allowing for bigger and better models while having smaller edge compute devices on the spot instead of needing all the compute to be on or near spot. Opens up the option to easily add more models of all sizes if interested




def bool_arr_to_str(arr: np.ndarray) -> str:
Expand Down
Loading