Support Zillow Indoor dataset

README.md

@@ -3,40 +3,44 @@
 This is the implementation of our CVPR'19 "[
 HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation](https://arxiv.org/abs/1901.03861)" ([project page](https://sunset1995.github.io/HorizonNet/)).
 
-**News, April 3, 2021** - Check out our new project [HoHoNet](https://github.com/sunset1995/HoHoNet) on this task and more!\
-**News, Mar. 14, 2021** - (1) Use mesh instead of point cloud as layout viewer. (2) Update lsd detector dependency.\
-**News, June 15, 2019** - Critical bug fix for general layout (`dataset.py`, `inference.py` and `misc/post_proc.py`)\
-**News, Aug. 19, 2019** - Report results on [Structured3D dataset](https://structured3d-dataset.org/). (See [the report :clipboard: on ST3D](README_ST3D.md)).
-
 ![](assets/teaser.jpg)
 
+### Update
+- **2021.11.04: Report results on [Zillow Indoor dataset](https://github.com/zillow/zind). (See [the report :clipboard: on ZInD](README_ZInD.md)).**
+- 2021.04.03: Check out our new project [HoHoNet](https://github.com/sunset1995/HoHoNet) on this task and more!
+- 2021.03.14: (1) Use mesh instead of point cloud as layout viewer. (2) Update lsd detector dependency.
+- 2019.08.19: Report results on [Structured3D dataset](https://structured3d-dataset.org/). (See [the report :clipboard: on ST3D](README_ST3D.md)).
+- 2019.06.15: Bug fix for general layout (`dataset.py`, `inference.py` and `misc/post_proc.py`)
+
+### Feature
 This repo is a **pure python** implementation that you can:
 - **Inference on your images** to get cuboid or general shaped room layout
 - **3D layout viewer**
-- **Correct pose** for your panorama images
-- **Pano Stretch Augmentation** copy and paste to apply on your own task
-- **Quantitative evaluatation** (3D IoU, Corner Error, Pixel Error)
-    - cuboid shape
-    - general shape
+- **Correct rotation pose** to ensure manhattan alignment
+- **Pano stretch augmentation** copy and paste to apply on your own task
+- **Quantitative evaluatation** of 2D IoU, 3D IoU, Corner Error, Pixel Error of cuboid/general shape
 - **Your own dataset** preparation and training
 
-**Method Pipeline overview**:
+### Method overview
 ![](assets/pipeline.jpg)
 
-## Requirements
-- Python 3
-- pytorch>=1.0.0
-- numpy
-- scipy
-- sklearn
-- Pillow
-- tqdm
-- tensorboardX
-- opencv-python>=3.1 (for pre-processing)
-- pylsd-nova
-- open3d>=0.7 (for layout 3D viewer)
-- shapely
-- torchvision
+### Installation
+Pytorch installation is machine dependent, please install the correct version for your machine. The tested version is pytorch 1.8.1 with python 3.7.6.
+
+<details>
+  <summary> Dependencies (click to expand) </summary>
+
+   - numpy
+   - scipy
+   - sklearn
+   - Pillow
+   - tqdm
+   - tensorboardX
+   - opencv-python>=3.1 (for pre-processing)
+   - pylsd-nova
+   - open3d>=0.7 (for layout 3D viewer)
+   - shapely
+</details>
 
 
 ### Download
@@ -57,16 +61,22 @@ This repo is a **pure python** implementation that you can:
         - `finetune_general` is re-annotated by us from `train` and `valid`. It contains  65 general shaped rooms.
 - Structured3D Dataset
     - See [the tutorial](https://github.com/sunset1995/HorizonNet/blob/master/README_ST3D.md#dataset-preparation) to prepare training/validation/testing for HorizonNet.
+- Zillow Indoor Dataset
+    - See [the tutorial](https://github.com/sunset1995/HorizonNet/blob/master/README_ZInD.md#dataset-preparation) to prepare training/validation/testing for HorizonNet.
 
 
 #### Pretrained Models
 - [resnet50_rnn__panos2d3d.pth](https://drive.google.com/open?id=1aieMd61b-3BoOeTRv2pKu9yTk5zEinn0)
     - Trained on PanoContext/Stanford2d3d 817 pano images.
     - Trained for 300 epoch
 - [resnet50_rnn__st3d.pth](https://drive.google.com/open?id=16v1nhL9C2VZX-qQpikCsS6LiMJn3q6gO)
-    - Trained on Structured3D 18362 pano images with setting of original furniture and lighting.
+    - Trained on Structured3D 18362 pano images
+    - Data setup: original furniture and lighting.
+    - Trained for 50 epoch.
+- [resnet50_rnn__zind.pth](https://drive.google.com/open?id=1FrMdk7Z4_sTZOOW65Ek77WbjiDbV98uJ)
+    - Trained on Zillow Indoor 20077 pano images.
+    - Data setup: `layout_visible`, `is_primary`, `is_inside`, `is_ceiling_flat`.
     - Trained for 50 epoch.
-    - Select 50th epoch according to loss function on validation set.
 
 
 ## Inference on your images

README_ZInD.md

@@ -0,0 +1,94 @@
+# Results on Zillow Indoor dataset
+
+References:
+- [Zillow Indoor Dataset: Annotated Floor Plans With 360º Panoramas and 3D Room Layouts](https://openaccess.thecvf.com/content/CVPR2021/papers/Cruz_Zillow_Indoor_Dataset_Annotated_Floor_Plans_With_360deg_Panoramas_and_CVPR_2021_paper.pdf)
+- [ZInD github](https://github.com/zillow/zind)
+
+
+## Dataset preparation
+- Visit [ZInD](https://github.com/zillow/zind) to get the datas.
+    - Also download the official split `zind_partition.json`.
+- Run below to extract the rgb and layout to support HorizonNet training and evaluation.
+    ```bash
+    python misc/zind_prepare_dataset.py --partition {PATH_TO}/zind_partition.json --indir {PATH_TO_DATA_ROOT} --outdir data/zind_horizonnet/
+    ```
+  Add `-h` to see more option. The default setting extract only `layout_visible`, `is_primary`, `is_inside`, `is_ceiling_flat`.
+
+
+## Training
+```bash
+python train.py --train_root_dir data/zind_horizonnet/train/ --valid_root_dir data/zind_horizonnet/val/ --id resnet50_rnn__zind --epochs 50
+```
+See `python train.py -h` or [README.md#training](https://github.com/sunset1995/HorizonNet#training) for more detail.
+
+Download the trained model: [resnet50_rnn__zind.pth](https://drive.google.com/open?id=16v1nhL9C2VZX-qQpikCsS6LiMJn3q6gO).
+- Trained on Zillow Indoor 20077 images with default data extraction setup.
+- Trained for 50 epoch.
+
+
+## Restuls
+
+### Testing
+```bash
+python inference.py --pth ckpt/resnet50_rnn__zind.pth --img_glob "data/zind_horizonnet/test/img/*" --output_dir ./output/zind/resnet50_rnn_post/ --visualize
+python inference.py --pth ckpt/resnet50_rnn__zind.pth --img_glob "data/zind_horizonnet/test/img/*" --output_dir ./output/zind/resnet50_rnn_nopost/ --visualize --force_raw
+```
+- `--output_dir`: A directory you want to dump the extracted layout.
+- `--visualize`: Visualize raw output from HorizonNet.
+- `--force_raw`: Disable post-processing, export raw output as `1024*2` vertices polygon instead.
+
+### Quantitative evaluation
+```bash
+python eval_general.py --dt_glob "./output/zind/resnet50_rnn_post/*" --gt_glob "data/zind_horizonnet/test/label_cor/*"
+python eval_general.py --dt_glob "./output/zind/resnet50_rnn_nopost/*" --gt_glob "data/zind_horizonnet/test/label_cor/*"
+```
+
+:clipboard: Below is the quantitative result on Zillow Indoor testing set.
+
+#### 2D IoU
+|          | all    | 4     | 6     | 8     | 10+   | odd   |
+|----------|--------|-------|-------|-------|-------|-------|
+| w post   | 68.48  | 79.31 | 75.20 | 70.80 | 56.70 | 58.25 |
+| w/o post | 89.80  | 94.13 | 93.56 | 91.31 | 84.36 | 85.59 |
+
+#### 3D IoU
+|          | all    | 4     | 6     | 8     | 10+   | odd   |
+|----------|--------|-------|-------|-------|-------|-------|
+| w post   | 67.52  | 78.23 | 74.28 | 69.96 | 55.85 | 57.32 |
+| w/o post | 88.47  | 92.83 | 92.32 | 90.06 | 83.03 | 84.15 |
+
+#### Conclusion
+We find that current post-processing fails on Zillow Indoor dataset.
+A new algorithm to rectify model's raw output (i.e., polygon w/ `1024*2` vertices) to Manhattan layout is required.
+
+
+### Qualitative results
+
+We visualize model raw output on Zillow Indoor testset.
+We also compare with the results from a HorizonNet trained only on Structured3D dataset.
+
+#### Different layout definition
+Whether to reconstruct the boundary out of current camera position is application dependent.
+
+| Trained only on Structured3D | Trained on Zillow Indoor |
+|:--:|:--:|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_10_pano_6.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_10_pano_6.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_05_pano_24.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_05_pano_24.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_14_pano_18.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_14_pano_18.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_02_pano_9.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_02_pano_9.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_01_pano_25.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_01_pano_25.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_07_pano_14.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_07_pano_14.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_12_pano_2.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_12_pano_2.raw.png)|
+
+
+#### Domain gap?
+Probably solvable by domain adaptation in future work.
+
+| Trained only on Structured3D | Trained on Zillow Indoor |
+|:--:|:--:|
+|![](assets/result_zind/trained_st3d/0011_floor_01_partial_room_07_pano_12.raw.png)|![](assets/result_zind/trained_zind/0011_floor_01_partial_room_07_pano_12.raw.png)|
+|![](assets/result_zind/trained_st3d/0021_floor_01_partial_room_03_pano_7.raw.png)|![](assets/result_zind/trained_zind/0021_floor_01_partial_room_03_pano_7.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_08_pano_22.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_08_pano_22.raw.png)|
+|![](assets/result_zind/trained_st3d/0010_floor_01_partial_room_09_pano_23.raw.png)|![](assets/result_zind/trained_zind/0010_floor_01_partial_room_09_pano_23.raw.png)|
+|![](assets/result_zind/trained_st3d/0011_floor_01_partial_room_06_pano_18.raw.png)|![](assets/result_zind/trained_zind/0011_floor_01_partial_room_06_pano_18.raw.png)|
+