SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models

Get Started

1. Clone Github repo.

git clone https://github.com/zhangyuejoslin/Spartun3D.git
cd Spartun3D

2. Following LEO to create a conda environment and install third-party libraries for point cloud backbones.

Generating Data for Spartun3D

1. Generate Metadata
   Run scene_process.py to collect raw spatial information. We provide the metadata at: Google Drive Link.

2. Generate Data with GPT-4o
   Run the code in gpt_code to generate data using GPT-4o. Please provide your OpenAI API key.

3. Post-Process the Generated Data
   Use data_process/3Rscan/gpt_code/post_process.py to refine the generated data.

4. Generated Captions and QA Pairs
   We provide the generated captions and question-answer pairs in data_process/spartun3D_data.
   Please download the corresponding scene data from LEO.

Train and Test

python launch

python launch.py --mode python --config configs/default.yaml

accelerate launch

python launch.py --mode accelerate --config configs/default.yaml

SQA3D Eval

sqa3d checkpoint