Question about datasat?

[Qmebius]

I would like to ask if the dataset provides the 2D keypoints of SMPL. I have downloade but not found

[jiangwei221]

We used mmpose to detect the 2D keypoints:

ml-neuman/preprocess/gen_run.py

Lines 95 to 104 in 0149d25

	# Run 2D keypoints detector(mmpose)
	commands.append(f'echo ========================================')
	commands.append(f'echo 6/{steps}: 2D keypoints')
	commands.append(f'echo ========================================')
	commands.append(f'cd {os.path.join(code_dir, "mmpose")}')
	if not os.path.isdir(os.path.join(video_dir, f'{video_name}/output/keypoints')):
	commands.append('conda activate open-mmlab')
	commands.append(f'python demo/bottom_up_img_demo.py configs/body/2d_kpt_sview_rgb_img/associative_embedding/coco/higherhrnet_w48_coco_512x512_udp.py https://download.openmmlab.com/mmpose/bottom_up/higher_hrnet48_coco_512x512_udp-7cad61ef_20210222.pth --img-path {os.path.join(video_dir, f"{video_name}/output/images")} --out-img-root {os.path.join(video_dir, f"{video_name}/output/keypoints")} --kpt-thr=0.3 --pose-nms-thr=0.9')
	commands.append('conda deactivate')
	commands.append(f'cd {code_dir}')

[Qmebius]

Thanks，I would also like to ask, the bin files provided in the dataset contain depth information, which is very sparse in the human body area. Is there any way to obtain the depth information of the human body?

[jiangwei221]

Yes, we run BoostingMonocularDepth to estimate the human depth and fuse with colmap MVS depth.

1. BoostingMonocularDepth:
   

   ml-neuman/preprocess/gen_run.py

   Lines 106 to 121 in 0149d25

   	# Monocular depth estimation
   	commands.append(f'echo ========================================')
   	commands.append(f'echo 7/{steps}: Monocular depth')
   	commands.append(f'echo ========================================')
   	commands.append(f'cd {os.path.join(code_dir, "BoostingMonocularDepth")}')
   	if not os.path.exists(os.path.join(code_dir, 'BoostingMonocularDepth/pix2pix/checkpoints/mergemodel')):
   	os.makedirs(os.path.join(code_dir, 'BoostingMonocularDepth/pix2pix/checkpoints/mergemodel'))
   	if not os.path.isfile(os.path.join(code_dir, 'BoostingMonocularDepth/pix2pix/checkpoints/mergemodel/latest_net_G.pth')):
   	commands.append(f'wget https://sfu.ca/~yagiz/CVPR21/latest_net_G.pth -O {os.path.join(code_dir, "BoostingMonocularDepth/pix2pix/checkpoints/mergemodel/latest_net_G.pth")}')
   	if not os.path.isfile(os.path.join(code_dir, 'BoostingMonocularDepth/res101.pth')):
   	commands.append(f'wget https://cloudstor.aarnet.edu.au/plus/s/lTIJF4vrvHCAI31/download -O res101.pth')
   	if not os.path.isdir(os.path.join(video_dir, f'{video_name}/output/mono_depth')):
   	commands.append('conda activate ROMP')
   	commands.append(f'python run.py --Final --data_dir {os.path.join(video_dir, f"{video_name}/output/images")} --output_dir {os.path.join(video_dir, f"{video_name}/output/mono_depth")} --depthNet 2')
   	commands.append('conda deactivate')
   	commands.append(f'cd {code_dir}')

2. Depth fusion:
   

   ml-neuman/data_io/neuman_helper.py

   Lines 77 to 86 in 0149d25

   	@property
   	def fused_depth_map(self):
   	if self._fused_depth_map is None:
   	valid_mask = (self.depth_map > 0) & (self.mask == 0)
   	x = self.mono_depth_map[valid_mask]
   	y = self.depth_map[valid_mask]
   	res = scipy.stats.linregress(x, y)
   	self._fused_depth_map = self.depth_map.copy()
   	self._fused_depth_map[~valid_mask] = self.mono_depth_map[~valid_mask] * res.slope + res.intercept
   	return self._fused_depth_map

[Qmebius]

And I also want to ask how you obtain the ground truth of the camera pose, and is the 'trans' processed by alignment.py the gt value of the human body's position?

[jiangwei221]

And I also want to ask how you obtain the ground truth of the camera pose, and is the 'trans' processed by alignment.py the gt value of the human body's position?

We don't have a "ground truth camera pose" which usually requires manual annotations or extra hardwares. We use COLMAP to reconstruct the "reference camera poses".

[Qmebius]

So can this reference pose be used as the ground truth for the camera tracking task？

…

---Original--- From: "Wei ***@***.***> Date: Wed, Jul 10, 2024 01:00 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [apple/ml-neuman] Question about datasat? (Issue #99) And I also want to ask how you obtain the ground truth of the camera pose, and is the 'trans' processed by alignment.py the gt value of the human body's position? We don't have a "ground truth camera pose" which usually requires manual annotations or extra hardwares. We use COLMAP to reconstruct the "reference camera poses". — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[MikeAiJF]

另外想问一下你是怎么得到相机姿态的ground truth的，还有alignment.py处理后的trans是不是人体位置的gt值呢？

我们没有“地面真实相机姿势”，这通常需要手动注释或额外的硬件。我们使用 COLMAP 来重建“参考相机姿势”。

你好，请问我应该怎么才能得到4d-humans文件下的东西