Diffusion inference failed

[terryaic]

PYTHONPATH=$(pwd) python cosmos1/models/diffusion/inference/video2world.py
--checkpoint_dir /models/nvidia
--diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Video2World
--input_image_or_video_path cosmos1/models/diffusion/assets/v1p0/video2world_input0.jpg
--num_input_frames 1
--video_save_name Cosmos-1.0-Diffusion-7B-Video2World
--offload_prompt_upsampler

[01-18 01:30:50|INFO|cosmos1/models/diffusion/inference/world_generation_pipeline.py:609:generate] Run with prompt upsampler: True
[01-18 01:30:50|INFO|cosmos1/models/diffusion/inference/world_generation_pipeline.py:621:generate] Run prompt upsampler on image or video, input prompt is not used
Traceback (most recent call last):
File "/workspace/cosmos1/models/diffusion/inference/video2world.py", line 178, in
demo(args)
File "/workspace/cosmos1/models/diffusion/inference/video2world.py", line 141, in demo
generated_output = pipeline.generate(
File "/workspace/cosmos1/models/diffusion/inference/world_generation_pipeline.py", line 622, in generate
prompt = self._run_prompt_upsampler_on_prompt_with_offload(image_or_video_path=image_or_video_path)
File "/workspace/cosmos1/models/diffusion/inference/world_generation_pipeline.py", line 189, in _run_prompt_upsampler_on_prompt_with_offload
self._load_prompt_upsampler_model()
File "/workspace/cosmos1/models/diffusion/inference/world_generation_pipeline.py", line 464, in _load_prompt_upsampler_model
self.prompt_upsampler = create_vlm_prompt_upsampler(
File "/workspace/cosmos1/models/diffusion/prompt_upsampler/video2world_prompt_upsampler_inference.py", line 56, in create_vlm_prompt_upsampler
return AutoRegressiveModel.build(
File "/workspace/cosmos1/models/autoregressive/model.py", line 255, in build
len(vit_checkpoint) > 0 and len(projector_checkpoint) > 0
AssertionError: vit_checkpoint and projector_checkpoint cannot be empty. We do not support random initialization for vision_encoder and mm_projector.

[ethanhe42]

@sophiahhuang can you reassign? this is pytorch path