How to offload when using multi GPU (NeMo)

[clairelee5740]

As in https://github.com/NVIDIA/Cosmos/tree/main/cosmos1/models/diffusion/nemo/inference#run-the-inference-script-with-base-model, I run
NVTE_FUSED_ATTN=0 \ torchrun --nproc_per_node=$NUM_DEVICES cosmos1/models/diffusion/nemo/inference/general.py \ --model Cosmos-1.0-Diffusion-7B-Text2World \ --cp_size $NUM_DEVICES \ --num_devices $NUM_DEVICES \ --video_save_path "Cosmos-1.0-Diffusion-7B-Text2World.mp4" \ --guidance 7 \ --seed 1 \ --prompt "$PROMPT" \ --enable_prompt_upsampler

I got cuda out of memory error, so I tried
NVTE_FUSED_ATTN=0 \ torchrun --nproc_per_node=$NUM_DEVICES cosmos1/models/diffusion/nemo/inference/general.py \ --model Cosmos-1.0-Diffusion-7B-Text2World \ --cp_size $NUM_DEVICES \ --num_devices $NUM_DEVICES \ --video_save_path "Cosmos-1.0-Diffusion-7B-Text2World.mp4" \ --guidance 7 \ --seed 1 \ --prompt "$PROMPT" \ --enable_prompt_upsampler \ --offload_tokenizer \ --offload_diffusion_transformer \ --offload_text_encoder_model \ --offload_prompt_upsampler \ --offload_guardrail_models

but it says
general.py: error: unrecognized arguments: --offload_tokenizer --offload_diffusion_transformer --offload_text_encoder_model --offload_prompt_upsampler --offload_guardrail_models

Did anyone encounter similar question when using NeMo framework? Is there any solution?
btw I am using 4 RTX 4090 24G and there is still not enough GPU memory

[ethanhe42]

it is currently not supported yet. you can try modify the code to delete models after they're used. e.g. del t5_model