Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Again - FaceDetailerPipe: torch.OutOfMemoryError - Allocation on device-2 - Desktop-V1 #33

Open
Zandebar opened this issue Feb 12, 2025 · 1 comment
Open

Again - FaceDetailerPipe: torch.OutOfMemoryError - Allocation on device-2 - Desktop-V1 #33

Zandebar opened this issue Feb 12, 2025 · 1 comment

Comments

@Zandebar
Copy link

I switched the SamLoader as you suggested and set to CPU, but I also switched from ComfyUI Embedded to ComfyUI.exe Desktop Version 1, I don't know if this switch will have an impact or not. Also am not sure if the checkpoint that I was using could impact these error, I ran 200 image generations with JuggernautXL_juggXI which so far hasn't produced an error. When using PhotopediaXL_45 checkpoint it produces the error report below, not sure if it's being poorly built and that's the reason for producing these error's or it's a wider issue. It's also the same checkpoint used that produced the previous ComfyUI Error Report (Comfy Embedded), not sure if it's a checkpoint issue but when I switch out the checkpoint to JuggernautXL_juggXI the process queue is executed fully.

ComfyUI Error Report

Error Details

  • Node ID: 476
  • Node Type: FaceDetailerPipe
  • Exception Type: torch.OutOfMemoryError
  • Exception Message: Allocation on device

Stack Trace

  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 327, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 202, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 174, in _map_node_over_list
    process_inputs(input_dict, i)

  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 163, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\impact_pack.py", line 1438, in doit
    enhanced_img, cropped_enhanced, cropped_enhanced_alpha, mask, cnet_pil_list = FaceDetailer.enhance_face(
                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\impact_pack.py", line 559, in enhance_face
    sam_mask = core.make_sam_mask(sam_model_opt, segs, image, sam_detection_hint, sam_dilation,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\core.py", line 732, in make_sam_mask
    detected_masks = sam_obj.predict(image, points, plabs, dilated_bbox, threshold)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\core.py", line 603, in predict
    predictor.set_image(image, "RGB")

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\predictor.py", line 60, in set_image
    self.set_torch_image(input_image_torch, image.shape[:2])

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\predictor.py", line 89, in set_torch_image
    self.features = self.model.image_encoder(input_image)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 112, in forward
    x = blk(x)
        ^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 174, in forward
    x = self.attn(x)
        ^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 234, in forward
    attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 358, in add_decomposed_rel_pos
    attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~

System Information

  • ComfyUI Version: 0.3.14
  • Arguments: C:\Users\MSI-Graphics\AppData\Local\Programs@comfyorgcomfyui-electron\resources\ComfyUI\main.py --user-directory E:\ComfyUI-Desktop-V1\user --input-directory E:\ComfyUI-Desktop-V1\input --output-directory E:\ComfyUI-Desktop-V1\output --front-end-root C:\Users\MSI-Graphics\AppData\Local\Programs@comfyorgcomfyui-electron\resources\ComfyUI\web_custom_versions\desktop_app --base-directory E:\ComfyUI-Desktop-V1 --extra-model-paths-config C:\Users\MSI-Graphics\AppData\Roaming\ComfyUI\extra_models_config.yaml --listen 127.0.0.1 --port 8000 --lowvram
  • OS: nt
  • Python Version: 3.12.8 (main, Jan 14 2025, 22:49:36) [MSC v.1942 64 bit (AMD64)]
  • Embedded Python: false
  • PyTorch Version: 2.6.0+cu126

Devices

  • Name: cuda:0 NVIDIA GeForce RTX 4060 : cudaMallocAsync
    • Type: cuda
    • VRAM Total: 8585216000
    • VRAM Free: 6832831284
    • Torch VRAM Total: 1006632960
    • Torch VRAM Free: 450149172

Logs

2025-02-10T16:30:05.924420 - loaded completely 5307.092982894897 4897.0483474731445 True
2025-02-10T16:30:28.871222 - 
100%|██████████| 35/35 [00:22<00:00,  1.79it/s]2025-02-10T16:30:28.871222 - 
100%|██████████| 35/35 [00:22<00:00,  1.54it/s]2025-02-10T16:30:28.871222 - 
2025-02-10T16:30:29.139375 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:30:29.139375 - 
2025-02-10T16:30:29.155373 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:30:29.155373 - 
2025-02-10T16:30:29.228454 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:30:29.228454 - 
2025-02-10T16:30:29.302453 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:30:29.302453 - 
2025-02-10T16:30:29.376454 - CLIP: []2025-02-10T16:30:29.376454 - 
2025-02-10T16:30:29.376454 - Requested to load SDXLClipModel
2025-02-10T16:30:30.296492 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:30:30.566492 - Detailer: segment upscale for ((23.00374, 30.963318)) | crop region (69, 92) x 8.348506975529396 -> (576, 768)2025-02-10T16:30:30.566492 - 
2025-02-10T16:30:30.890522 - Requested to load SDXL
2025-02-10T16:30:32.531887 - loaded completely 5339.599501617431 4897.0483474731445 True
2025-02-10T16:30:58.404364 - 
100%|██████████| 35/35 [00:25<00:00,  1.58it/s]2025-02-10T16:30:58.404865 - 
100%|██████████| 35/35 [00:25<00:00,  1.36it/s]2025-02-10T16:30:58.404865 - 
2025-02-10T16:30:58.473872 - loaded completely 469.42038345336914 159.55708122253418 True
2025-02-10T16:30:58.716207 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:30:58.716207 - 
2025-02-10T16:30:58.759207 - 
2025-02-10T16:30:58.772206 - 0: 640x384 1 face, 6.5ms
2025-02-10T16:30:58.772206 - Speed: 1.0ms preprocess, 6.5ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:31:03.267752 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:31:03.267752 - 
2025-02-10T16:31:03.342360 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:31:03.342360 - 
2025-02-10T16:31:03.415906 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:31:03.415906 - 
2025-02-10T16:31:03.487859 - CLIP: []2025-02-10T16:31:03.487859 - 
2025-02-10T16:31:03.488359 - Requested to load SDXLClipModel
2025-02-10T16:31:04.405722 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:31:04.673722 - Detailer: segment upscale for ((91.63095, 132.67871)) | crop region (274, 398) x 1.9299757564342959 -> (528, 768)2025-02-10T16:31:04.673722 - 
2025-02-10T16:31:05.128342 - Requested to load SDXL
2025-02-10T16:31:06.782542 - loaded completely 5305.985982345581 4897.0483474731445 True
2025-02-10T16:31:32.132910 - 
100%|██████████| 35/35 [00:25<00:00,  1.61it/s]2025-02-10T16:31:32.132910 - 
100%|██████████| 35/35 [00:25<00:00,  1.39it/s]2025-02-10T16:31:32.132910 - 
2025-02-10T16:31:32.403914 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:31:32.403914 - 
2025-02-10T16:31:32.921111 - [Impact Pack] WARN: FaceDetailer is not a node designed for video detailing. If you intend to perform video detailing, please use Detailer For AnimateDiff.2025-02-10T16:31:32.921111 - 
2025-02-10T16:31:32.929612 - 
2025-02-10T16:31:32.944111 - 0: 640x384 1 face, 8.0ms
2025-02-10T16:31:32.944111 - Speed: 1.0ms preprocess, 8.0ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:31:37.535518 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:31:37.535518 - 
2025-02-10T16:31:37.610017 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:31:37.610017 - 
2025-02-10T16:31:37.858017 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:31:37.858017 - 
2025-02-10T16:31:37.931517 - CLIP: []2025-02-10T16:31:37.931517 - 
2025-02-10T16:31:37.931517 - Requested to load SDXLClipModel
2025-02-10T16:31:38.822251 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:31:39.077253 - Detailer: segment upscale for ((84.87775, 123.82224)) | crop region (254, 371) x 2.070495524331133 -> (525, 768)2025-02-10T16:31:39.077253 - 
2025-02-10T16:31:39.540503 - Requested to load SDXL
2025-02-10T16:31:41.340316 - loaded completely 5426.889122161865 4897.0483474731445 True
2025-02-10T16:32:06.680551 - 
100%|██████████| 35/35 [00:25<00:00,  1.62it/s]2025-02-10T16:32:06.681051 - 
100%|██████████| 35/35 [00:25<00:00,  1.39it/s]2025-02-10T16:32:06.681051 - 
2025-02-10T16:32:06.746488 - loaded completely 443.52636337280273 159.55708122253418 True
2025-02-10T16:32:06.968989 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:32:06.968989 - 
2025-02-10T16:32:06.991989 - 
2025-02-10T16:32:07.009489 - 0: 640x384 3 faces, 10.5ms
2025-02-10T16:32:07.009489 - Speed: 1.0ms preprocess, 10.5ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:32:19.745952 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:32:19.745952 - 
2025-02-10T16:32:19.819564 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:32:19.819564 - 
2025-02-10T16:32:19.891062 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:32:19.891062 - 
2025-02-10T16:32:19.964062 - CLIP: []2025-02-10T16:32:19.964062 - 
2025-02-10T16:32:19.964062 - Requested to load SDXLClipModel
2025-02-10T16:32:20.883737 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:32:21.151738 - Detailer: segment upscale for ((87.924835, 126.03647)) | crop region (263, 378) x 2.0318871110135177 -> (534, 768)2025-02-10T16:32:21.151738 - 
2025-02-10T16:32:21.758206 - Requested to load SDXL
2025-02-10T16:32:23.380087 - loaded completely 5320.702197479248 4897.0483474731445 True
2025-02-10T16:32:48.723784 - 
100%|██████████| 35/35 [00:25<00:00,  1.61it/s]2025-02-10T16:32:48.723784 - 
100%|██████████| 35/35 [00:25<00:00,  1.39it/s]2025-02-10T16:32:48.723784 - 
2025-02-10T16:32:48.788096 - loaded completely 594.7089347839355 159.55708122253418 True
2025-02-10T16:32:49.057201 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:32:49.057201 - 
2025-02-10T16:32:49.076700 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:32:49.076700 - 
2025-02-10T16:32:49.151200 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:32:49.151200 - 
2025-02-10T16:32:49.224700 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:32:49.224700 - 
2025-02-10T16:32:49.302700 - CLIP: []2025-02-10T16:32:49.302700 - 
2025-02-10T16:32:49.302700 - Requested to load SDXLClipModel
2025-02-10T16:32:50.223818 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:32:50.497319 - Detailer: segment upscale for ((39.733932, 61.124237)) | crop region (119, 183) x 4.196873031396889 -> (499, 768)2025-02-10T16:32:50.497319 - 
2025-02-10T16:32:50.767956 - Requested to load SDXL
2025-02-10T16:32:52.314930 - loaded completely 5312.510795944214 4897.0483474731445 True
2025-02-10T16:33:15.438714 - 
100%|██████████| 35/35 [00:22<00:00,  1.79it/s]2025-02-10T16:33:15.438714 - 
100%|██████████| 35/35 [00:22<00:00,  1.54it/s]2025-02-10T16:33:15.438714 - 
2025-02-10T16:33:15.694902 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:33:15.694902 - 
2025-02-10T16:33:15.709901 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:33:15.709901 - 
2025-02-10T16:33:15.787403 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:33:15.787403 - 
2025-02-10T16:33:15.860902 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:33:15.860902 - 
2025-02-10T16:33:15.933901 - CLIP: []2025-02-10T16:33:15.933901 - 
2025-02-10T16:33:15.933901 - Requested to load SDXLClipModel
2025-02-10T16:33:16.819573 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:33:17.089574 - Detailer: segment upscale for ((17.894772, 47.417877)) | crop region (53, 142) x 5.408933573616332 -> (286, 768)2025-02-10T16:33:17.089574 - 
2025-02-10T16:33:17.236620 - Requested to load SDXL
2025-02-10T16:33:18.775844 - loaded completely 5332.410776901245 4897.0483474731445 True
2025-02-10T16:33:36.260095 - 
100%|██████████| 35/35 [00:17<00:00,  2.36it/s]2025-02-10T16:33:36.260095 - 
100%|██████████| 35/35 [00:17<00:00,  2.03it/s]2025-02-10T16:33:36.260095 - 
2025-02-10T16:33:36.418053 - [Impact Pack] vae decoded in 0.2s2025-02-10T16:33:36.418053 - 
2025-02-10T16:33:36.437553 - 
2025-02-10T16:33:36.453053 - 0: 640x384 1 face, 8.5ms
2025-02-10T16:33:36.453053 - Speed: 1.5ms preprocess, 8.5ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:33:40.912324 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:33:40.912324 - 
2025-02-10T16:33:40.988324 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:33:40.988324 - 
2025-02-10T16:33:41.064324 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:33:41.064324 - 
2025-02-10T16:33:41.139324 - CLIP: []2025-02-10T16:33:41.139324 - 
2025-02-10T16:33:41.139324 - Requested to load SDXLClipModel
2025-02-10T16:33:42.053930 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:33:42.325431 - Detailer: segment upscale for ((87.04758, 127.57683)) | crop region (261, 382) x 2.010796273369991 -> (524, 768)2025-02-10T16:33:42.325431 - 
2025-02-10T16:33:42.865047 - Requested to load SDXL
2025-02-10T16:33:44.402259 - loaded completely 5343.694297943115 4897.0483474731445 True
2025-02-10T16:34:09.727239 - 
100%|██████████| 35/35 [00:25<00:00,  1.62it/s]2025-02-10T16:34:09.727740 - 
100%|██████████| 35/35 [00:25<00:00,  1.39it/s]2025-02-10T16:34:09.727740 - 
2025-02-10T16:34:09.783724 - loaded completely 564.9731407165527 159.55708122253418 True
2025-02-10T16:34:10.003725 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:34:10.003725 - 
2025-02-10T16:34:10.032330 - 
2025-02-10T16:34:10.045740 - 0: 640x384 1 face, 6.5ms
2025-02-10T16:34:10.045740 - Speed: 1.0ms preprocess, 6.5ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:34:14.393602 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:34:14.393602 - 
2025-02-10T16:34:14.475602 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:34:14.475602 - 
2025-02-10T16:34:14.549102 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:34:14.549102 - 
2025-02-10T16:34:14.622609 - CLIP: []2025-02-10T16:34:14.622609 - 
2025-02-10T16:34:14.623109 - Requested to load SDXLClipModel
2025-02-10T16:34:15.548085 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:34:15.810586 - Detailer: segment upscale for ((81.641205, 116.52451)) | crop region (244, 349) x 2.2007728757630596 -> (536, 768)2025-02-10T16:34:15.810586 - 
2025-02-10T16:34:16.267032 - Requested to load SDXL
2025-02-10T16:34:17.971884 - loaded completely 5302.564728439331 4897.0483474731445 True
2025-02-10T16:34:43.418123 - 
100%|██████████| 35/35 [00:25<00:00,  1.61it/s]2025-02-10T16:34:43.418123 - 
100%|██████████| 35/35 [00:25<00:00,  1.39it/s]2025-02-10T16:34:43.418123 - 
2025-02-10T16:34:43.694722 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:34:43.694722 - 
2025-02-10T16:34:43.724746 - 
2025-02-10T16:34:43.738601 - 0: 640x384 1 face, 6.5ms
2025-02-10T16:34:43.738601 - Speed: 1.5ms preprocess, 6.5ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:34:47.892950 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:34:47.893450 - 
2025-02-10T16:34:47.968949 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:34:47.968949 - 
2025-02-10T16:34:48.043449 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:34:48.043449 - 
2025-02-10T16:34:48.120949 - CLIP: []2025-02-10T16:34:48.120949 - 
2025-02-10T16:34:48.120949 - Requested to load SDXLClipModel
2025-02-10T16:34:49.028449 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:34:49.302950 - Detailer: segment upscale for ((75.33392, 116.01729)) | crop region (226, 348) x 2.2070361674386896 -> (498, 768)2025-02-10T16:34:49.302950 - 
2025-02-10T16:34:49.819547 - Requested to load SDXL
2025-02-10T16:34:51.444040 - loaded completely 5316.917091720581 4897.0483474731445 True
2025-02-10T16:35:14.433568 - 
100%|██████████| 35/35 [00:22<00:00,  1.78it/s]2025-02-10T16:35:14.433568 - 
100%|██████████| 35/35 [00:22<00:00,  1.54it/s]2025-02-10T16:35:14.433568 - 
2025-02-10T16:35:14.691660 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:35:14.691660 - 
2025-02-10T16:35:14.724123 - 
2025-02-10T16:35:14.737124 - 0: 640x384 1 face, 6.5ms
2025-02-10T16:35:14.737625 - Speed: 1.0ms preprocess, 6.5ms inference, 1.5ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:35:18.923473 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:35:18.923473 - 
2025-02-10T16:35:18.998473 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:35:18.998473 - 
2025-02-10T16:35:19.071973 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:35:19.071973 - 
2025-02-10T16:35:19.144472 - CLIP: []2025-02-10T16:35:19.144472 - 
2025-02-10T16:35:19.144973 - Requested to load SDXLClipModel
2025-02-10T16:35:20.070474 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:35:20.339476 - Detailer: segment upscale for ((89.2077, 132.99567)) | crop region (267, 398) x 1.9298907577714854 -> (515, 768)2025-02-10T16:35:20.339476 - 
2025-02-10T16:35:20.827756 - Requested to load SDXL
2025-02-10T16:35:22.496309 - loaded completely 5374.560376068115 4897.0483474731445 True
2025-02-10T16:35:45.534764 - 
100%|██████████| 35/35 [00:22<00:00,  1.79it/s]2025-02-10T16:35:45.534764 - 
100%|██████████| 35/35 [00:22<00:00,  1.53it/s]2025-02-10T16:35:45.534764 - 
2025-02-10T16:35:45.598267 - loaded completely 441.48730087280273 159.55708122253418 True
2025-02-10T16:35:45.816278 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:35:45.816278 - 
2025-02-10T16:35:45.851852 - 
2025-02-10T16:35:45.864851 - 0: 640x384 2 faces, 6.5ms
2025-02-10T16:35:45.865351 - Speed: 1.0ms preprocess, 6.5ms inference, 1.4ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:35:53.921560 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:35:53.921560 - 
2025-02-10T16:35:54.002560 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:35:54.002560 - 
2025-02-10T16:35:54.247559 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:35:54.247559 - 
2025-02-10T16:35:54.320668 - CLIP: []2025-02-10T16:35:54.321168 - 
2025-02-10T16:35:54.321168 - Requested to load SDXLClipModel
2025-02-10T16:35:55.246167 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:35:55.513667 - Detailer: segment upscale for ((83.46866, 125.05523)) | crop region (250, 375) x 2.048235606528671 -> (512, 768)2025-02-10T16:35:55.513667 - 
2025-02-10T16:35:56.072560 - Requested to load SDXL
2025-02-10T16:35:57.664664 - loaded completely 5374.439747711182 4897.0483474731445 True
2025-02-10T16:36:20.652904 - 
100%|██████████| 35/35 [00:22<00:00,  1.79it/s]2025-02-10T16:36:20.652904 - 
100%|██████████| 35/35 [00:22<00:00,  1.54it/s]2025-02-10T16:36:20.652904 - 
2025-02-10T16:36:20.714105 - loaded completely 537.8686256408691 159.55708122253418 True
2025-02-10T16:36:20.929860 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:36:20.929860 - 
2025-02-10T16:36:20.946864 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:36:20.946864 - 
2025-02-10T16:36:21.023864 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:36:21.023864 - 
2025-02-10T16:36:21.099592 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:36:21.099592 - 
2025-02-10T16:36:21.175094 - CLIP: []2025-02-10T16:36:21.175094 - 
2025-02-10T16:36:21.175094 - Requested to load SDXLClipModel
2025-02-10T16:36:22.069032 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:36:22.340033 - Detailer: segment upscale for ((21.810347, 28.000977)) | crop region (65, 84) x 9.144749896733254 -> (594, 768)2025-02-10T16:36:22.340033 - 
2025-02-10T16:36:22.677637 - Requested to load SDXL
2025-02-10T16:36:24.326814 - loaded completely 5340.906896148682 4897.0483474731445 True
2025-02-10T16:36:51.883942 - 
100%|██████████| 35/35 [00:27<00:00,  1.48it/s]2025-02-10T16:36:51.884442 - 
100%|██████████| 35/35 [00:27<00:00,  1.28it/s]2025-02-10T16:36:51.884442 - 
2025-02-10T16:36:51.949769 - loaded completely 436.92428970336914 159.55708122253418 True
2025-02-10T16:36:52.199770 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:36:52.199770 - 
2025-02-10T16:36:52.235009 - 
2025-02-10T16:36:52.248009 - 0: 640x384 1 face, 6.5ms
2025-02-10T16:36:52.248009 - Speed: 1.0ms preprocess, 6.5ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:36:56.129621 - LOAD LORA: SDXL1_\add-detail-xl.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:36:56.129621 - 
2025-02-10T16:36:56.209122 - LOAD LORA: SDXL1_\SDXL_BetterFaces-LoRA_v1.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:36:56.209122 - 
2025-02-10T16:36:56.285122 - LOAD LORA: SDXL1_\PerfectEyesXL.safetensors: 1.0, 1.0, LBW=None, A=None, B=None2025-02-10T16:36:56.285122 - 
2025-02-10T16:36:56.363122 - CLIP: []2025-02-10T16:36:56.363122 - 
2025-02-10T16:36:56.363122 - Requested to load SDXLClipModel
2025-02-10T16:36:57.276122 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-02-10T16:36:57.545622 - Detailer: segment upscale for ((91.21991, 133.35849)) | crop region (273, 400) x 1.9201059695553695 -> (524, 768)2025-02-10T16:36:57.545622 - 
2025-02-10T16:36:58.004133 - Requested to load SDXL
2025-02-10T16:36:59.535221 - loaded completely 5307.341806564331 4897.0483474731445 True
2025-02-10T16:37:25.098300 - 
100%|██████████| 35/35 [00:25<00:00,  1.61it/s]2025-02-10T16:37:25.098300 - 
100%|██████████| 35/35 [00:25<00:00,  1.39it/s]2025-02-10T16:37:25.098300 - 
2025-02-10T16:37:25.365567 - [Impact Pack] vae decoded in 0.3s2025-02-10T16:37:25.366067 - 
2025-02-10T16:37:26.458399 - Loads SAM model: E:\ComfyUI-Desktop-V1\models\sams\sam_vit_b_01ec64.pth (device:Prefer GPU)2025-02-10T16:37:26.458399 - 
2025-02-10T16:37:26.533400 - [Impact Pack] WARN: FaceDetailer is not a node designed for video detailing. If you intend to perform video detailing, please use Detailer For AnimateDiff.2025-02-10T16:37:26.533900 - 
2025-02-10T16:37:26.620899 - 
2025-02-10T16:37:26.677899 - 0: 640x384 1 hand, 25.0ms
2025-02-10T16:37:26.677899 - Speed: 1.0ms preprocess, 25.0ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 384)
2025-02-10T16:37:26.775216 - !!! Exception during processing !!! Allocation on device 
2025-02-10T16:37:26.781716 - Traceback (most recent call last):
  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 327, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 202, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 174, in _map_node_over_list
    process_inputs(input_dict, i)
  File "C:\Users\MSI-Graphics\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 163, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\impact_pack.py", line 1438, in doit
    enhanced_img, cropped_enhanced, cropped_enhanced_alpha, mask, cnet_pil_list = FaceDetailer.enhance_face(
                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\impact_pack.py", line 559, in enhance_face
    sam_mask = core.make_sam_mask(sam_model_opt, segs, image, sam_detection_hint, sam_dilation,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\core.py", line 732, in make_sam_mask
    detected_masks = sam_obj.predict(image, points, plabs, dilated_bbox, threshold)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\custom_nodes/comfyui-impact-pack\modules\impact\core.py", line 603, in predict
    predictor.set_image(image, "RGB")
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\predictor.py", line 60, in set_image
    self.set_torch_image(input_image_torch, image.shape[:2])
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\predictor.py", line 89, in set_torch_image
    self.features = self.model.image_encoder(input_image)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 112, in forward
    x = blk(x)
        ^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 174, in forward
    x = self.attn(x)
        ^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 234, in forward
    attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\ComfyUI-Desktop-V1\.venv\Lib\site-packages\segment_anything\modeling\image_encoder.py", line 358, in add_decomposed_rel_pos
    attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
torch.OutOfMemoryError: Allocation on device 

2025-02-10T16:37:26.782717 - Got an OOM, unloading all loaded models.
2025-02-10T16:37:27.175089 - Prompt executed in 1253.31 seconds

Attached Workflow

Please make sure that workflow does not contain any sensitive information such as API keys or passwords.

Workflow too large. Please manually upload the workflow from local file system.

Additional Context

(Please add any additional context or steps to reproduce the error here)

Originally posted by @Zandebar in #32
@Zandebar
Copy link
Author

I posted this a week ago with the error being produce again after your fix for #32 not sure what is happening can you please advise, unable to show this error on #32 as ticket was closed.

@Zandebar Zandebar changed the title FaceDetailerPipe: torch.OutOfMemoryError - Allocation on device-2 - Desktop-V1 Again - FaceDetailerPipe: torch.OutOfMemoryError - Allocation on device-2 - Desktop-V1 Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Zandebar