Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Memory Growth Differ error in Multi-GPU #89

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gh-BumsooKim
Copy link

@gh-BumsooKim gh-BumsooKim commented Jul 22, 2021

In Multi-GPUs environment, previous code occur ValueError: Memory growth cannot differ between GPU devices in object_tracker.py file.

- Previous object_tracker.py

physical_devices = tf.config.experimental.list_physical_devices('GPU')

if len(physical_devices) > 0:
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

-Modified object_tracker.py : Line 8

physical_devices = tf.config.experimental.list_physical_devices('GPU')

if len(physical_devices) > 0:
    tf.config.experimental.set_visible_devices(physical_devices[0], 'GPU')
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

Environment

  • OS : Windows10
  • Envs : Anaconda
  • Python : 3.7.0
  • GPU : NVIDIA GeForce RTX 3090 24GB * 2 (Multi-GPU)
  • CUDA/cuDNN : 11.1.1/8.1.1

GPU is :

> nvidia-smi 
Mon Jul 19 16:01:51 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 456.81       Driver Version: 456.81       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090   WDDM  | 00000000:01:00.0  On |                  N/A |
| 67%   59C    P2   126W / 370W |   2221MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 3090   WDDM  | 00000000:02:00.0 Off |                  N/A |
|  0%   32C    P8     7W / 370W |    593MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

ValueError

Traceback (most recent call last):
  File "object_tracker.py", line 18, in <module>
    from core.yolov4 import filter_boxes
  File "C:\Users\USER\Downloads\yolov4-deepsort-master\yolov4-deepsort-master\core\yolov4.py", line 292, in <module>
    def filter_boxes(box_xywh, scores, score_threshold=0.4, input_shape = tf.constant([416,416])):
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\constant_op.py", line 264, in constant
    allow_broadcast=True)
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\constant_op.py", line 275, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\constant_op.py", line 300, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\constant_op.py", line 97, in convert_to_eager_tensor
    ctx.ensure_initialized()
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\context.py", line 524, in ensure_initialized
    config_str = self.config.SerializeToString()
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\context.py", line 986, in config
    gpu_options = self._compute_gpu_options()
  File "C:\Users\USER\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\context.py", line 1045, in _compute_gpu_options
    raise ValueError("Memory growth cannot differ between GPU devices")

ValueError: Memory growth cannot differ between GPU devices

See more about tensorflow docs : https://www.tensorflow.org/guide/gpu

In Multi-GPU environment, previous code occur `ValueError: Memory growth cannot differ between GPU devices`

See more about tensorflow docs : https://www.tensorflow.org/guide/gpu
@PHCLenzi
Copy link

PHCLenzi commented May 9, 2022

thank you very much @gh-BumsooKim !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants