Multi-threaded or parallel processing with CPUs: how to set via Device

[hmf]

Hi,

I was looking at taking advantage of multiple CPUs.
In the tutorial we use:

Device.getDevices

to set the number of GPUs to use. However I don't see how we can
set the number of CPUs. In fact, in the source code I always see only
one CPU reference used.

When I look at the thread use however, I see about 8 threads. So I have
two questions:

1. Are those threads used for training and evaluation?
2. If they are how can I increase the number of CPUs used?

TIA

[stu1130]

@hmf You can't set CPU index as native engine(TensorFlow/PyTorch/MXNet) use CPU as a pool. Usually operator is executed in multithreading way.

1. Yes
2. the native engine will consume as many cpu cores as possible unless you explicitly set the not to. For example, to prevent native threading from competing the resource with Java thread. In multithreading inference, we set OMP_NUM_THREAD=1 which only allows operator to use 1 thread

[hmf]

Missed the page https://djl.ai/docs/development/inference_performance_optimization.html but your reference to OMP_NUM_THREAD helped me.

the native engine will consume as many cpu cores as possible

Strangely enough I have 12 CPUs (6 cores) but it looks like only 8 get picked up.
Cannot say for sure. This is the reason why I asked.

I did not set MXNET_ENGINE_TYPE so will experiment further with these flags.

Thank you

EDIT: just realized link above is just for inference and not training

[stu1130]

If you are using MXNet and doing the inference, we recommend to do

export MXNET_ENGINE_TYPE=NaiveEngine
export OMP_NUM_THREADS=1

The only variable left is java thread
If you have 12 cpu cores, you can set Java threads to 12.

[frankfliu]

For you train, you should not set OMP_NUM_THREADS and MXNET_ENGINE_TYPE.
For thread options for MXNet, please read: https://mxnet.apache.org/versions/1.7.0/api/faq/env_var

[hmf]

Nice. Going to try this.
Thank you.