inter_op_num_threads and intra_op_num_threads

[formater]

Hi,
I saw that in utils_vad.py we have:

        opts = onnxruntime.SessionOptions()
        opts.inter_op_num_threads = 1
        opts.intra_op_num_threads = 1

        if force_onnx_cpu and 'CPUExecutionProvider' in onnxruntime.get_available_providers():
            self.session = onnxruntime.InferenceSession(path, providers=['CPUExecutionProvider'], sess_options=opts)

And browsing the ONNX documentation ( link ), they say if these are set to 0, then ONNX will decide how many CPU cores to use.
Would it be wise to set these to 0 to get the maximum performance utilizing all cores, or there was some other underlying intention, and we must keep it at 1?

Thanks

[snakers4]

Hi,

Would it be wise to set these to 0 to get the maximum performance utilizing all cores, or there was some other underlying intention, and we must keep it at 1?

It is best to keep it at 1 for two reasons:

• Using more may be slower (and sometimes libraries default to using all cpu threads, which massively slows down the whole CPU);
• 1 is enough for the VAD, it is small and optimized enough;

[formater]

Thank you for your answer! :)

And if I have a WebSocket server, that handles 100 audios at the same time, and each needs VAD, and I want Silero-VAD to use ONNX and run on CPU, what do you suggest?

• Have a single model loaded, and the underlying ONNX framework will handle the concurrency when calling from different threads at the same time? (and still have inter_op_num_threads and intra_op_num_threads at 1)

• Make a pool of 10 models and use Queue to get a free one? Or it's not feasible with ONNX CPU version?

• Something else? :)

  For the GPU version I saw the example code, but I read conflicting informations about ONNX models on CPU :(

[snakers4]

First off, you see, the model is not stateless. It keeps state, i.e. is semi-recurrent. The ONNX wrapper shows it explicitly.

Keeping this in mind, the best option is to:

• Adapt the ONNX wrapper so you can pass state back-and-forth;
• Have some separate job / worker / process / executor that loads the model from scratch;
• Be able to launch several instances of this job on demand;
• Have some messenging / routing system that passes chunks to these workers in an RPC fashion;
• Ideally isolate the compute resources between the workers;
• Do not forget to set the thread limit when loading each model.

This can be anything - gRPC (complicated), FastAPI, celery, rabbit-mq, or even built-in python classes like ProcessPoolExecutor (with custom initialization procedure).

Note that both Pytorch and onnx-runtime create internal C++ objects, so sharing the objects between python threads or processers may be problematic. It is best to load from scratch in each process / job.

Have a single model loaded, and the underlying ONNX framework will handle the concurrency when calling from different threads at the same time? (and still have inter_op_num_threads and intra_op_num_threads at 1)

It would work for large chunks of compute per invocation. This model is very small and applied to only one audio chunk.

Make a pool of 10 models and use Queue to get a free one? Or it's not feasible with ONNX CPU version?

This works fine if all of the above precautions are taken into consideration.

Something else?

You can consider batching and setting higher thread numbers, but the communication code may get more involved and error-prone. And maybe we cut some corners somewhere when packaging the model. At first we offered the users this option, but there was too much misunderstanding because of batching + recurrent model.

[formater]

Thank you!