multiple IExecutionContext for getDeviceMemorySize

[tp-nan]

I have multiple IExecutionContexts, each with an different OptimizationProfile.

virtual size_t nvinfer1::ICudaEngine::getDeviceMemorySize() const

      Return the amount of device memory required by an execution context.

Is ICudaEngine's getDeviceMemorySize unrelated to OptimizationProfile?

[zerollzeng]

I think the OptimizationProfile was determined when you build the engine, so each each engine has its own OptizationProfile.

I have multiple IExecutionContexts, each with an different OptimizationProfile.

In this case I think What you mean is multiple ICudaEngine with different IOptimizationProfile?

Is ICudaEngine's getDeviceMemorySize unrelated to OptimizationProfile?

I think the answer it's no, IOptimizationProfile can affect the size returned by getDeviceMemorySize.

These are just my understanding, @ttyio @pranavm-nvidia please correct if I made any mistake :-)

[tp-nan]

Hi, thanks for your reply.

I build one ICudaEngine with multiple IOptimizationProfiles and multiple IExecutionContexts, and get Device Memory by ICudaEngine.getDeviceMemorySize, then I call IExecutionContext's setDeviceMemory. So I'm wondering this device memory size is sum of all IExecutionContexts(IOptimizationProfiles), or is only for one IExecutionContext? If the answer is the former, when I call setDeviceMemory for each IExecutionContext, is there any wasted memory? If the answer is the later, does IExecutionContext with different IOptimizationProfile range need the same device memory size?

[zerollzeng]

or is only for one IExecutionContext?

I think it's this.

does IExecutionContext with different IOptimizationProfile range need the same device memory size?

I think the memory size should be different(but not always, it depends on kernel tactics), you can verified this by experiment.

[tp-nan]

or is only for one IExecutionContext?

I think it's this.

does IExecutionContext with different IOptimizationProfile range need the same device memory size?

I think the memory size should be different(but not always, it depends on kernel tactics), you can verified this by experiment.

Thx for your reply. If so, I would suggest getDeviceMemorySize should be a method of IExecutionContext/IOptimizationProfile, not ICudaEngine. It is quite strange and I donot quite understand...

[zerollzeng]

@oxana-nvidia @nvpohanh any comments here ^ ^

[oxana-nvidia]

We are currently discussing possible API changes in this area for the next major release.

[nvpohanh]

I think the answer it's no, IOptimizationProfile can affect the size returned by getDeviceMemorySize.

@zerollzeng I don't think this is correct. I think engine->getDeviceMemorySize() returns the maximum of the device mem sizes from all optimization profiles. So the value is a fixed value, regardless of which profile(s) the contexts are set to.

[zerollzeng]

Oh my understanding is wrong. sorry.

[nvpohanh]

No worries! As Oxana said, we are discussing some improvements on these APIs in next TRT version. Hopefully things will be improved in the future

[ttyio]

closing since no activity for more than 3 weeks, thanks all