Is it possible to use more recent CUDA 11.x with 10.x libraries?

[hmf]

In the initial experiments I was able to put MxNet working on a GPU. I now upgraded the OS to Linux 20.04. So the default CUDA is 11.2 with gcc 9. The driver is working fine and I don't won't to mess with the set-up - finally I can use on-demand GPU with the apps. In addition to this the compute nodes I use also upgraded to 11.2 - MxNet with GPU was also ok with 10.2.

While installing CUDA I noticed that my old CUDA 10.2 compiled examples can still run on the 11.2 run-time. So my question is, is it possible to "force" DJL to use the GPU version of the native libraries? I tried just including the GPU libraries in the Java path but I still get:

14:42:42.085 [main] INFO  examples.djl.TrainMNIST$ - runExample: parse arguments
14:42:42.170 [main] DEBUG ai.djl.mxnet.jna.LibUtils - Using cache dir: /home/hmf/.djl.ai/mxnet
14:42:42.531 [main] WARN  ai.djl.mxnet.jna.LibUtils - No matching cuda flavor for linux found: cu112mkl/sm_61.
14:42:42.534 [main] DEBUG ai.djl.mxnet.jna.LibUtils - Loading mxnet library from: /home/hmf/.djl.ai/mxnet/1.7.0-backport-mkl-linux-x86_64/libmxnet.so
14:42:42.726 [main] DEBUG ai.djl.engine.Engine - Engine loaded from provider: MXNet
14:42:42.729 [main] DEBUG ai.djl.tensorflow.engine.LibUtils - Using cache dir: /home/hmf/.djl.ai/tensorflow
14:42:42.746 [main] WARN  ai.djl.tensorflow.engine.LibUtils - No matching CUDA flavor for linux found: cpu/sm_61, fallback to CPU.
14:42:42.746 [main] DEBUG ai.djl.tensorflow.engine.LibUtils - Loading TensorFlow library from: /home/hmf/.djl.ai/tensorflow/2.3.1-cpu-linux-x86_64/libjnitensorflow.so
2021-03-25 14:42:43.180459: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
14:42:43.181 [main] DEBUG ai.djl.engine.Engine - Engine loaded from provider: TensorFlow
14:42:43.181 [main] WARN  ai.djl.engine.Engine - More than one deep learning engines found.
14:42:43.182 [main] DEBUG ai.djl.engine.Engine - Found default engine: MXNet

I assume that all I need to do is not use the auto but the trace above still seems to show that DJL is autmatically identifying the CUDA version.

TIA

[zachgk]

I think this is more on the MXNet engine side. MXNet has a specific build for each version of CUDA so it is expecting that version.

We package up a few builds of MXNet for a few different CUDA flavors. All the auto does is choose between our packaged builds based on your system. Right now, we haven't done the CUDA 11.x ones but I am working on releasing some with the new MXNet 1.8 right now. This will include a build of CUDA 11.0 as that is what MXNet seems to support by their install page.

You can also use a custom build of MXNet with DJL that would work on other CUDA flavors, environments, etc. Just specify the path to the MXNet build with the environment variable MXNET_LIBRARY_PATH and DJL will use that.

[hmf]

Thanks for the information. I assume this also applies to the other engines. If not please let me know.

You can also use a custom build of MXNet with DJL that would work on other CUDA flavors, environments, etc. Just specify the path to the MXNet build with the environment variable MXNET_LIBRARY_PATH and DJL will use that.

Just to check. You mean use an mxnet:*:* target to build the DJL library. Find were the *.so are and set the MXNET_LIBRARY_PATH to point there. Then use my project as is, and DJL 1.7 will pick up the *.so. Seeing as I have cloned the DJL repo this seems feasible for my machine, but for the compute nodes this is not doable.

Please note, in the question above I am asking if there is any way to bypass the ai.djl...LibUtils. In other words, instead of automatically determining the name cu112mkl/sm_61 can I force a cu102mkl/sm_61 for the libs that already exist. I know I am asking for it, but if this worked then I could use the compute nodes. Also, this may be an escape route for those that cannot control what libs are installed in the OS.

TIA

[frankfliu]

@hmf
Yes you supply your own version of libmxnet.so file:

export MXNET_LIBRARY_PATH=/anaconda3/lib/python3.7/site-packages/mxnet/

See: http://docs.djl.ai/docs/development/troubleshooting.html#4-how-to-run-djl-using-other-versions-of-mxnet