RAVE Output Gives: "TypeError: An op outside of the function building code is being passed"

[RMCrean]

Hello,

I am trying to run RAVE using unbiased MD simulation frames as input. I have a total of 6 OPs (all CA-CA distances) determined using AMINO. Unfortunately I obtain several error messages when I attempt to run it (full output at the bottom of this message) which I am not sure how to resolve. Please could you take a look and see if you can figure out where I am going wrong?

Background
I have git cloned RAVE into my testing directory and updated the necessary parameters inside the files: "Analyze_prave.py" + "COLVAR2npy.py" + "P_rave.py" for my system. (Included in Zipped document attached).
Also in this directory is the "input" folder with my colvar file and the "output" folder.

I am using an Anaconda environment with python version: 3.8.5, [GCC 7.3.0]
My (relevant) python Modules are the following versions:
tensorflow 2.3.1
tensorflow-estimator 2.3.0
Keras 2.4.3
Keras-Preprocessing 1.1.2
numpy 1.18.5

Below you will find my input command and the output (similar results were obtained when I also tried with a GPU minus the GPU missing warning). A copy of my input files/directory setup is also attached if needed.

Please also do let me know if you need anymore information.

Kind regards and thank you,
Rory

RAVE_Issue.tar.gz

When I try to run RAVE using:
`[x_rorcr@tetralith1 0_RAVE_Round0]$ ls */
RAVE/:
Analyze_prave.py COLVAR2npy.py LICENSE P_rave.py README.md

input/:
COLVAR_PTP1B_RAVE_Round0_0

output/:
[x_rorcr@tetralith1 0_RAVE_Round0]$ python3 RAVE/P_rave.py`

I obtain the following output:
`[x_rorcr@tetralith1 0_RAVE_Round0]$ python3 RAVE/P_rave.py
2020-10-12 11:33:22.654852: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-10-12 11:33:22.654907: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
length of data:99999
number of order parameters:6
min reweighting factor:1.000000
max reweighting factor:1.000000
2020-10-12 11:33:25.066710: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-12 11:33:25.109185: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2020-10-12 11:33:25.109296: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: tetralith1.nsc.liu.se
2020-10-12 11:33:25.109322: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: tetralith1.nsc.liu.se
2020-10-12 11:33:25.109522: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 430.50.0
2020-10-12 11:33:25.109605: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 430.50.0
2020-10-12 11:33:25.109626: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 430.50.0
2020-10-12 11:33:25.110855: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-12 11:33:25.135616: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2100000000 Hz
2020-10-12 11:33:25.138767: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5638d4fe3ae0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-12 11:33:25.138811: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
80000 data points are used in this training
80000 data points are used in this training
Epoch 1/20
Traceback (most recent call last):
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
@tf.function
def has_init_scope():
my_constant = tf.constant(1.)
with tf.init_scope():
added = my_constant * 2
The graph tensor has name: input_1:0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "RAVE/P_rave.py", line 269, in
History = prave.fit( [train_x,train_w1,train_w2], train_y,
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
tmp_logs = train_function(iterator)
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 780, in call
result = self._call(*args, **kwds)
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
return self._stateless_fn(*args, **kwds)
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2829, in call
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1843, in _filtered_call
return self._call_flat(
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 545, in call
outputs = execute.execute(
File "/home/x_rorcr/miniconda2/envs/myownenv_Py3_ML/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 72, in quick_execute
raise core._SymbolicException(
tensorflow.python.eager.core._SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'input_1:0' shape=(2000, 6) dtype=float32>, <tf.Tensor 'input_2:0' shape=(None, 1) dtype=float32>, <tf.Tensor 'input_3:0' shape=(None, 1) dtype=float32>]
`

[spiwokv]

We have the same error :-(