Reverb issue with DQN tutorial

[windowshopr]

Buckle up for this one. New user trying to get this working.

My environment is:

Python 3.8
Windows 10
Tensorflow 2.7.0
TF_Agents 0.11.0
(...which according to this page is correct. NOT using nightly)

So, when I run the DQN tutorial on my machine, I get:

Traceback (most recent call last):
  File "2main_tf.py", line 24, in <module>
    import reverb
ModuleNotFoundError: No module named 'reverb'

Which is fine as I ran my installs separately, but looking at the TF-Agents documentation, the command to run is pip install --user dm-reverb, but when I do that, I get:

ERROR: Could not find a version that satisfies the requirement dm-reverb (from versions: none)
ERROR: No matching distribution found for dm-reverb

Which isn't a TF-Agents problem I know, but still an issue. So when I then try pip install --user reverb, it imports reverb, but it looks like it's made for python 2.x as I get a syntax error in reverb.py, so that's not the answer either.

From there, I uninstalled tensorflow, tf-agents, tensorflow_probability, etc. etc., then ran a clean pip install --user tf-agents[reverb] command. During the installation, I see the warning:

WARNING: tf-agents 0.5.0 does not provide the extra 'reverb'

...but I just let it go for now. So now my versions of everything are:

tf-agents==0.5.0
tensorflow-probability==0.15.0

and no tensorflow yet, so looking at the documentation for tf-agents==0.5.0, we should have tensorflow-2.2.0, so I run

pip install tensorflow==2.2.0

ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.7; Detected an installation of version 2.2.0. Please upgrade TensorFlow to proceed.

So, looking at the releases page for probability, we want 0.10.0 for tensorflow 2.2.0, so I run

pip uninstall tensorflow_probability -y
pip install tensorflow_probability==0.10.0

...then when I run the project, I get the error:

File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow_probability\python\layers\distribution_layer.py", line 28, in <module>
    from cloudpickle.cloudpickle import CloudPickler
ImportError: cannot import name 'CloudPickler' from 'cloudpickle.cloudpickle' (C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\cloudpickle\cloudpickle.py)

Looking up this error I found this page which said they pushed a 0.10.1 update to fix this issue, so re-ran

pip uninstall tensorflow_probability -y
pip install tensorflow_probability==0.10.1

...then when I run the project, I get the error:

ImportError: cannot import name 'sequential' from 'tf_agents.networks'

...which is because the project is now using a different version of tf-agents than what I have installed, so now comes the manual changing of the imports to the proper naming conventions for tf-agents 0.5.0, as well as updating the calls to those classes in the code:

from tf_agents.networks import sequential_layer
...
q_net = sequential_layer(dense_layers + [q_values_layer])

ImportError: cannot import name 'reverb_replay_buffer' from 'tf_agents.replay_buffers'

etc etc. And I've ran out of gas in trying to manually do this line by line.

Can someone provide some guidance on how to do a proper install of everything I need to make this DQN Tutorial work? As you can see, I've tried lol

[sguada]

Try

pip install -force-reinstall tf-agents[reverb]

[windowshopr]

Thanks, I tried that now. Installed TF==2.2.0 but got that "probability" version error again (my bad), so I installed 2.7.0, and now get:

File "G:\Other computers\My Computer\RL_2021\envs\TF_Env.py", line 6, in <module>
    from tf_agents.environments import py_environment
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\environments\__init__.py", line 26, in <module>
    from tf_agents.environments import utils
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\environments\utils.py", line 25, in <module>
    from tf_agents.policies import random_py_policy
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\policies\__init__.py", line 18, in <module>
    from tf_agents.policies import actor_policy
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\policies\actor_policy.py", line 29, in <module>
    from tf_agents.networks import network
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\networks\__init__.py", line 18, in <module>
    from tf_agents.networks import actor_distribution_network
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\networks\actor_distribution_network.py", line 26, in <module>       
    from tf_agents.networks import categorical_projection_network
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\networks\categorical_projection_network.py", line 26, in <module>   
    from tf_agents.networks import network
  File "C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tf_agents\networks\network.py", line 33, in <module>
    from tensorflow.python.keras.engine import network as keras_network  # TF internal
ImportError: cannot import name 'network' from 'tensorflow.python.keras.engine' (C:\Users\maltesers\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\__init__.py)

...so this is probably some other kind of version issue?

[windowshopr]

I see that dm-reverb is only available on Linux based OS's, I wonder if that's an issue in all this? Would be good to see the DQN tutorial updated for all OS's.

UPDATE

Yeah, so upgrading to Tensorflow 2.7, the latest TF-Agents, and the latest TF-Probabilities works now, up until importing reverb, which is not available for Windows currently, so this issue can be closed, but it would be good to see the DQN tutorial updated with a non-OS specific package for handling the replay buffers as not everyone wants to use Colab/Linux :P

[BaderTim]

I'm experiencing the same issue. A complete tutorial without reverb or one that works on Windows would be nice.

[arslanqadeer]

@BaderTim , Following tutorial is written without reverb. I hope this helps:

https://github.com/tensorflow/agents/blob/master/docs/tutorials/9_c51_tutorial.ipynb