ValueError: Exception encrounted when calling layer "QNetwork" (type QNetwork)

[saandre15]

TF-Agent Version: 0.12.0
TF Version: 2.8.0
Python Version:3.7
So I'm trying to create a Q-Learning network for to train a computer to learn how to solve a rubrix cube. However, I have the issue where my input only takes 2 input instead of 1 input representing the action.

Initialization of Rubrix Cube

class RubricCubeEnv(PyEnvironment):
  
  def __init__(self, cube: RubricCube, randomize_steps: int):
    self._action_spec = BoundedArraySpec((), dtype=np.int32, minimum=0, maximum=cube.action_size() - 1, name='action')
    self._observation_spec = BoundedArraySpec((1, len(cube.to_matrix())), dtype=np.int32, minimum=0, maximum=len(cube.matrices) - 1)
    self.cube = cube
    self._state = cube.to_matrix()
    self._episode_ended = False
    self.action_chain = []
    self.observation_chain = []
    self.randomize_steps = randomize_steps

Here is how I setup my Q-Network

q_net = QNetwork(train_env.observation_spec(),
      train_env.action_spec(),
      fc_layer_params=(100,),
    )
      # preprocessing_combiner=tf.keras.layers.Concatenate(axis=-1))
    # Initalizes the agent
    agent = DqnAgent(train_env.time_step_spec(), train_env.action_spec(), q_net, adam_v2.Adam())
    agent.initialize()

Expectation:
I create a q learning nerual network that accepts one input from range 0-17 and spits out a prediction state of an array containing integers from 0-5 with a size of 54 to represent the state of each rubrix cube piece.

What I'm getting:
Error I get when creating my TensorFlow training.

Input 0 of layer "dense_2" is incompatible with the layer: expected min_ndim=2, found ndim=1. Full shape received: (40,)

Call arguments received:
  • observation=tf.Tensor(shape=(1, 54), dtype=int32)
  • step_type=tf.Tensor(shape=(1,), dtype=int32)
  • network_state=()
  • training=False

I do not know why it taking 2 input when the action generate provides 1 input.

[saandre15]

Hmmm, I got it to work by turning the
shape of the observation_spec boundedarrayspec from (1, len(cube.to_matrix())) to (len(cube.to_matrix()))

But now I'm getting the previous issue I had with OpenAI Gym
https://stackoverflow.com/questions/71261921/how-to-flatten-a-tensorflow-policystep-timestep-and-trajectory

How would I flatten a trajectory?

[saandre15]

It weird because TensorFlow environment automatically generates an observation of
<tf.Tensor: shape=(1, 54, dtype=int32, numpy=array([[list of values]]))>
however, I've already inputted
self._observation_spec = BoundedArraySpec((len(cube.to_matrix()),), dtype=np.int32, minimum=0, maximum=len(cube.matrices) - 1, name='observation')

It should only generate a 1D array and not a 2D matrix.

[saandre15]

When I remove TFPyEnviroment from the RubricCubeEnv, I get the 1D array, but now I'm back on the original problem.

[saandre15]

ValueError: The agent was configured to expect a sequence_length of '2'. Value is expected to be shaped [B, T] + spec.shape but at least one of the Tensors in value has a time axis dim value '1' vs the expected '2'.
First such tensor is:
value.step_type.
Full shape structure of value:
Trajectory(
{'action': TensorShape([1, 1]),
'discount': TensorShape([1, 1]),
'next_step_type': TensorShape([1, 1]),
'observation': TensorShape([1, 1, 54]),
'policy_info': (),
'reward': TensorShape([1, 1]),
'step_type': TensorShape([1, 1])})

[MarkosMuche]

Hey @dretechtips, I have encountered a similar problem but with TD3 Agent. Did you solve your problem?