Recurrent networks quantized_bits with alpha

[laurilaatu]

quantized_bits should be the same using the default value None and alpha = 1.0.

scale set as 1.0 with self.alpha = None:
https://github.com/google/qkeras/blob/master/qkeras/quantizers.py#L550

scale set as 1.0 with self.alpha = 1.0:
https://github.com/google/qkeras/blob/master/qkeras/quantizers.py#L597

And can they return the same values as demonstrated with this example:

q_noalpha = quantized_bits(14, 4, 1)
q_alpha = quantized_bits(14, 4, 1, alpha=1.0)
testvalues = np.arange(-30,30,0.00001, dtype='float32')
np.argwhere((q_noalpha(testvalues) == q_alpha(testvalues)) == False)

Which returns an empty array meaning that both quantizers return the same values.

However with recurrent networks changing between these two changes the result. I have attached a small reproducible output that is based on the test code from qrecurrent_test.py to demonstrate the behavior. QLSTM and QSimpleRNN both give results that do not match.

np.random.seed(31)

tf.random.set_seed(31)

inputs = 2 * np.random.rand(10, 2, 4)

rnn = qkeras.QSimpleRNN

x = x_in = Input((2, 4), name='input')
x = rnn(
16,
activation=quantized_tanh(bits=8),
kernel_quantizer=quantized_bits(8, 0, 1, alpha=1.0),
#recurrent_quantizer=quantized_bits(8, 0, 1, alpha=1.0),
bias_quantizer=quantized_bits(8, 0, 1, alpha=1.0),
state_quantizer=quantized_bits(4, 0, 1, alpha=1.0),
name='qrnn_0')(
    x)
x = qkeras.QDense(
  4,
  kernel_quantizer=quantized_bits(6, 2, 1, alpha=1.0),
  bias_quantizer=quantized_bits(4, 0, 1),
  name='dense')(
      x)
x = Activation('softmax', name='softmax')(x)

model = Model(inputs=[x_in], outputs=[x])


# save weights
save_weights = model.get_weights()

original_output = model.predict(inputs).astype(np.float16)


x = x_in = Input((2, 4), name='input')
x = rnn(
16,
activation=quantized_tanh(bits=8),
kernel_quantizer=quantized_bits(8, 0, 1),
#recurrent_quantizer=quantized_bits(8, 0, 1),
bias_quantizer=quantized_bits(8, 0, 1),
state_quantizer=quantized_bits(4, 0, 1),
name='qrnn_0')(
    x)
x = qkeras.QDense(
  4,
  kernel_quantizer=quantized_bits(6, 2, 1),
  bias_quantizer=quantized_bits(4, 0, 1),
  name='dense')(
      x)
x = Activation('softmax', name='softmax')(x)

model = Model(inputs=[x_in], outputs=[x])

model.set_weights(save_weights)


output_no_alpha = model.predict(inputs).astype(np.float16)

print(original_output-output_no_alpha)

Expected output:

[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]

Actual output:

[[-0.02148  -0.0864   -0.01233   0.12036 ]
 [ 0.02441  -0.03333   0.01453  -0.00537 ]
 [ 0.08765  -0.0879   -0.003296  0.003418]
 [ 0.02734  -0.1011   -0.006287  0.0801  ]
 [-0.005615 -0.0481    0.03137   0.02222 ]
 [ 0.004883 -0.0454    0.001862  0.03857 ]
 [-0.04565  -0.03235  -0.00908   0.0874  ]
 [-0.007324 -0.04956  -0.002075  0.05884 ]
 [ 0.02588  -0.04425   0.01611   0.001953]
 [ 0.02344  -0.04398   0.0426   -0.02197 ]]

[YogaVicky]

I think you should use model_save_quantized_weights(model) to save the quantized weights.