Solve the "KukaGymEnv.py" problem in the pyBullet sample code

[ZaWaLuDo77]

The "Kukagymenv" environment has the problem of failure to converge in any RL method training.

I found that the main reason is that in "def step2(self, action):", "self._kuka.applyAction(action)" is quoted more frequently than "p.stepSimulation()".
This will cause "stepSimulation" to fail to keep up with the execution of the action, causing problems when lifting kuka.
Therefore, we must execute p.stepSimulation() multiple times in each action.

The "def step2" code is replaced as follows:

def step2(self, action):

for i in range(self._actionRepeat):
  self._kuka.applyAction(action)
  for i in range(100):        #Add this area
      p.stepSimulation()
  if self._termination():
    break
  self._envStepCounter += 1
if self._renders:
  time.sleep(self._timeStep)
self._observation = self.getExtendedObservation()
done = self._termination()
npaction = np.array([
    action[3]
])

actionCost = np.linalg.norm(npaction) * 10.  
reward = self._reward() - actionCost
blockPos, blockOrn = p.getBasePositionAndOrientation(self.blockUid)


return np.array(self._observation), reward, done, blockPos[2]

Hope it will be helpful to other researchers, thank you.

Tienen

[erwincoumans]

Haven't used that KukaGymEnv for many years, thanks for reporting on this. 100 physics steps sounds quite a lot, have you tried with fewer?

[ZaWaLuDo77]

Thanks for your reminder~
I used 5 physical steps and it seems to do the trick. The actual measurement found that the higher the number of physical steps, the more stable the overall environment.