Reinforcement Learning

[harshraj22]

Benchmarking My Learning path for Reinforcement Learning.

Things I have idea about:

• States, Actions, Policy, Values
• There is something called belmann equation which is used for policy iteration, value iteration to solve RL problems
• Neural Networks (The global function approximators) are used as policy in deep reinforcement learning models.
• There is something called policy gradient, using which we can train Deep Neural Networks whose loss function need not be differentiable (ie. gradient is calculated without derivative). Not sure how correct I know now xp

P.S. Maintaining a references list here

[harshraj22]

Nov 28th, 2021

• Skimmed through the basics of Value Iteration and Q learning using this blog from kaggle.
• Would try out some of this on my own on Google Colab 😃

To Do:

• Go through Deep RL Course Blogs

[harshraj22]

• Value Iteration on Taxi-v3

[harshraj22]

Nov 29th, 2021

• Skimmed through the RL crash course. He was using pre-trained kind of models. Not too insightful.
• Went through this blog about policy gradient. Not understood completely, but seems a good resource.
• Spent time understanding policy gradients from Pieter Abbeel's video. See this video for intution about the trajectories he is talking about. (Dynamics Model = Frozen lake kind of env, where you don't always get the action you want to take)
• went through various pytorch implementations of Monte Carlo Policy Gradient (The REINFORCE algorithm). Most of the things seem clear now, except the fact that at some places, authors use torch.gather, whose reason is unclear. Would try to dig deeper into this tomorrow, along with implementing it on one of the gym environments tomorrow.

[harshraj22]

Nov30th, 2021

• went through Pieter Abbeel's video video again. This time, things were more clear.
• No updates on torch.gather.
• Would finish up the vanilla policy gradient implementation by today. Going through this implementation currently. Sounds insightful...
• Would look up in detail about value estimation methods before jumping on A2C or Actor Critic methods.

[harshraj22]

Some clarity about torch.gather[1] from this post. the gather method helps in selecting the probabilities corresponding to the already chosen actions, as the Neural network (policy) returns the probabilities corresponding to all actions.

[harshraj22]

To Do:

• Read out this promising looking blog about "intuitive explanation about policy gradient"

[harshraj22]

Dec 1st, 2021

[harshraj22]

Dec 2nd, 2021

• Went through TRPO and PPO video from Foundations of Deep RL Series and Deep RL bootcamp. Did not understand completely.
• Would go through some additional blogs & articles about the same.

[harshraj22]

• Went through this series of article about implementing PPO using pytorch. Things became more clear, hopefully after a couple of implementation code, it would have more clarity.

[harshraj22]

Dec 3rd, 2021

• Not much progress.
• watched videos on how to write better python code: by Raymond Hettinger
• Would try to finish up Actor Critic By today

[harshraj22]

Dec 4th, 2021

• Went through Phil's video, implementing PPO in pytorch. things seem clear now.
• Would go through a few more implementations, and shift to Actor-Critic, DDPG etc soon.

[harshraj22]

A few points to note:
(Comparison across Phil's implementation, and another popular implementation)

• The way Advantage function is calculated differs. Seems some difference in the critic's loss as well.
• Need to understand distributions package from pytorch.

Another popular PPO implementation, Another

---

To Do:

• Dig deep into the various Advantage functions, understanding the differences between the various implementations of PPO
• Clearly understand the difference between Monte Carlo (MC) and Temporal difference (TD)
• Read about Importance Sampling.

[harshraj22]

A set of best practices to follow while implementing PPO

[harshraj22]

Dec 5th, 2021

• Went through the basics of Advantage Actor Critic (A2C) through this freecodecamp blog. Doesn't seem too complex. Would go through some existing implementations to solidify my understanding about the same.

[harshraj22]

• Went through parts of Pieter Abbeel's video. Would try to finish it off tomorrow.

[harshraj22]

Dec 6th, 2021

• Not Much Progress
• Going through Self critical Sequence Training Paper, trying to understand the Reinforcement learning part that I skipped earlier. Hopefully, now I will have a better understanding of all this 🤞

[harshraj22]

Dec 7th, 2021

• No updates today. Would be on a small break.

[harshraj22]

Dec 8th, 2021

• Break

[harshraj22]

Dec 9th, 2021

• A good read about RL in computer vision

[harshraj22]

Dec 10th, 2021

• On a Break till 16th,
• Good Reads: intro to Bayesian Optimization

[harshraj22]

Dec 11th, 2021

• On a break

[harshraj22]

Dec 12th, 2021

• On a break
• Good Reads on RL overview and its usage in Computer Vision

[harshraj22]

Dec 13th, 2021

• On a break

[harshraj22]

Dec 14th, 2021

• On a break

[harshraj22]

Dec 15th, 2021

• On a break

[harshraj22]

Dec 16th, 2021

• On a break

[harshraj22]

Dec 17th, 2021

• Went through REINFORCE algorithm again.
• Going through the code for PPO. Would try to finally finish off this implementation by tonight.

[harshraj22]

Empty

[harshraj22]

Dec 27th, 2021

• reddit post for implementing and debugging RL
• Best practices for PPO

[harshraj22]

Discussions on successfully Training the PPO

• this reddit comment mentions about batch size, and memory buffer size.
• some other comments on debugging RL

[harshraj22]

this was a good. bug fix.