Pointer offset issue after zoom/pan

[beltebelt]

I'm sure similar question has been asked here before, since it seems to be a common use case. But I couldn't find solutions for my issue on docs or discussions, hence posting a new question.

I am trying to implement zoom/pan features into my whiteboard app alongwith the freedrawing tool. The issue I face is, after I have zoomed in or out, I notice some offset between my pointer location and where the actual shape is drawn on the board. How do I fix this?

I have created a reproducible example in the codepen below, using code samples from the docs.
https://codepen.io/shubhendumadhukar/pen/wBwpaPq

1. Draw something using mouseclick and mousedrag.
2. Zoom using mousewheel
3. Draw something again. There should be an offset as mentioned before.