zoom in + control net video guidance

[agustincaniglia]

So, I want to make a video with a control net video as input, where I have a walking man in the same position.
I also want a constant zoom in in the general video, but the zoom gets applied to my control net video as well so my walking man gets out of frame in a couple of frames. How can I keep the zoom in camera noise movement + my controlnet video fixed?
thanks!