If you want to zoom into a particular point in view space (along a ray in 3D space) you should probably not be using the hover controller, at least not while zooming, because the hover controller will dictate where the camera points to.
Because of how 3D perspective work, you can’t zoom into a certain point of the 2D view plane (or at least not easily, nor is it desirable). It would create rather weird perspective, sort of as if you took a photo where the vanishing point is in the center and then cropped and zoomed into the top left corner of it in Photoshop. As you can imagine, that will leave you with a rather odd perspective.
Instead, what you have to do, is lock the orientation of the camera when the user starts zooming, but move it along it’s local X/Y axes (e.g. using moveLeft(), moveRight(), moveUp(), moveDown()) to re-align around the mouse position. The math probably doesn’t need to be perfect, so you can probably do just a basic estimation of where the camera should be every frame, and animate it towards that point. At the same time, change the FOV as you have suggested to get the zoom effect you’re after.
This will prove problematic with the hover controller, because even if you stop updating while zooming, as soon as you resume updating the controller it will snap back to look at the lookAtTarget.
Another way that you may be able to achieve it using the HoverController though is to have a dummy look-at target (e.g. an empty container) and when the user zooms towards some off-center point you move the target towards that point. When it’s time to go back, you just animate the target back to it’s center-position.
It’s hard to give a great explanation when I have no references to what exactly it is that you’re doing. If the above comes off as confusing, please provide a more in-depth explanation of your project, what it is that you’re displaying, why you are using the hover controller et c, which might help us give better explanations.
