I think that the most important time to have control is at very slow speeds
The problem with that is that when using small control surfaces they are fairly ineffective at lower velocities. I have seen one set of tests that showed little effect under 250fps. That is partly dependent on the throw of the system at the time but in line with my expectations. If you make the fins bigger, static stability is compromised and dynamic control is more twitchy and harder to control.
As to noise and minor movements when traveling fast, these can be dealt with appropriately by the control loop(s). That is what they are there for. Remember, system noise (externally and internally generated) is part of the input variables too, and needs to be dealt with in the system design.
I guess controlling a motor gimbal would be a way to get good low speed control of the airframe, but you pay that weight penalty for the rest of the flight after burnout.