There's a pretty big difference between doable and advisable. That center-of-gravity shift is a big deal.
It's weird enough getting on a horse with a pack on, or a lot of upper body gear, especially if it shifts in the slightest - and nobody's going to have their gear cinched down tight like it's just part of your torso. Now imagine what happens with that shifting weight-blob if that horse bucks, side-hops, or just bolts. It's going to supercharge your momentum away from that horse, with awkward timing like a kind of two-strike blow on your muscles, with you trying to control your body mass to stay in the saddle as the horse does something, and then that pack hitting against your body or pulling it away from the horse again a second after - all while the horse has already shifted again underneath you.
So, wearing heavy gear - especially something like a pack that sticks off your body a bit, is the kind of thing you do if you just have no other choice. So find a different way to carry your gear, because there's virtually always a different choice.
Separately, horses safely max out at carrying 20% of their body weight, especially if we're talking working them over long distances and rugged terrain. A normal quarter horse is going to be about 1000 pounds, give or take a hundred. A dude in his hunting clothes alone can easily be 200 pounds. Add on a 10 pound rifle and a pound or two of scabbard, a 30 pound pack, 5 pounds of binos and bino harness gear, a couple pounds of water, and you're putting more strain on that horse than is healthy for it. People get away with more all the time, sure, but it's not healthy for the horse - especially if there's bouncing, jostling, or a lot of rugged terrain causing that weight to pound down.