For low recoil and high performance on deer you might want to try the 120gr ballistic tip nosler. I’ve used it for many years and it’s been a solid performer. Recoil is mild, not much more than a .243.
I have this pack for sale due to ending up with 2 of them and only needing one. Brand new with tags. This is the lightest weight grey version. Weighs about 1.25 lbs. great little pack, made in USA. $80 delivered.
I have the same situation as you and just do as you are saying. I just use 400 grm insulated boots with the boot blankets and warmers in them. That does it for me