I agree with your idea of being able to have your handgun with you after dropping your pack and or hydration rig. I also like to drop my binocular harness when fishing, just hiking or hanging out in camp, but still have my firearm readily available.
I use Diamond D Custom leather Guides Choice Chest holster with a lightweight (12 oz. 3") handgun as a stand alone rig. I can then add whatever else I need, bino harness, hydration rig. day pack or multi day pack depending on my needs that day. The holster is accessible with right or left hand and is totally out of the way at least for me. With this set up I can put on and take off packs, bino harnesses and still maintain access to my firearm. Photo is with Kuiu Bino harness without the cover and an old rangefinder pouch which has been replaced by newer Kuiu rangefinder pouch. Good luck, stay safe