I ran a small Blackhawk fanny pack with my pistol inside and a Kelty baby carrier when my son was a baby whenever we went hiking. For most of that time period, we were living and working in El Salvador and due to my position at the U.S. embassy, I was authorized by the Salvadoran government with 24/7 carry. I was always armed in that country, especially when we went hiking in the mountains and forests. The baby carrier waist belt didn't have any straps or MOLLE to hook a pocket on like other packs do so I went with the fanny pack and looped it over the waist belt. It carried much better that way than having the fanny pack strap under the pack belt. Also, that way it doesn't rub or chafe any more than you would carrying the pack by itself.
I tried to shift the fanny pack to either hip since carrying it in front of you will nail your groin bad. I ended up swinging it behind the small of my back since the actual baby carrier rode high and there was room under my son's seat where the fanny stayed. I kept the fanny pack strap loose enough that i could swing it around 180 degrees with 1 yank of the belt and still get to my weapon quickly without flagging myself or my son.
All in all, it worked OK but having quick access to a concealed pistol was the only reason I used that set up. Otherwise, I would have just put whatever I wanted to carry in my pack. I prefer that setup over the battle belt for recreational hiking/hunting.