I had(keyword) a few lightweight rifles, including a Kimber Montana in 308 that was 6lb 1oz with a scope and full magazine.
It’s not that light rifles shoot badly, they just magnify any form issues you have greatly. If you’re inconsistent, you’ll see it on the target. The 308, even with a side port muzzle brake still let you know when it went off. It wasn’t bad, but it definitely had more recoil than the 9lb 300wsm I’m shooting now(also braked).
The reason I ended up getting rid of it? It’s hard enough to shoot it consistently at the range - add in mountain hunting when finding a reasonably stable rest can be a challenge at best to impossible(the bear I missed this weekend

) at worst and yeah, my current rifle weighs a few more pounds but I’ll take all the advantages I can get. At 400-500 yards it started being quite a challenge to shoot consistently, whereas my shorty is quite forgiving even out to 700-800.
With all this said, my best advice to you is to adjust that trigger down to 2lbs(or thereabouts) and really work on your dry fire practice when not able to shoot. Get yourself a side port brake… and PRACTICE. You’re gonna need it. They’re also finicky in the way you hold them and how they respond - mine responded well to slight downward pressure on the scope - free recoil would be all over the target.
Mine was able to be adjusted down to 2lbs safely - please make sure you drop the rifle a few times/work the bolt HARD to make sure it’s safe. Mine was, but every rifle is different, and that’s not something you want to discover in the mountains.