The general consensus is that light or ultra light (depending on how you view them) rifles, are in general more difficult to shoot accurately. Personally, for me, I have no problem shooting light rifles. But then I grew up shooting and hunting. As a kid, I went out for rabbits, quail and dove almost every day, during the season, by myself. I would not shoot unless I could hit them in the head. Subsequently, I became a pretty extraordinary shot. In highschool, I was a part of a group that went shooting every weekend, competing with each other with rifle, pistol and shotguns. And the story continues. But since this is not about me, I will continue.
Shooting light rifles is about form, achieving a stable solid rest and lots and lots of practice. A heavier rifle simply stabilizes much easier; but nobody wants to pack a heavy rifle miles and miles in the backcountry, unless someone or something else is carrying it. So my advice is to first analyze your hunting style.