None of my lightweight rifles shoot terrible. Stay within the means of what the rifle will do and the rest is up to the shooter not the gun.
All about practice. If people dont want to become proficient with their rifle then having a heavier rifle wont help either.
Me personally I would stick to the 7mm-08.
But if you decide to go 308 and want to sell the 7mm-08, I would be interested. I have the Adirondack in 6.5 creed