No, quite the opposite. If you only have 5 days to hunt all year, but bust your ass to make the most of every minute of your time hunting and have put a ton of effort into planning your hunt and your preparation - you should absolutely buy premium glass if you can afford it.
On the other hand, if you spent more time thinking about what glass to buy rather than getting fit for your hunt - it probably doesn’t matter too much what use if you can’t even get to where the animals are.
Now this is 180 to my thinking, most guys i know that are anal about gear, put a lot thought and effort into choosing it tend to put effort into using hunting as well.
Equating any of that to ones fitness or lack there is a serious stretch.