I don't think you can be too fit, but I also think that there are diminishing returns when you get to higher levels of fitness. I would consider myself on the upper end of the fitness bell curve, yet realizing that there are many more folks that are far more fit than I am. So how much is good enough?
I guess you have to ask yourself how you want to hunt. How much ground and elevation do you want to or NEED to cover in order to hunt effectively? Are you fit enough to pack an animal out of the area you are in, doing so in a timely manner and without suffering an injury?
When I travel out of state to elk hunt, I want to be able to hunt every day of the trip, which is usually 7-14 days. I don't want to take rest days, unless it's because we tagged out and drank all the whiskey. The areas we hunt are usually either a 2-3 mile hike into each day, or a backpack trip in for 3-5 miles. We may hunt 2-3 miles from camp (depending on road density) sometimes a little more.
I guess to put some numbers to your question, last year on a deer hunt I went in 5.5 miles the first night, gained 2700 feet in elevation, and was carrying a 60# pack. I was able to cover that in a 3.5 hour period and that included time glassing on the way. The next day, I covered about 7 more miles in about 4 hours, including time spent filling up with water and chatting with some other hunters. I did not feel overly taxed at all. That is the level of fitness I want to be at for my hunts.