With all due respect, anecdotal memories are what they are. And without actual data to back it up, you really have no idea what has an effect on buck age class.
Did you listen to the podcast? In eastern Montana ~50% of bucks taken are 4pt or better and 4.5yrs and older.
I do agree that sustained doe harvest is foolish. We need does to make more deer. Very simple. Past that, I will not make assumptions because I am neither a biologist nor have seen studies that claim causation. I suspect you haven't either. Could be habitat changes. Could be weather/climate. It could be buck harvest during the rut. But without data, the cause is just a guess.
I am in the more opportunity camp which I'm sure comes across. My family of 4 loves to hunt and we eat solely wild game for our red meat. Hunting is a very important part of our family bonding each year and an incredible learning and growing opportunity for my kids. That's important to me. I honestly could care less about chasing big bucks if it means I don't put meat in the freezer and don't watch my kids and wife succeed in killing.