Tons of comments in here feel like excuses for why someone is not as successful as others in life.
OP- work hard early in your career to set yourself up to have plenty of discretionary income. There is often lots of personal sacrifice. Some guys are willing to, some guys choose not to. I devoted almost 4 years of my life to my profession, basically putting it first. My wife and I knew it was going to be a rough 4 years, but in the end, it would be all worth it. Fast forward 12 years and I live a pretty comfortable life and have been on some hunts many dream of going on. I still choose to not live my life above my means and have money for things I enjoy in life. I still live in my same house I bought in 2009. This affords me the money and time away from work to do the things I enjoy. I could work more hours and make considerably more money but at my stage in life, I look for a balance, leaning toward family time and experiences.
To each their own, but I rarely look at other people and assume they are in debt. I just don't care if they are, that is their choice and the only real reason to constantly say things like that are to justify your current situation.