By far, the biggest downside of a new job is losing the PTO you've banked for years. Thanks to Covid, the landscape of my work (Epic builder/analyst) has drastically changed and allowed me to move on to greener pastures that did not exist pre-Covid. (fully remote work)
My previous employer, where I'd worked for almost 16 years, not only tolerated a toxic work environment, they seemed to foster it. Laziness, incompetence and downright stupidity were ignored and a culture of "good work was rewarded with mountains of more work" was in place. A combination of below market-value salary and the toxic work environment were taking a toll on me mentally, the job was turning me into someone didn't want to be. For my own mental and physical health, I had to move on.
So at the end of August I resigned my position and walked away smiling. I have zero regrets in making the job change. I'm now part of a fantastic team of people and make WAY MORE money doing the exact same work. Unfortunately I went from over 400 hours of PTO to 16 (starting out). While the PTO hit sucks for sure, in the grand scheme of things, I'm much better off for taking the plunge and moving on from a toxic job.
Thankfully, because I had over 400 hour of PTO that was paid out in full when I resigned, I was able to take a month off between jobs. My understanding wife (who had to be a single parent while I was gone), let me spend almost 3 of those weeks out in ID chasing elk this September! The biggest downside, however, is that the next few years I'll have to use my PTO sparingly so that I'll have enough to spend 2 weeks every fall chasing elk out west somewhere.
-BHWI