I just finished a pretty rough elk hunt; my first mountain hunt in barefoot boots 100% of the time and was afraid of this happening to me. I am using resoled Jim green troopers which I’d look at if you really like barefoot boots. More solid construction than vivo trackers for sure and to my surprise, even the pack out, zero issues, not even sore feet like I would get in my kenetreks.
Now it was also wet or snowy for 5 of 6 days. Waterproof socks worked decently, but after 4 days my boots were just wet, but so were my buddies kenetreks.
The biggest downside of the barefoot boot imo now having used it from bouldering to snow and most in between; steep snowy terrain. My buddies could easily grab going up, down and side hilling, I think because of the rigid edge of the boot. I was swearing in my Jim greens and wishing I had my micro spikes, which imo in a scales with for barefoot boots in steep slick conditions. I resorted to supporting my foot on a planted trekking pole asa fall would have resulted in serious injury.
Anyways all that to say don’t give up. I’d explore some different boots. Jim green or those new nicks look to be a more appropriate boot for rough terrain.