There's no doubt being colorblind hinders glassing ability. People saying it doesn't impact them negatively have nothing to compare against, other than their buddies who spot less game than them (it's not like they were ever not colorblind, then became colorblind, so they can compare what it's actually like to have color vision). That doesn't mean they're not impacted negatively by being colorblind- it just means they're better than their buddies are at spotting game (maybe their buddies are crappy or they are great)! You can still be good at finding deer and be colorblind. You're just not as good as you would have been if you weren't colorblind.
My son and daughter have been my trackers since they were very little. I can't find blood at all, unless it's on snow. I've also had times when my buddy spots something and tries to point out a deer because "it sticks out like a sore thumb" due to its color. I usually eventually find it, but it's tough!
Wish I had something brilliant to help the OP with, but I don't. It's a disadvantage, but you can still generally be good at glassing. Other than having good glass and continuing to practice year around (sounds like the OP is doing that), ya got what ya got!