Yeah, I'd pass. He may turn into something better, he may not.
I've listened to numerous deer biologists say that you will not change the genetics on your property by culling, unless you're a mega ranch in Texas.
As stated above by cmwhitmoyer, the does give half of the genetics. If they're in poor nutrition during gestation, recent studies suggest that their buck fawns will be behind the 8-ball for antler production.
Shoot him if he makes you feel happy, but I'd be looking to shoot his momma, aunties and girl cousins to get the extra mouths off the landscape. Should help with overall herd health and result in better nutritional status for the does during gestation.