For deer I used to be very meticulous and remove all silver skin, fat etc. I've slowly become less picky with my grind pile. Now I just remove all the fat, glands, some of the thick silver skin, and cut off the ends where the connective tissue comes together on the shanks for example.
In fact we did a taste test last fall comparing perfectly clean meat and my choice of grind ^ and my buddy looked me dead in eyes and said "I've been wasting so much time. I can't tell the difference". These are MN farmland deer, so I don't know if that makes a difference.