I'm a strong advocate of personal choice and personal responsibility, but I'm also pretty big on self preservation. I wear orange during rifle season, and so does everyone I hunt with. I also wear my seat belt and have life preservers on the boat. Personal choice.
The only reason I give a darn if someone else isn't wearing orange is the legal aspect (yeah, I hate to have to even think about it, but that's reality). I don't want to have to spend years and a ton of money defending myself if I accidentally shoot some yahoo wearing a white suit in the snow that was behind the bull I was shooting at (like the guy I saw a couple of weeks ago in MT while elk hunting). Even though you'd think it was a slam dunk, the guy wasn't wearing orange like the law requires, you still could and probably would be sued by his family. There have been a couple of times when I spotted an orange dot in the distance and made a mental note to not shoot in that direction.