It's funny that folks worry about muzzleloader but not the fact that rifle bear season is almost all of archery, so there are guys out there with high powered rifles at the same time as well.
I hunt both muzzy and archery and I would say that the only concern that I have is if someone happens to be behind the elk I'm shooting at. We hunt mostly private or smaller chunks of BLM that few people hunt, so it isn't as big of an issue, but I would suggest an orange hat or bandana just in case I come into your area and you would like me to see you, you can pull it out of your bag. That probably goes for just plain archery season too if you don't want someone walking right through where you're sitting, just wave a hat at them and they'll most likely go the other way.
The only time I have thought there was a dangerous situation was this past year when we called in these archers, but they came in calling with a heads up decoy sticking out behind a bush. If you have one, it may be worthwhile to put an orange line on it or something so that a person can tell that it's fake. I'm not going to blindly shoot through a bush when I see a head, but I wouldn't bet on everyone being that way.