While I enjoy hunting and fishing as much as anyone, I also enjoy flying a drone when on public land where it is legal to do so. I'm sorry that it might annoy some people but that is how I choose to enjoy the outdoors and public lands at times. I understand that the noise could possibly be disrupting to a peaceful day of fishing but that is similar to an "anti" argument that gun shots are disruptive to a peaceful day of hiking. It's a slippery slope when we start restricting things because of how it might disrupt others using the area.
Now, as far as them actually watching you, well, maybe. However, unless they are flying a very high tech drone with a very expensive camera, they really aren't going to be able to see much without being very close to you. Below is a video I shot a few weeks ago of my buddies and I playing with our Tac-14 and Shockwave "shotguns". The drone was maybe 30 feet up and at different positions around us. Shot in 4K, you still don't see that much without some serious enhancements. It's like filming with a GoPro. It's meant to take in a wide view, not super zoom images.
Clay Pigeons with the Remington TAC-14 and Mossberg Shockwave - YouTube
Besides, unless you are doing something illegal (in the example of fishing, hunting or whatever on public land), why does it matter?
It is a federal offense just like shooting at any other aircraft.
A quick quote: