Drone operators are, obviously, the problem here. Drones are very useful and incredibly powerful tools, but like any tool, they can be and often are used inappropriately by the person using it.
I've had too many negative encounters with idiot/asshole drone operators to recall, but one stands out.
Last fall we were at a public park with some friends, families with small kids, having a birthday party for my 5 year-old son. A couple of older dudes started flying their drones a few feet over the heads of the kids on the playground, close enough that the kids were jumping and trying to grab them. I walked over to the two operators and politely asked them to not do that any more and that I thought it was unsafe if not totally inappropriate.
Not long after that interaction one of the operators apparently lost control of his drone and it flew into/under the picnic shelter where a bunch of us were hanging out under, crashed into one of the pillars holding the roof up and narrowly missed hitting my wife's face by a few inches at most. After bouncing off of the structure a few times, it remained hovering at about face level and stationary. I walked over, grabbed it out of the air, and returned it to the owner with the promise that if it came near us again, the drone would not be the only thing getting broken.
The guy was totally unapologetic and seemed to think that I was the one being unreasonable...