My past experience on public hunting some beaver swamps and wood duck holes on National Forest has been pretty good overall. I get there early, trying to have my small number of decoys out by 5-5:30 am. Guys roll up after me, sometimes right at shooting light. I'm generally by myself and they have come in groups of 2-4.
Let them know I have coffee, snacks, extra shells and would be happy for them to sit with me. They've all declined, no pissing match, etc. They've walked 100-200 yards away and I've gotten my 3 wood ducks before them and told them to take my spot.
The first time I ever had it happen, I was asked why did I bring all those decoys/stuff when wood ducks don't decoy anyway. The three I killed that morning all circled back and landed in my spread. Oh well.
OP, I would get there first if you really like the spot, get all set up and if they come in after you, let them know that the company is welcome (as long as they're not intoxicated or squirrely acting). I'd be courteous, but if they get their panties in a wad ask them to pick up your blocks for you and then go on your merry way.