You can still share a meal with someone who you disagree with... It's impossible to tell "tone" and "opinions" on the internet or from one potentially heated conversation across a lot line... I promise, he'll be just fine, and maybe even grow as a person from it. The guy will likely say "no" anyway but the neighborly thing to do is to extend the olive branch just about every time.I'd leave him completely alone - he has already shown you what he thinks of your hunting. To further pursue him is gonna piss him off.
You've already talked with him, what's breakfast or lunch or even drinks gonna do? Answer is no and he spent X hours building a wall to confirm his stance. That's what telling him where the deer come & go did.
You can open a hole in his wall and have a serious hunting funnel. Shoot em as soon as they get thru it so they run the other way.
Most folks don't really care for hunting, or they'd do it. Some dislike it - as this guy apparently does.