If its the right 20 acres with the right nieghbors, it could be plenty of property to provide you with a place to hunt, but no matter the size of a parcel, you can only control what is done within your boundaries, so the dynamics of the area as far as potential development should be something you have a good feel for before buying. That perfect spot wedged between two large agricultural or timber tracts could one day be surrounded by houses or some type of commercial development. People = problems.
I own my own place, and it is the single greatest source of joy, and greatest source of misery, as far as time I spend in nature. There is nothing like working your own dirt, but you also are never 100% just recreating because if you are like most people, there is always work to be done, and that list of things to do will creep into your mind when you are sitting in the stand just trying to relax. You will have a list of things you want to do every time you go, but when you get there you will find one or five things you weren't planning on doing as well. I enjoy hosting friends and family, but also deal with tresspassers, stray dogs, hogs, mother nature (trees falling across the road, clogged ditches, washed out roads, overgrown areas that need bush hogging, etc,). I run cellular cameras so get updates daily of who or what is on my land. There is never much down time between "problems" that need to be addressed.
If you can afford it, buy the land. If you read the threads on here, you see that most peoples frustrations with hunting have to do with opportunity to hunt (tags), decline/lack of game, or competition/negative impacts on their experience due to other people. Most of these people are hunting on public land. Having some private land in whitetail country where you can go year round whenever you feel like it, and are essentially guaranteed the opportunity to hunt every year and be successful, limited only by your own time and trigger control, is something that most people do not have, and is only going to get harder to come by as time goes by. Your only regret will be that you didn't buy more. A wise man once told me, "Buy all the land you can afford, so you can pick your nieghbors...but, you'll find that the best nieghbors are the trees."
I did notice you said your wife would not go....this is not good. I'd get her involved in a vision of the future where you all have a cabin or something where she would want to go. Talk to her often about your vision for the property and get her involved in that. You don't want it to be something that drives a wedge between you, or that she resents in any way. Long term, you will have a much better chance of keeping it if she gets enjoyment from it as well. Owning land is usually a long term play as far as the fulfillment you will get out of it, as well as any financial return on investment, but the rewards for both can be tremendous.