Since I see you are a GA guy, I’ll give some more input. I mainly know the fly fishing industry so that’s what I’ll speak on. Every fly shop in GA that I know of is owner or was started by someone with another job. There have been some other shops but they have come and gone. The only shop I’m aware of started ground up is the fishhawk, but from what I’ve heard Gary still used his “college” money to start it- so he didn’t exactly start from zero. They almost don’t count because they are in the absolute perfect location.
That being said, the Hunting/conventional fishing/camping market is much more sizable than flyfishing but you have many more competitors too.
As an example of competition, AO started out as a flyfishing/hiking/outdoor store, but I think they only saw real success once they decided to focus on fly fishing. That alongside competition from REI and online probably drove them into being solely a fly shop.
We have a few very successful gun shops in the area, most I see are going for sales volume over making a high margin. Today’s customer are so savvy you pretty much have to be. FFL gives gun shops a slight advantage because people can’t purely price shop once you account for the transfer fee.
As a specialized shop, employees are harder to come by too. The average highschool or college student doesn’t have the in depth knowledge needed. At the same time, you typically can’t afford to pay a ton either.