The EL's and NL's are both fantastic optics. That said, I generally think the word "Swaro" is to high-end binos what Kleenex is to all forms of bath tissue. "Hey Bob pass me those kleenex" even though the box says Kroger on the side. I don't personally believe it's because Swaro is actually the best, it's just the most well known, and that's why all other binos are compared against it is because everybody recognizes the brand and what they are.
Swaro has name recognition, it has status tied to it. I'm convinced 100000% that if we could put the optics of all the EL/NL/SF/Noctovid/Meopro/UHD into the same "shell" and force people to look through them all and make an objective choice for what their favorite was, I think many of the Swaro purists would end up elsewhere. That's not to say they don't make great glass, but I think most people do just what you're describing. They want "the best" so they just go buy Swaro and don't even look at anything else because such-and-such guide services they hunted with was using them and said they were best, or X forum online said it's the top of the line.
I'm also in the camp of Vortex Razor UHD over the EL's. My Dad is the same way. We prefer the truer color palette through the UHD's and the superior (to our eyes) low light performance gains. But it took some true soul searching and admittance to get to that point.
Bottom line, there are TONS of fantastic optics out there. You would regret NOT looking at the other high end options on the market, and would potentially find something your eyes like better than the EL's or NL's.