I would imagine that's much more likely due to other ferrous items in your kit than magnetic enclosures.
Neodymium magnets are widely used in applications like this because they pack a lot of "gauss" in a small puck. They're rated in grades, typically N35 being on the low end and N52 on the high. Since those are expensive most consumer products use something in the N42-N48 range.
See
https://www.kjmagnetics.com/magnet-strength-calculator.asp. An N48, somewhat on the high end for the sake of the argument, at 0.25" dia and 0.125" thick is at the high end of "affordably strong" but is barely detectable past an inch away from it. There's a reason these things don't wipe your credit cards. You can prove this yourself by just getting a few "rare earth magnets" from any source and placing some random other magnets / paper clips a quarter inch, an inch, a foot, etc away. Past a few inches they won't make a difference at all.
What a lot of folks don't realize is that the Earth's magnetic field is actually quite weak. An N48 magnet is like 13k gauss. The Earth's magnetic field is like 0.5 - thousands of times weaker. The only reason a compass registers it at all is because its field is enormous - it extends 40,000 miles above the Earth's surface. A paltry N48 disc magnet is thousands of times stronger, but only detectable an inch away. So here's the thing. You may not detect other MAGNETS a foot or two away from your body. But all ferrous items on your person will disrupt Earth's magnetic field for a fair distance away from your torso. Your rifle, binoculars, hunting knife, and all those other items will bend the lines of magnetic force for quite some distance from you even if they're not magnets themselves. This makes it easy to blame a bino harness when it's really all the other stuff on you.
Either way, holding your compass or phone away from you a fair ways is smart anyway. But it's almost certainly not the magnetic enclosures in your bino harness causing the problem.