I did the following experiment on myself many years ago, and have done it with a dozen or so others, with the same outcome.
I started with a base pack weight of 22.5 pounds. This weight included some hunting gear and 2-2.5lbs of weights from a vest. I also had an additional 2.5lb weight that I gave my wife. I asked her to either add the additional 2.5lb weight (25lb total), remove one of the existing 2.5lb weights (20 lb total), or do nothing to the pack at all (22.5lb total). I left the room while she was adjusting things, so I had no clue what she did. My goal was to see if I could physically tell the difference between each of these adjustments. I lifted the pack by the grab handle with one hand. I lifted the pack with both hands on the sides, as if loading it into my rig, and put it on, as if I was going for a hike. I repeated the process in 10-pound increments up to 80lbs.
Results I have seen indicate that a 2.5-pound variance in pack weight was most accurately detected at a base weight of 22.5 pounds, but it could not be accurately detected at loads above that.
I agree that a whole bunch of slight gear weight reductions can add up. If you are new to backpack hunting, you may be able to get enough reductions to add up to 4-5lbs — or more—that's noticeable—but most dedicated backpack hunters are already fairly dialed in and will not be able to find an additional 2-3 lbs of gear reduction to make a physically noticeable difference. To your point, they will likely add more gear to make up for any weight loss. Yes, people spend ridiculous amounts of money to lighten their pack load, but that doesn't necessarily mean they're getting any physical benefit from that expense; it just means their bank account is lighter.
My point is that adding or subtracting 2 pounds to your pack load will not make or break a hunt because it's not noticeable on your back; it's only noticeable on a scale.