JMHO...you are correct something is wrong. 100 rounds in you should be seeing better groups.
Since you are trying all the recommended things I will give you one more I haven't seen mentioned.
@hereinaz has alluded to it, but didn't finish.
Take your action and just snug the rear screw. Hold it at an angle but barrel up towards vertical. Then put your finger between the barrel and top of the stock on the forend. Now snug up the front action screw. If there is motion that you can see and feel you have action torque. If there is no movement, just snug the front screw and then repeat with your finger on the barrel by snugging up the back screw. If you have motion you have torque. If you do you can try to locate the source and "remove it". Or you can bed the chassis.
I have heard some have had to relieve the bolt stop area as it was making contact. Check the bolt stop and trigger areas for contact.
Last thing to check is that the face of the recoil lug is actually making contact in the chassis. You can put silver sharpie or something on it. Then install the stock and move the action side to side. The marker should wear off or transfer. If you only have slight contact spot bed the lug. Of course if you have torque then bedding it takes care of it too.
Good luck. Troubleshooting is always a tough one. Eliminate what you can, try new scope, etc.....If it still won't shoot talk to Legacy Sports, re-barrel it, or trip it.