I'll echo the above with a couple of points unmentioned so far.
Your 10 shot "group" can be done in two 5 shot groups at different bulls on your target and provide the same data (as long as the bulls are identical).
You don't mention how fast you shot or whether you allowed the barrel to cool between shots. My standard practice is 3 minutes between shots and I try to keep the sun off the barrel. You can get away with 2 or 3 shots in fairly rapid succession if that is all you are going to shoot. I don't do that. Even if shooting to confirm zero after travel I am going to give 3 min cooldown between shots in case I need to do a sight adjustment. Note: I'm not saying 3 min is the magic formula, it is just what works for me and it is easy to track with an egg timer.
Keep your ammo out out of the sun.
Make sure all your ammo is the same.
Shoot every shot over a chronograph so you can tell if something other than group size is in play. In my .30-06, the POI shifts up and to the right as velocity increases.
Don't let a round sit in the chamber to cook. If you don't get a shot off in the normal time after loading, pull it and set it aside to cool.
I have a little bit different perspective on this...
If a friend were engaged to a girl and he was having trouble keeping her from having psychotic meltdowns when she's hungry, or tired, or stressed out, would you advise him to figure out how to make sure she's always fed, rested, and relaxed? Or would you advise him to maybe rethink marrying her and try to find someone that doesn't lose their mind when things aren't going great?
Not saying it can't have an effect, but if leaving my ammo in the sun or shooting five shots in a minute causes me to go from "all good" to "measurable problems", the system is no good. Either I'm loaded way too hot, or my barrel isn't stress relieved and of good quality, or whatever other issues might be, I'm going to work toward fixing the rifle and/or ammo rather than try to walk on eggshells trying to keep my rifle happy.
Edit to add:
OP, do you have a scope level (installed correctly)? If no, cant could account for part of the R-L shift at distance.
Much more likely though, is what multiple people have said. If I were in your shoes, I'd shoot a 10 round group at 100 to verify zero, and if it's spot on then shoot what you think you need for elevation, 10 round group at 500. Do it slow if you want, there's a big gray area between waiting 3 minutes per shot and ripping off 20 shots in 2 minutes.
Those two 10 round groups will tell you more than 10 different 3 round groups making adjustments in between.
