Cut strips of cardboard off a cereal box and loosen the actions screws, wedge the cardboard down the sides of the action, and tighten it back up - if the horizontal goes away your bedding has slop in it. Even round receivers can have a flat area on the stock and the action moves back and forth. I have a wood Remington stock and the flat spot is really obvious. This is such a common issue there is cardboard in my range bag.
Some shooting styles put side pressure on the pistol grip - if you’re pulling toward you each shot with the trigger hand to line up the crosshairs it will cause weird things because the pressure is different each shot - only move the crosshairs with your shoulder.
Worst case, defects in the barrel can cause it to walk as it warms up. Even custom barrels can get bad steel, factory barrels more so. I recently heard of a new custom barrel that moved point of impact 2 moa in 10 shots. The company that made the barrel replaced it after seeing metal in the chamber was wonky.
There also can be partial contact on one side of the barrel - the barrel vibrates against the tight spot throwing shots sideways different amount.
Loose scope bases also move side to side causing wide groups.
Internal loose scope lenses can move in any direction, but could be left to right.