I am not an engineer or designer. All I’m offering is what I’ve seen and some of what I think is happening.
First, it is common for the scope tube itself to bend. Maybe slight, sometimes not, but they do. Long, thin tubed scopes with large objectives are not kind to zero retention.
Second, shifts are generally in the direction of drop on the sides- left side drop cause POI to move left, right side to the right, top impacts cause POI to go high- not always, but mostly.
Third, some scope do it relatively consistently, some randomly- but it is not uncommon for a drop to cause a shift, then the next shot either moves back to POA or moves back closer.