Primers affect the speed at which the powder column is ignited, and if you'll let me use an engineering term, the rate of pressure rise. That in turn will impact velocity and possibly barrel harmonics somewhat.
Are primers completely interchangeable? No.
Can you get a number of different primers to work well with a given load? Probably so.
Will you have to tweak the charge? It's likely if you're on a node and you'd like to stay there.
Will it change the way the load behaves? Probably.
But except in extremely rare cases, it won't change 3 inch groups into 1-in groups.
There are some corner conditions, like using light primers with Magnum powders that are exceptions to these general ideas. Another is using a small rifle primers in larger capacity cases at extreme low temperatures. But I think your question was more general than that.
That being said.... If you're precision shooting, sometimes it's useful to test several primers. In theory, low ES indicates that your case capacity, powder, primer, and bullet weight are working well together. One primer type May shine out better than the rest.