I own a Z3 and have used a Z5 quite a bit. I didn't notice the fit and finish difference that Todd did. I am also surprised a bit by Todd saying the reticles are thicker on the Z3. I did not notice that (I saw BRH reticles in both), and Swarovski's literature I have seen shows the specific reticle subtensions are the same throughout the model ranges (Z3, Z5, Z6).
The differences I noticed:
Of course the Z5 offers a 5x magnification range, while the Z3 offers a 3x range (thus the model names).
Optically, both utilize the same optical technologies when it comes to raw glass, coatings, etc. This does not mean they perform the same, however, as each scope model has its own optical design. I will say both lines perform at a very high level.
A big difference that doesn't get much press is the Z3 uses a traditional leaf spring erector system. The Z5 on the other hand offers Swarovski's overbuilt coil erector system. My Z3 is quite accurate in adjustment and holding zero, but there is little doubt the coil system should be stronger.
I would guess the more complicated design and additional glass elements for a 5x zoom range combined with the coil erector adjustment system are the reason for the increased price and weight for the Z5 series.