If you are moving, strip down to the bare minimum and layer back up once stationary. While some folks may like to have lots of sweaty layers to dry out, it is not my cup of tea.
I'll assume that you are day hunting and camp is not on your back...
I personally am not a fan of using a rain jacket in the desert. No point in using it for blocking the wind and protecting the under layers when the end result will likely be a "holey" rain jacket once done. Holes can range from pinpricks to full on tears. If so inclined, someone could use a Frogg Togg and consider it disposable due to cost (~$20).
You can look at fabric out layers such as the Kuiu Peloton 240 or StrongFleece. The Peloton 240 does well at blocking the wind and the material offers decent protection against cactus spines; however, the material can still "pick" and snag. The StrongFleece really doesn't do much for the wind and does better against "picking and piling" than many other fabrics; but it is still very snaggable. However, it is called StrongFleece and not Snag_Pick_Pile_Spine_Resistant_andor_Proof_Fleece. But then again there are several WKRs that appear to love the Velcro look; maybe they feel it breaks up their silhouette better.
Your best option is a low cost soft shell such as Kryptek Dalibor. This can often be found for a deep discount; Black Ovis has it for $101 right now. It does well at blocking the wind, shedding the rare light rain, and offers above average protection against cactus spines and other things that will poke and snag you.
