I always liked Durango and Pagosa Springs.
I'm not really a fan of desert terrain either, but I'd pick Grand Junction over almost anything on the Front Range. I can't tell where "Denver" begins and ends while traveling north to south or south to north. It's one giant mass of people and traffic, but that's just my perspective.
I like Glenwood Springs also. Housing costs are out of control there also did to proximity to Aspen.
Other small towns that I liked:
Rifle, Montrose, Buena Vista, Salida, Eagle, Gypsum. Of those towns, cost of housing will be highest in Eagle and Gypsum due to proximity to Vail/Beaver Creek.
I lived in Leadville and worked in Avon. Leadville is not for everyone (many). Almost made it 5 years. Avon just costs too dang much.