Yes, if you sit above the deer on the same slope, the morning thermal may take your scent to them. That would be an odd glassing spot, though, IMO. If there is a prevailing wind, then, yes, it would dictate where the scent ends up.
I set up., usually as high as I can, on the opposite side of where I expect the deer to be so that I can see into the timber, and see the most country possible. Most of the time, I am approaching from lower, and I hike to the ridge top through the timber. Then I find a good spot to glass from (typically predetermined) and glass across and up the drainage toward the head of the basin. My scent goes down the drainage to country I've already been through.
You have to make the call in the terrain you are hunting, though. Might make sense to glass from the bottom, or the top. You're usually far enough away, it isn't as critical.
Jeremy