EC4reel,
I'm glad that you did this pre-season test and found out what will likely happen. It always surprises me that people will come up with an idea, be it for changing their gear or some new recipe for dehydrated food, and then ask people what they think of the idea when the best thing to do is to try it out at home first. To go with the food analogy, if you can cook it on the side of a mountain you can cook it in your kitchen too to see if you like the taste.
As for going UL on your gear we all need to keep in mind that UL is about camping and hiking rather than hunting. It also, as others have alluded to, about taking just enough to be safe and enjoy your trip which may or may not be enough to have enough energy or enough sleep to be able to hunt effectively and to enjoy the whole experience. So, keep in mind that UL hunting is a bit different from UL hiking/camping.
If you want to use something different for your sleep system I would suggest you take a poncho and poncho liner/woobie combination. You can use this during the day to keep yourself warm while sitting or glassing. At night you can use it as a quilt with your sleeping pad. The poncho, or a sheet of tyvek if you want some breathability, will keep the wind off of you and help keep the warm air in to fight the drafts you experienced.
Another thing to consider taking is the American Medical Kit (AMK) Survive Outdoors Longer (SOL) bivy sacks. They are lightweight, reflect heat back onto you, and keep the wind off. Their downside is they are not made for those of us who are taller or over 200 pounds.