I'm similar weight and an inch shorter than you and my size L works great for layering similar to what you describe. I also have room for additional layers when it gets down around 0.
Large would be perfect as long as you don't plan on wearing anything too lofty underneath...I'm 5'10/160lbs (28.5" draw length) and a Medium is spot on for me.
I’m 5’11 195 and the large was tight on me that was the older model now I have the updated version and got the XL and fits way better I do layer under with merino LW then fanatic hoody and the Sitka midi jacket keeps me toasty and can still draw my bow with no problems.. good luck