I think a lot of this depends on your internal thermostat. If I were going to put a down jacket or on it would need to be getting close to single digits.
I wear a light merino top 90% of the time early season. I have a Kryptec hunting shirt, rarely ever put it on, Kryptek vest and raincoat. That is all I take in with me. If I get chilly glassing or sitting I throw the vest on. I have yet to have to put the rain coat on for warmth, even when windy. I think I'm going to ditch the hunting shirt this year and just take the baselayer, vest and rain coat.