^^^This^^^Get the highest rated pad you are willing to carry and try it again.
Your sleeping pad is way under R-rated for a down bag in the temperatures you shared with us...regardless of bag sizing, you will sleep cold with that R-rating. Get a new pad with a minimum of R-7...even higher if feasible for your intended uses.
I have a WM Badger MF and a high R-rated pad, and it is perfect for me for late fall hunts up here in the temperatures you shared. I'm 5'10" 185 and my shoulders are wide and I sleep warm, so the Badger is perfect for me, but not if sleeping on a lower R-rated pad in colder temps. In fact, I was gifted the Nemo Tensor pad for Christmas a few years back, and took it with me on one hunt to test out...nope.
The best to you OP...WM are great bags!