I mean if you've got your sleeping bag in your pack I dont think you would need puffy pants too. I havent, however, ever heard of anybody carrying their sleeping bag with them on day hunts...
If it were me I'd keep the puffy layers and ditch the bag.
Now you have
It's a 50 degree Apex quilt- weighs 12 oz and packs small. It also comes with a "poncho hole" so I can use it as a final layer when glassing when it's really cold. Obviously with an unexpected night out it would be most welcomed.
I don't use my puffy pants a ton, but mid-season on (earlier this year!) they go in my pack. 7-8 oz and pack small, so worth it for me. When I do need them, seems like a I really need them