Works great. It’ll be warmest with windproof shell OVER the insulation, as wind blowing thru the down face-fabric will remove more heat—the shell outside prevents this. You will get some condensation under a waterproof-breathable shell if thats what you’ll use and it can get your down damp, but generally not a problem for day trips. The extra space between layers can trap more air than if the same amount of insulation was in one garment, so it can actually be warmer than one insulated pant, and as you said its more versatile this way, ie 2 garments has three wearing options. And the down is way light/packable. I tend to do this with a jacket more than pants, but same/same, its part of my normal kit.
The down-side (funny, huh?) is down is typically in loud fabric so may not be great for very close range hunting, and the shell fabrics and down fabrics don't tend to be very durable so they can get shredded easily in briars or krummholz. I keep wishing for this “mobile hunting” thing to actually produce some legit packable AND quiet gear for this sort of application, but apparently not.
From a convenience of laying point of view I tend to use my down jacket as insulation under a shell or soft shell a lot. If its really cold and I want pants I have a set of synthetic insulated zip in pants, but down would be same—I generally put these on over my regular pants and call it good. If its warm enough to be really wet for me to want rain pants, that means its warm enough that I dont need the insulated pants in the first place. I might have a set of shell pants I could put on over those if I were winter camping or climbing, but thats not generally a hunting setup for me. Could be nice in late season prairie setting with cold, snow and lots of wind though.