Hi sneaky;
Sounds about right. Double wall vacuum is supposed to prevent heat/cold transfer between inside and outside. Single wall will transfer heat from bottle to bag very well.
Also, if air has somehow got into the vacuum space, then heating the bottle to boil water or melt snow could -in theory- burst/explode the bottle.
The other poster, valtteri, said his throat gets sore from below zero temps, not that his face gets too cold. That's why he breathes inside the bag. I'd try not to do it myself, but I guess it works for our Polish friend, valtteri.
Edited to tighten up. Not like Archie Bell, though.