It is really relative.
I currently own 3 Blaser r8 rifles. 2 of them are fully custom with custom stocks. Financially they cost more than a custom built off of a 700 clone, proof barrel and a fiberglass stock.
I have owned in the past 4 years 2 Bighorn TL3s, and 7 or 8 American Rifle Company Nucleus actions.
Previous low-high end guns factory owned; Sako, Tikka, Weatherby Mark V, Howa, Remington, Winchester, Savage, Sauer 90, Blaser r93, and a mountain of other Blaser R8s and K95s.
1. It won't make you a better hunter.
2. It may make you a better shooter as the firearms can be customized to fit your physical dimensions and shooting style.
3. I have seen 80s and 90's era custom rifles by people like Rigby, Holland and Holland and other big name gun companies built on Sakos and M70 Winchesters. So it is the old standard to use a factory rifle and modify it. But in the 80's and 90's there was not a source for custom M98 new production mauser actions.
4. Any custom or customized or factory rifle can be fraught with problems. Any rifle can, doing so may lessen the possibility, but it doesn't lock it to zero.
5. You can easily come to resent a poorly thought out and executed custom rifle. You can spend $1000-2000 on an action $1000 on a barrel, 400-800 on barrel work, $250-300 on a brake, 100-500 on a trigger, $300-1500 on a stock or chassis and still not like what you built. Your selection or the selection by your gunsmith or friends must be heavily researched and well though out. You need to spend time on the phone and on the company pages building your rifle virtually by adding the weights.
6. It is very important to try as many custom stocks and chassis as possible, so as not to end up with something you don't like because you don't actually like the way it looks or feels. Or you like both, but you don't like the recoil impulse.
7. Companies like Gunwerks and others who are selling you a turn-key package might be cheaper in the long run, after you have spent your time and money chasing rainbows in a custom system that doesn't do what you want it to do.
8. Yes I have done all these things and hunted with 13 pound rifles numerous times because I am an idiot and I didn't research all the things I should have in a build.