This. 223 was a poodle shooter; 30-06 was the bare minimum for hunting in Alaska; the 270 shot light bullets and was too small for an ethical hunter to use on anything but deer; a 375 H&H was a good place to start on big bears and 338 win mag was for wimps, but I really liked a coworkers 458 Lott; Etc.
Those are all opinions I held not too long ago. Then I sat down and decided to prove I was right, now I carry a 223 bolt gun in country with bears; came across some brown bear tracks last weekend that my entire foot fit into without touching the edges, of course I have pathetic little size 9-10 feet, so that is not necessarily a "big bear."
If I was not able to pick my bullets, I would certainly opt for a larger caliber. But, being able to pick my bullet, I will go for a smaller caliber.
To the OP,
@Billy Goat covered it well with the vehicle comparison (and less wordy than what I first typed). Energy does not kill, energy transfer kills. Energy transfer is driven by the projectile, this is why an arrow can kill, or a baseball bat. Energy transfer to vital organs will kill more efficiently than energy transfer to the entire organism (why 120,000ish ft-lbs of a vehicle going 35 mph is not instantly lethal in all cases, of course that energy transfer is quite a bit more complicated than just the KE number as well).
Edit: Last point, it is KE, not energy. Energy covers a significantly broader range to include momentum, temperature, gravity, magnetism, Etc. Energy transfer applies to all of these.