Admittedly the dwarves in our world are only about a foot tall, so there's certainly that. But we also have giants at a hundred feet, and Little Oars Jr. is three times the regular size of a giant.

I suppose though, it's not really the height itself so much as what one does with it. I would like to be taller so I can stretch my mass more easily, but my brother Jozu lost his arm because of what a big target he makes. It's not easy being big either, eh?