Have you ever tried dragging a hole around?!
But seriously: in a conductor, and even in a semi-conductor, electrons float freely in the material, so moving them is easy (they're already moving). A hole, however, is a gap in the orbital of an atom. To "move" a hole, an electron must be pulled from an adjacent orbital to fill that hole and open an adjacent hole (i.e. the hole created by the vacating electron). That takes extra time (and energy).
BTW: Holes don't actually move. Imagine an egg carton, and imagine that there's one egg missing on the left side of the carton. The empty slot in the carton represents a hole. To "move" that hole from the left side of the egg carton to the right side of the carton, the way that holes move in a semiconductor, remove the egg that is to the right of the empty slot and place it in the empty slot. That created a new empty slot next to (and to the right of) the previous empty slot. So, conceptually, the empty slot (or hole) moved one egg carton slot to the right. Now remove the egg that is next to, and to the right of the new "hole" and fill the new hole with it. That opens a new-new hole to the right of the previous new hole, etc. If you do this 3 more times, the hole will be on the left side of the carton.
Note that it takes longer to move the empty egg carton slot, than it would to just toss an egg from one side of the carton to the other. Thus the difference between moving electrons and moving holes :)
BTW: if you want a more scientific (quantitative) explanation with equations and such, you're more likely to find it at a forum dedicated to physics [most engineers are probably like me--they studied this in school, but forgot most of it, because, to apply semiconductor components, one does not need this detail of understanding. Probably the only engineers that retain this information are the ones that actually fiddle with the chemistry of semiconductors].
