This is one of the places where formal taxonomy and common usage diverge. People tend to think of "things like us" and "everything else." In a very broad sense, "vertebrates" are things like us: they at least have bones, unlike all those weird things. Dividing the world into X and not-X is conceptually appealing and sometimes useful, even when the only thing the not-X have in common is that definition by exclusion. In this case, the excluded group, invertebrates, makes up 97% of animals in the world.